---
title: An AI you can leave on watch
slug: sentinels
summary: "Designed Omniscient's Sentinels monitoring surface: the agent behaviour, a tiered-rigour funnel, and an investigator-plus-vetter loop so an unattended AI earns every interruption."
url: https://charlieholland.com/#work/sentinels
---

# An AI you can leave on watch

**Company:** Omniscient AI · **Role:** Founding Product Designer (sole designer, working across product strategy, design, and code) · **Year:** 2025 to present · **Stack:** TypeScript and React (my code); Semantic Kernel and Azure Durable Functions (the agent platform).

Sentinels is the always-on monitoring surface of Omniscient, a signal intelligence platform. It's the standing watch that keeps working after everyone stops reading. I designed its agentic layer, from the agents that investigate, vet, and alert while nobody is looking, to the moments a human steps in.

Headline outcome: AI filters around 70% of the noise that keyword-based tools alone let through.

## Overview: the AI is the product, so the AI is the design problem

A sentinel is a standing instruction: "alert me when production halts at our Morocco plants." Behind that one sentence sits the hardest AI UX problem in the product. Everywhere else, the AI works supervised, with someone looking at every output. A sentinel works unattended. It reads the news firehose, investigates candidates, second-guesses its own conclusions, and decides what deserves a human's attention.

The loops the agents run, what an alert owes its reader, and how a person stays in control of a system built to run without them.

## Challenge: an alert is a claim on someone's judgement

A comms lead's phone lights up at 7am. Production has halted at one of the company's Morocco plants. The watch fired exactly as asked. Forwarding it to leadership and being wrong means the next alert gets ignored. Sitting on it and being right means the team is behind a story it should have led. The first challenge is one question. Why should I believe this alert?

The second challenge sits behind the curtain. Agents aren't interchangeable. A keyword match is instant and free, but literal. A classifier is fast and cheap, but shallow. A full investigation is thorough, but slow and expensive. Sentinels v1 flattened those differences and ran its heaviest agents on everything: poll the news index on a timer, investigate whatever comes back, alert on the next tick. Alerts waited on the timer, however fast the story moved, and a low-stakes brand mention ran the same expensive pipeline as a geopolitical crisis. The real failure wasn't speed or cost. It was treating every job as the same job.

## Approach: from human scenarios to agent behaviour

### Start with who the watch serves

Before any agent behaviour, I mapped the scenarios into a matrix: who's asking, what they ask in their own words ("any critical cyber signals this week?"), where they are when the alert lands, and what they need from it. Two things stood out:

- Two altitudes, one watch. Analysts do the heavy work of detection, curation, and deep dives. CxOs read a ranked threat list, the top issues that landed overnight. Most watches serve both, so the same alert had to hold up at both altitudes.
- Alerts arrive where people already are. Slack, Teams, mobile, email, not the platform. The comms lead reading that 7am alert is in a chat app, so the alert has to carry its credibility with it rather than borrow it from the product around it.

### Tier the rigour, not just the speed

The next decision was how hard the sentinels should think. Sentinels v2 replaced polling with a real-time firehose feeding a three-stage funnel of progressively smarter, progressively more expensive gates. Keywords do the first cut for nothing and stop about 99 articles in 100. Only what survives reaches the AI, and only the highest stakes reach a full investigation.

Scored against each other, the three tiers look like this:

| Tier | Speed | Noise | Cost |
| --- | --- | --- | --- |
| Keyword only | High | High | Low |
| Keyword classified (default) | Medium | Medium | Medium |
| Deep analysis | Medium | Low | High |

- Keyword only fires on keywords alone, for tight, well-scoped topics where you want every literal match and no model deciding which ones you see.
- Keyword classified (the default) adds the AI classifier on top of keywords. It reads every match against your plain-English criteria and drops most of what a keyword tool would have sent you.
- Deep analysis runs the full agent loop, for reputational, regulatory, or geopolitical stakes where a false positive lands in front of the board.

Each tier runs more of the funnel. Keyword only stops at the keyword gate, keyword classified adds the classifier, and deep analysis runs the full loop.

The funnel engineering was the team's. My job was the behaviour spec, and shipping the user-facing side directly: what each tier promises, what it's allowed to get wrong, which one is the default, and how the three are named so a comms professional picks correctly without ever seeing the machinery. The principle was simple. Spend the AI's effort where mistakes are expensive.

### Close every loop with evidence or a human

Most of the important design decisions here never rendered as pixels. They were rules about the loops agents run and what it takes to exit one.

So I gave deep analysis one hard rule. The investigator agent never reaches a person on its own. Its draft goes to a second agent, the vetter, which scores the evidence on timeliness, source validity, attribution, and consistency. Only its approval sends the alert. The vetter never sees the investigator's reasoning, only its claims and the sources behind them, so it cannot inherit the same mistake. A rejection goes to the log instead of someone's phone, so if the story develops later the trail is already there. Keyword tiers skip this entirely, because there is no investigation to vet.

A person still sees the watch before it starts. Every sentinel is previewed at creation, so you read what you are switching on.

For me this is the most interesting part of AI design. Deciding what an agent has to show, check, and ask before it's allowed to interrupt someone.

## Results: every high-stakes alert arrives with its evidence

In demos, the thing people stop on is an alert that opens with its sources already attached.

This work is young, and I'd rather be straight about that than dress internal milestones up as ROI. What's real today:

- From polling to real time. v1 alerts fired on the next scheduled check, not when the article landed; v2, now rolling out, triggers off the firehose itself, in seconds on keyword only and around a minute once a classifier or a full investigation runs.
- The AI catches what keywords let through. Of everything the keyword gate passes to the AI, the classifier and investigation tiers drop around 70% before it reaches a person. That is a drop rate off our own counts, not a precision score.
- Every high-stakes alert is auditable. Deep-analysis notifications ship with the reasoning and every source behind it.

The numbers that will matter most are still accruing: alert precision by tier, the share of alerts users act on, whether deep analysis pays for itself. The tiers are separate on purpose, so precision reads per tier. A blended average would hide which tier is weak.

## What was hard: the honest version

Designing for the absence of events. A monitoring surface mostly shows nothing, and has to make nothing feel like coverage rather than silence. Even v1 wrestled with this. High-priority sentinels ran the investigator on every check precisely so it could reason about the absence of news, which is part of what made v1 so expensive to run.

The cost, latency, noise triangle. Every tier trades these against each other, and the temptation is to hide the trade. I put it in the user's hands instead, with a sensible default. The hard part was finding names that carry the trade-off on their own, with no manual required.

Confidence compounds slowly and breaks instantly. One misleading alert to leadership costs more than a hundred correct ones earn back. That asymmetry justified the vetting agent, the mandatory sources, and the preview at creation. On an unattended AI, being wrong loudly is survivable; being wrong quietly is not.