Alerting, from zero to one
I owned Alerting on Operations Advisor from its first framing in late 2024 through production. There was no alerting for cases outside the operating plan. I took it from zero to one—a reusable rule, an alert instance, a recommended action, and an operator response the system can audit—and then through a live deployment that showed us where the spec was wrong. Built with four backend engineers, three frontend engineers, a UI/UX designer, and data engineering. That opened maintenance and asset-performance cases the plan-only product could not reach.
role: product owner, Alerting · window: Q3 2024 – 2026 · status: in production at a district-cooling plant (13 chillers, ~1,300 historian tags) · team: 4 BE · 3 FE · 1 design · data eng · channel: email, one per new alert · evaluation cadence: site-configurable (10-min at the live site)
At a glance
| Problem | Operations Advisor could tell a shift whether it was on plan, but nothing outside the plan—an instrument drifting, a chiller approaching a fault, a loop out of band—had a detection, a notification, or an owner. |
| What shipped | Reusable Rules with a nested expression builder · Alert Configurations with draft → publish → archive lifecycle · system-owned Open/Closed instances · operator-owned Response Status (New → Acknowledged → In Progress → Completed) · structured, status-keyed comments · Live Operations dashboard, process diagram, Root Cause chart, version history · one email per new alert · Sensor Health labels on alerts. |
| Where it runs | Production at a district-cooling operator in the Gulf: 13 centrifugal chillers, ~1,300 historian tags, 63K+ CMMS records, 30+ trained users, a 40-use-case functional design (18 high-impact) signed before go-live in early 2026. |
| Three decisions I would defend | (1) Completed ≠ Closed: the system owns alert truth, the operator owns the response. (2) Sensor Health ships as labels, not suppression, until the detectors earn trust. (3) After the outage: current evaluation beats backlog catch-up, and cadence is a site setting, not a platform constant. |
| What production taught us | If the pipeline stalls, the loop goes silent—and silence looks like health. The rebuild spec treats no-silent-loss and freshness honesty as product requirements. |
Screens, slices, and sequencing
A large part of my PM work on this track was scope slicing—turning a broad alerting charter into a sequenced set of screens and releases the org could execute without losing the thread. With design and engineering, we mapped six core surfaces: Draft Alerts, Alert Creation, Active Alerts, Rule Creation (modal), the Live Alerts dashboard, and Alert Details—then explicitly parked adjacent ideas (work orders, work-order history, asset health timeline) so they did not collapse the MVP.
| Pass | Scope | Why in this order |
|---|---|---|
| V1 | Credible tables, foundational authoring fields, an early Live Alerts list. | Prove the spine end to end with real triggered work before investing in polish. |
| V2 | Plant-wide summary, urgency-driven prioritization, investigation detail, lifecycle and audit, notification policy. | The “serious ops” layer—what makes the loop legible across shifts. |
| V3 | Maintenance integration, longitudinal asset views, work orders. | Deliberately held back until detect → notify → resolve was trustworthy in production. |
Detect, notify, resolve
Industrial teams rarely suffer from a pure lack of signals. They suffer from a broken response chain: who owns this, what is the next step, did anything actually change, and how do we know it worked when the shift turns over?
The pattern that keeps recurring is three movements, in order:
- Detect. Something meaningful happens—or is predicted. In this product that means a rule over live tags, where a “tag” can be a raw historian signal, a calculated tag, or an ML-derived tag (anomaly score, forecast). The design choice is not only sensitivity; it is classification: urgency and context stable enough that two operators on two shifts do not reinterpret the same event differently.
- Notify. Delivery has to match operational reality, and the classic failure mode is not “we forgot a channel.” It is the unfiltered firehose that trains people to ignore the channel entirely. We shipped the narrowest honest version: the Open Alerts list for the control room and exactly one email per newly opened alert for configured recipients.
- Resolve. This is where many products quietly stop. They celebrate the red badge. Industrial work needs a response status, a record of who moved it, structured reasons when the recommended action was not taken, and traceable history—so “we saw it” becomes “we did something about it,” with accountability that survives audits and turnover.
String those together and you get a sentence that sounds almost too simple: detect → notify → resolve. Simple to say. Hard to ship—because each step crosses data models, permissions, UX, and organizational politics.
Where Alerting sits in the system
Alerting is a peer to the plan-adherence surface, not a feature nested inside it. It has its own evaluator (the Alert Service) and does not depend on the plan-adherence reasoner or on a live plan being active. The two share the plant’s tag universe, which is the point: an alert firing and a plan priority entering mitigation are independent facts about the same signals, and the UI, the docs, and later the copilot work all had to preserve that distinction.
Three properties of that path mattered more than any screen. Cadence: raw samples arrive roughly every minute, but rules evaluate on aggregated windows, and the window size turned out to be a customer-visible requirement (more below). Backfill: late data corrects historical windows, so open/close history has to be consistent with corrected values without re-sending notifications. Freshness: an alert evaluated on a stale window is worse than no alert, because it looks like coverage.
Two state systems, never conflated
The single most important modeling decision was to keep what the plant is doing and what the humans are doing about it in separate state machines, controlled by different actors.
| State system | Controlled by | Values | Answers |
|---|---|---|---|
| Alert instance | System (rule evaluation) | Open · Closed | Is the configured condition true right now? |
| Response Status | Operator | New → Acknowledged → In Progress → Completed | Where is the human response in the workflow? |
Completed ≠ Closed. Marking an alert Completed means every recommended action was taken. It does not close the alert; only the condition clearing does. That rule gave us an escalation signal for free: an alert that is Completed but still Open means either the recommended actions did not address the cause or the rule itself needs review. It also resolved the tension we had documented early between instance churn (a template re-firing many times) and stable ownership (one coherent thread per real-world issue): the system is free to open and close instances on rule truth, while response history and comments stay attached to the operator’s thread.
A consequence we accepted openly: Assignee in v1 is the last user who changed the Response Status, not a managed assignment. Dedicated assignment and automatic escalation were specified and cut (see decisions below).
Live alerts dashboard & details
The operator half of the product lives under Live Operations → Alerts: a real-time view of alerts that are not tied to a single plan, aggregated across the site so teams can see what fired, what matters most, and what to do next.
Overview panel: total triggered alerts as an open vs closed split; open alerts broken down by urgency—High (red), Medium (orange), Low (yellow). Work-order count and estimated preventive-maintenance benefit appear as placeholders for integrations that were specified but not shipped; I chose to leave them visible as a roadmap promise rather than fake a number.
Process diagram: the customer’s own PFD / P&ID recreated as an interactive map. Units with open alerts are highlighted by the highest urgency present in the unit, with a badge for total open count. It requires a one-time onboarding step (customer supplies the diagram, we digitize and map alert locations), which is an honest operational cost of a spatial view and one I would want self-serve eventually.
Open Alerts table: alert, urgency, triggered timestamp with “open X days,” location (Site / Unit), asset type, labels, an inline Response Status dropdown, assignee, available actions count, and last comment. Filters cover overdue (past expected response time), urgency, response status, asset type, and a location tree. A second tab, All Published Alerts, shows the full configured inventory so the Open list can stay triage-only.
Alert detail: urgency, location, Response Status, last comment; the triggering rule(s) with a toggle between symbolic names and the actual expression; a Sensor Health panel listing unhealthy vs normal tags for this alert; numbered recommended Actions; threaded comments; and a Root Cause chart—the trigger tag’s time series, the rule threshold as a dashed line, and the periods the alert was open as shaded bands, over 24h / 3d / 7d / custom ranges, with a rule selector when an alert has multiple triggers. Alert Version History shows every publish, unpublish, and edit with user and timestamp, so “why did this alert’s behavior change” has an answer.
Authoring & publish semantics
Alerts share DNA with plan objectives in the underlying platform—the backend still calls an alert configuration a “site objective”—but operator-facing language stayed consistent: Alert and Rule in the UI, never the legacy vocabulary.
Rules are first-class, reusable entities. A rule is a tag expression of the form ( Tag Operation Expression ), where the right side can be a constant, another tag, or a nested sub-expression (( SensorA > ( SensorB * ( SensorC + 100 ) ) )). Rules can reference calculated tags, so a derived KPI is a valid trigger. One rule can drive many alerts. That reuse needed guardrails: an “In Alerts” usage count so an author sees the blast radius before editing, and a hard block on deleting a rule that any alert still references.
An Alert Configuration carries name and description, location (Site / Unit), urgency (High / Medium / Low), an optional expected response time that powers the Overdue filter, asset type and sub-type, free-form labels, one or more trigger rules, ordered recommended actions, and email recipients. Alerts at this stage scope to site and processing unit; zone-level scoping was deferred.
Lifecycle: Draft → Published → Archived. Publishing starts monitoring. Editing a published alert affects future instances only—open instances keep the configuration they fired under, which keeps the audit trail truthful. Archive stops new instances and preserves history; Delete is allowed only for drafts that never published.
Trigger delay was added after the first production weeks: a condition must hold for a configured duration before the instance opens. It is the simplest anti-flood mechanism we had, and it shipped with its own follow-on bugs (duplicate delays, multi-fire) that are a reminder that debounce semantics are never as simple as the ticket title.
Structured comments
Comments are not free text first. When an operator adds a comment, the form keys off the current Response Status and asks a specific question. For an alert In Progress, the categories include:
- In progress — which action are you working on?
- Not started due to an upset in my unit — what happened?
- Not started due to an upset in another unit — which unit, and what were you asked to do instead?
- Not started due to incorrect or unclear guidance — what was conflicting?
- Not started due to alternate verbal guidance from a shift manager, supervisor, or asset team — what was requested and why?
- Not started due to a constraint not listed in the recommended actions — list the missing constraints.
- Not comfortable with the actions listed — which ones, and why?
Threading is one level deep by design. The reason this matters beyond tidiness: it turns non-compliance into labeled data about guidance quality. “Operators ignored the recommendation” becomes “operators flagged a constraint our rule authors did not know about,” which is exactly the feedback a recommendation system needs and almost never gets.
Notification model
I specified an event-driven, idempotent notification model with a noise budget—one notification per meaningful event, templated by event type, with in-app and email channels and bounded retention. What shipped was narrower, and I think correctly so:
| Event | Email? |
|---|---|
| Alert instance opens | Yes — exactly once, to configured recipients |
| Instance closes, then reopens later | Yes — a new Open is a new event |
| Response Status change | No |
| Comment added | No |
| Instance closes | No |
| Configuration published / archived | No |
Email is the only channel. The in-app bell, feed, and read state I had written up never shipped, and for a long time our own documentation still advertised them. When I wrote the rebuild spec in 2026 I corrected that in writing—“email only; do not build or advertise a channel that does not exist”—because a notification toggle that does nothing is worse than no toggle. The one rule we protected through every backlog and backfill incident: no duplicate email for an Open that already notified.
Sensor Health
The alerting program assumed a harder problem than rules alone: if inputs lie, the loop lies. Before the first deployment I wrote and socialized a product note on sensor data quality so engineering, data, and design could agree on what “trustworthy detection” means. Production made it urgent: within weeks of going live, historian anomalies—continuous zeros, spikes, frozen values, missing rows, explicit BAD quality flags—produced alert floods and false positives, and operators had no way to tell a real process event from a bad tag.
Problem framing. Treating sensor streams as reliable creates two symmetric failures: false alerts on corrupt or stale values (operators burn trust chasing ghosts), and missed events when the stack looks healthy but is blind. A third is stale evaluation: rules that keep deciding on outdated samples as if nothing were wrong with the feed.
What shipped (Sensor Health MVP). A Data Quality Service runs upstream of rule evaluation on the tags used by published alerts, over 5-minute aggregates in a one-hour cleaning window, with five detectors: Spike, Missing Data (including forward-filled values), Stagnant, Continuous Zero, and Bad Value. Results surface as a row indicator on Open Alerts (“sensor data quality issue detected”) and a per-tag panel on alert detail showing unhealthy vs normal tags with their issue descriptors.
The deliberate limit: labels only. The MVP does not suppress, pause, or modify alert firing. I had sketched the stronger pattern—process alerts enter a suspended state while a dedicated sensor alert carries the “instrumentation broke” narrative—and chose not to ship it first. Suppression built on detectors that had not yet earned operator trust risks the exact failure the feature exists to prevent: silence that looks like health. Labels let operators calibrate the detectors in the open; suspension is the next slice once the false-positive rate of the detectors themselves is known.
When the pipeline lied
About four months after go-live, during the customer’s highest-stakes season, monitoring degraded for roughly three weeks before anyone had a complete picture. It was not a single bug. An upstream data-stream interruption left gaps in the time series; ML inference requests failed on the missing windows and retried aggressively (on the order of 100+ retries per message across three services); single-threaded consumers fell behind; consumer lag grew instead of shrinking; and backfill of the gap competed for I/O on a billions-row time-series table with the live writes needed to catch up. Historian tags ended up more than two days behind. High-impact rules kept “evaluating,” just on stale data. A real plant event went undetected.
Three product calls from that period that I would make again:
- Current state over backlog. I aligned engineering and the delivery lead on restoring real-time evaluation first and fixing history second, with explicit client communication about the gap. Replaying the full stale backlog was not practical and would have delayed detection further.
- Define “green” as five layers, not “fixes deployed.” Raw ingest current; time series written for current steps; historian lag within freshness budget; inference consumer lag decreasing with dead-letter loops stopped; high-impact rules evaluating on current data. Each had an owner.
- Cadence is a product requirement. Parts of the platform assumed 15-minute windows; this site ran at 5. When engineering proposed standardizing on 15 to reduce load, the customer rejected it—critical events at their plant can develop inside a 15-minute window—and we agreed on 10. That conversation belonged to product, not infrastructure.
The outage also ended the debate about incremental repair. Within days I wrote an Alerting-only functional specification as the contract for rebuilding the slice from the data pipeline up. Two things in that spec I am proud of. Every requirement carries a class: PRESERVE (current product truth; the rebuild must keep this behavior and UX) or PM-DIRECTED (an explicit product decision that corrects an assumption the old system baked in). And reliability is specified as outcomes, not mechanisms—retries, dead-letter queues, and queue topology belong to the architect:
| Outcome the product requires | |
|---|---|
| R1 | No silent loss. If a window, tag, or rule should have been evaluated and was not, the failure is detectable. |
| R2 | Freshness honesty. When evaluation is behind wall-clock time, the product does not present stale alert state as current without indication. |
| R3 | Site-configurable cadence. Aggregation and evaluation period is a site setting, not a platform constant. |
| R4 | Current over backlog. When live evaluation and catch-up contend for capacity, live wins; backlog must not starve new Opens and Closes. |
The spec closes with nine acceptance scenarios engineering can demo, from the happy path (calculated tag → rule → one email → response lifecycle → system close) to backfill correctness, shared-rule blast radius, and an overnight handoff with no user session open.
Hypothetical scenario (simulated)
The following is a composite, anonymized illustration inspired by real operating patterns (e.g. district cooling or process plants). It is not a literal transcript of a single customer event.
A rule encodes a breach: supply temperature on a critical loop drifts more than an agreed band from setpoint. The condition holds past the configured trigger delay, so the Alert Service opens an instance at High urgency with the affected unit and asset type attached and an expected response time stamped from the configuration. The Sensor Health panel shows the trigger tag as normal, so this is a process event, not an instrument.
Notify: the instance appears at the top of the Open Alerts list and lights its unit red on the process diagram. The configured recipients—including the overnight engineer—receive one email. No second email while it stays open.
Resolve: the shift lead moves the status to Acknowledged, then In Progress, and the structured comment records which recommended action they are on. The Root Cause chart shows the tag crossing the dashed threshold and the shaded open band beginning. When the loop is back within limits, the system closes the instance. If the lead had marked Completed and the alert stayed Open, the morning supervisor would see that at 08:40 as an escalation, with the full thread attached—without re-asking anyone what happened.
What shipped vs what I specified
Credibility comes from drawing this line yourself. Everything in the first list runs in production; the second list is work I specified that did not ship, with the reason.
Shipped
- Reusable Rules with nested expression builder, symbolic-vs-expression toggle, usage count, and in-use delete guard.
- Alert Configurations with metadata, triggers, ordered actions, recipients; Draft → Published → Archived; edits apply to future instances only; trigger delay; duplicate-configuration.
- System-owned Open / Closed instances and operator-owned Response Status, with assignee as last updater.
- Structured, status-keyed comments with one-level threading.
- Live Operations overview, urgency-colored process diagram, Open Alerts and All Published Alerts tables with overdue, urgency, status, asset, and location filters.
- Alert detail with triggers, actions, comments, Root Cause chart, Alert Version History, and Sensor Health panel.
- Email notification: one per new Open.
- Sensor Health MVP: five detectors upstream of rule evaluation; labels on list and detail.
- Rules over calculated tags and ML-derived tags (anomaly, forecast) through the same loop.
- Alerting-only functional specification for the pipeline-first rebuild, with requirement classes, reliability outcomes, and acceptance scenarios.
Specified, not shipped
| Item | Why it did not ship |
|---|---|
| In-app notification channel (bell, feed, read state) | Never built. Corrected in the rebuild spec to email-only so the product does not advertise a channel that does not exist. |
| SMS | Out of scope once email proved sufficient for off-shift accountability at the live site. |
| Dedicated assignee management and automatic escalation | Cut for v1. Last-updater semantics plus the Completed-but-Open pattern covered the audit need; manual escalation via comments. |
| Work-order integration on the alerts dashboard | A read-only CMMS work-order view shipped behind a feature flag; alert-to-work-order linking and work-order creation were explicitly not done. Placeholder left visible. |
| Estimated preventive-maintenance benefit | Placeholder. No defensible model for the number at the live site; I preferred an honest blank. |
| Sensor Health suspension of process alerts | Deliberately deferred until detector precision is known (see Sensor Health). |
| Zone-level alert scoping | Site / Unit covered every production use case; deferred. |
| LLM-generated root-cause narratives | Explicitly not doing. The Root Cause chart answers “why now” without a model that could be wrong in a control room. |
Decisions and cuts
| Decision | Why |
|---|---|
| System owns Open/Closed; operator owns Response Status | Keeps alert truth and human workflow independently auditable, and makes Completed-but-Open a free escalation and rule-quality signal. |
| Rules as shared primitives with blast-radius guardrails | Reuse reduces authoring error across dozens of alerts; the usage count and delete guard make the cost of reuse visible. |
| Editing a published alert affects future instances only | An open instance must be explainable under the configuration it fired with. |
| One email per new Open, nothing else | A noise budget you can state in one sentence is one operators will trust. Re-open after Close is a new event. |
| Sensor Health as labels, not suppression | Suppression on unproven detectors reproduces the silence-looks-like-health failure. Labels first; suspension when precision is known. |
| Current evaluation over backlog catch-up | During the outage, replaying history would have delayed detection of new events. Fix history second, with the customer informed. |
| Cadence is a site setting | A platform default of 15 minutes was wrong for a plant where events develop faster. Negotiated 10 with the customer and made it a requirement. |
| Reliability specified as outcomes R1–R4, not mechanisms | Product owns “no silent loss” and “freshness honesty”; the architect owns retries, dead-letter queues, and topology. |
| Process diagram requires onboarding | Accepted a manual digitization step to get a spatial view live; self-serve authoring is a later investment. |
Why this matters for AI
Every ambitious feature—recommendations, copilots, agents—eventually asks the same question: what happens after the system speaks? If the answer is “someone reads a paragraph and hopes,” you do not have operations. You have a workshop.
Three concrete reasons this loop is the substrate for the AI work that followed it:
- One event substrate. Multivariate anomaly scores, forecast tags, and virtual sensors enter Alerting as tags. A model-driven event and a threshold-driven event get the same instance, the same response workflow, the same audit. There is no separate “AI alerts” product to reconcile.
- Labeled feedback on recommendations. The structured-comment taxonomy records why an operator did not take the recommended action. That is the training signal a recommendation system needs and almost never gets from free text.
- State lives in the system, not in a conversation. The north-star scenario for the conversational work I led next: a rule fires at 02:15 with no chat session open; the overnight operator acknowledges, acts, and comments; at 08:40 the supervisor sees current state without re-asking. A copilot can summarize that thread. It cannot be the thread.
If you are framing this for leadership, emphasize risk reduction and cycle time. If you are framing it for practitioners, emphasize one place to see accountability—not another feed that substitutes motion for progress.
In production
Alerting runs in production for a district-cooling operator in the Gulf as the detection layer of an asset-performance deployment: 13 centrifugal chillers on the live plant, roughly 1,300 historian tags and 63,000+ CMMS records ingested, 30+ users trained, and a functional design of 40 use cases (18 high-impact) signed off before go-live in early 2026. Site acceptance followed in the spring; by mid-year the feature phase was closed and the engagement moved to support. The acceptance bar we agreed to—detection accuracy, false-alarm rate, and uptime targets—is the same bar the rebuild specification is written against.
The same stack is what current engagements see in demos—where a credible detect → notify → resolve story is often the difference between “interesting model” and “something we can run.”
What I got wrong
- I let the docs advertise a channel we never built. The in-app notification design lived in requirements and user-facing documentation long after it was clear only email would ship. I fixed it in the rebuild spec; I should have fixed it the sprint we descoped it.
- I treated sensor reliability as a planning note, not a launch gate. The data-quality thesis was written before go-live; the detectors shipped after the first floods. Sensor Health should have been in the first production release.
- I assumed cadence was an engineering default. It was a customer-visible product requirement, and we found out in production.
- We alerted on the plant but not on the pipeline. There was no consumer-lag or freshness alarm for the system that produces alarms. Silence looked like health for weeks. “No silent loss” is now requirement R1.
Closing
I care about this loop because it is the substrate everything else plugs into. Copilots can suggest; agents can propose. But if the organization cannot resolve, the smartest model in the world still decays into theater. The PM work here is unglamorous and essential: make the boring parts real, so the innovative parts can be trusted.