Skip to main content

Part of topic: Backend & Modular Systems, Full-Stack Systems Design

Building OpsFlow: Real-Time Incident Management and Operational Workflows

Building OpsFlow: Real-Time Incident Management and Operational Workflows

December 5, 2024 3 min read NestJS, Next.js, PostgreSQL, MongoDB, WebSockets, Incident Management, SLA

Building OpsFlow: Real-Time Incident Management and Operational Workflows is a Developer Journal article by Ancel Ajanga on https://ancel.co.ke. System design for a production-grade incident management platform: dual-database architecture with PostgreSQL and MongoDB, WebSocket event streaming, and on-demand SLA tracking. It focuses on: Engineering teams often rely on fragmented tools (spreadsheets, chat rooms, email threads) during outages, leading to lost context, delayed responses, and unrecorded postmortems. An…

System design for a production-grade incident management platform: dual-database architecture with PostgreSQL and MongoDB, WebSocket event streaming, and on-demand SLA tracking.

Who this is for

Site reliability engineers, operations leads, and full-stack developers building real-time internal tooling, incident coordination systems, and SLA monitoring.

Problem

Engineering teams often rely on fragmented tools (spreadsheets, chat rooms, email threads) during outages, leading to lost context, delayed responses, and unrecorded postmortems.

Business outcome

A unified incident response platform with role-based access control, live WebSocket incident timelines, on-demand SLA breach calculations, and structured postmortems.

Metrics

  • 50+ API endpoints with role-based access control
  • WebSocket incident timeline broadcasts
  • SLA tracking computed on demand, with no background cron dependency

When production services go down, seconds matter. Yet in many organizations, incident management is chaotic: developers coordinate in ad-hoc Slack channels, incident severity is disputed, updates are missed, and postmortems end up abandoned in Google Docs.

I designed and built OpsFlow to solve this operational breakdown. The platform combines the incident triage rigor of PagerDuty, the task coordination of Jira, and the structured documentation of Notion into a unified, lightweight system designed for high reliability.

The Problem: Fragmented Incident Coordination

During a major production outage, teams experience four recurring points of failure: 1. **Context Fragmentation**: Critical troubleshooting steps, metrics screenshots, and remediation logs get scattered across personal chat messages. 2. **Delayed Escalation**: Unclear ownership and lack of automated SLA tracking allow critical P1 incidents to sit unnoticed. 3. **Weak Auditability**: After an incident is resolved, constructing an accurate chronological timeline of actions is virtually impossible. 4. **Tool Sprawl**: Paying high per-seat licensing fees for multiple enterprise monitoring and ticketing tools strains mid-sized engineering budgets.

Architectural Approach: Dual-Database Strategy and Real-Time Event Rooms

OpsFlow was built on **NestJS** and **Next.js**, utilizing an intentional dual-database architecture:

### 1. Dual-Database Data Segregation Rather than forcing all operational data into a single paradigm, OpsFlow segregates data based on access patterns: - **PostgreSQL (Prisma ORM)**: Stores authoritative relational entities—users, organizations, teams, permissions, incident lifecycle states, and SLA policy configurations. Relational foreign keys and cascading constraints ensure administrative integrity. - **MongoDB**: Stores high-volume, append-only **Incident Timelines and Event Streams**. Every status change, comment, metric snapshot, and responder action generates an immutable document in MongoDB. This keeps PostgreSQL lean while providing high write throughput for fast-moving incident logs.

### 2. Real-Time WebSocket Incident Rooms When an incident is declared, responders join a dedicated incident dashboard. Using **Socket.IO**: - Responders join room `incident:${incidentId}`. - Live status changes, priority escalations, and timeline comments stream to all connected team members over WebSockets. - Presence indicators show who is actively viewing and managing the incident.

### 3. Dynamic, On-Demand SLA Computation Many incident platforms require heavy background worker daemons running continuous cron jobs to check for SLA breaches. OpsFlow eliminates this infrastructure overhead: - SLA windows (e.g., P1 = 15m response, 1h resolution; P2 = 30m response, 4h resolution) are evaluated dynamically during API requests and view hydrations. - By comparing the incident creation timestamp against the current server time and the configured policy, SLA breach indicators calculate on the fly with zero background compute costs.

ypescript
export function calculateSlaStatus(
  createdAt: Date,
  severity: IncidentSeverity,
  status: IncidentStatus
): SlaAssessment {
  const policy = SLA_POLICIES[severity];
  const now = new Date();
  const elapsedMinutes = (now.getTime() - createdAt.getTime()) / (1000 * 60);

  if (status === 'RESOLVED') {
    return { isBreached: false, remainingMinutes: 0, status: 'MET' };
  }

  const remaining = policy.resolutionTargetMinutes - elapsedMinutes;
  return {
    isBreached: remaining < 0,
    remainingMinutes: Math.max(0, Math.round(remaining)),
    status: remaining < 0 ? 'BREACHED' : remaining <= 15 ? 'AT_RISK' : 'HEALTHY'
  };
}

Multi-Tenant Team Isolation and Role-Based Access

OpsFlow enforces strict **Role-Based Access Control (RBAC)** across four tiers: - **Admin**: Full organization management, team creation, and SLA configuration. - **Manager**: Incident escalation, assignment, and postmortem approval. - **Engineer**: Incident creation, investigation, status transitions, and timeline updates. - **Viewer**: Read-only access to incident status pages and published postmortems.

Every database query is intercepted by an organization/team guard, ensuring that multi-tenant boundaries cannot be bypassed even if a client attempts to query an arbitrary resource ID.

Key Engineering Takeaways

1. **Fit the Database to the Data Lifecycle**: Using PostgreSQL for strict relationships and MongoDB for append-only audit streams provides better performance than forcing everything into a single database. 2. **Dynamic Computation Often Beats Background Workers**: Before spinning up background worker queues and Redis schedulers for SLA timers, consider whether on-demand mathematical evaluation meets your requirements with zero operational complexity. 3. **Postmortems are Software Artifacts**: Linking postmortems directly to the immutable incident timeline ensures organizational learning happens with complete factual context.

Explore the OpsFlow Case Study or review the [OpsFlow Project Page](/projects/opsflow).

Trade-offs

Dual-database design: PostgreSQL for relational entities (users, teams, permissions, incidents) and MongoDB for append-only audit event timelines; calculated SLA breaches dynamically on request rather than running polling daemon workers.

Challenges faced

Enforcing strict team data isolation across every query layer while broadcasting real-time updates exclusively to authorized incident room subscribers.