Skip to main content
Back to Guides

I Designed a System That Fixes Itself — Here’s How It Works

2026-01-304 min read

I Designed a System That Fixes Itself — Here’s How It Works is an expert-authored guide on https://ancel.co.ke/guides. Architecting autonomous infrastructure reliability without blind automation: Zero-Knowledge policy isolation, pull-based multi-cluster executor agents, Open Policy Agent authorization and Shadow Mode verification. Lessons from Aegis. It answers how teams should think about: When production infrastructure encounters anomalies—such as memory leaks, stuck worker queues, or pod crashes—engineers are t…

Part of topic: Zero-Trust & Auditable Infrastructure, Full-Stack Systems Design

NestJS
FastAPI
Next.js
PostgreSQL
Redis
Kubernetes
Docker
Prometheus
Grafana
OPA
OIDC
View full tech stack

What problem does this guide address?

When production infrastructure encounters anomalies—such as memory leaks, stuck worker queues, or pod crashes—engineers are typically alerted after user-facing degradation has already occurred. However, naive automated self-healing scripts introduce severe operational hazards: an AI or script executing destructive remediation commands directly against production can trigger cascading outages, restart loops, or data loss if it misdiagnoses the root cause.

How is the system architected?

Aegis decouples reasoning from execution through a Zero-Knowledge, three-tier architecture: the Brain, the Will, and the Law. The Brain (FastAPI AI engine) ingests OpenTelemetry metrics and logs to perform anomaly inference, outputting structured recommendations with confidence scores. The Brain has zero access to cluster credentials. The Will (NestJS control plane) receives recommendations, checks system health, and consults the Law (Open Policy Agent). OPA enforces default-deny governance policies. If authorized, the Will enqueues an execution task into Redis. Rather than pushing commands to remote clusters, cluster-local Executor agents securely pull tasks via HTTP polling with cryptographic HMAC signatures and replay protection.

What outcomes can you measure?

A resilient, policy-gated infrastructure guardrail platform. Prevents unaudited autonomous execution by requiring confidence thresholds and OPA policy evaluation. Introduces Shadow Mode to log theoretical remediation decisions against live telemetry for verification before enabling automated task queuing.

Deep dive

The Peril of Blind Automated Remediation

Modern cloud-native systems generate massive volumes of telemetry. Prometheus collects hundreds of thousands of time-series data points, and centralized logging tools capture gigabytes of text. When an incident strikes at 2:00 AM, the mean time to recovery (MTTR) is often dominated by human diagnosis time.

This reality has spurred interest in AI-assisted remediation. But connecting an LLM or heuristic script directly to Kubernetes API credentials with instructions to "fix the problem" is a recipe for catastrophic failure:

* Cascading Restarts: An AI observing high latency might decide to restart a database pod, destroying cached connection pools and taking down the entire platform. * Flapping Services: A flapping network route might trigger repeated rolling deployments that exhaust cluster compute resources. * Security Exposure: Granting centralized automation services root credentials across multiple remote clusters creates an enormous blast radius if the orchestrator is compromised.

Aegis was designed to solve this dilemma: how to leverage automated intelligence for rapid remediation while mathematically preventing unauthorized or dangerous operations.

The Tripartite Architecture: Brain, Will, and Law

To prevent uncontrolled actions, Aegis splits responsibilities into three distinct, mutually isolated architectural components:

``` +-------------------+ Telemetry Analysis +-------------------+ | BRAIN | ---------------------------> | WILL | | (FastAPI Inference) | | (NestJS Core API) | +-------------------+ +-------------------+ | Evaluate Policy v +-------------------+ Pull-Based Task Queue +-------------------+ | EXECUTOR | <=========================== | LAW | | (In-Cluster Pod) | (HMAC Signed Tasks) | (OPA Default-Deny)| +-------------------+ +-------------------+ ```

### 1. The Brain (Inference Only) Built with Python and FastAPI, the Brain is specialized for statistical analysis, metric anomaly detection, and LLM-assisted root-cause hypothesis generation. Crucially, the Brain possesses zero execution credentials. It cannot speak to Kubernetes, SSH into servers, or restart containers. It only generates an advisory diagnostic object:

```json { "anomaly_id": "anom_98234", "recommended_action": "restart_stuck_worker", "target_service": "invoice-processor", "confidence_score": 0.94, "rationale": "Memory consumption flatlined at 98% with 0 active job completions in 15 minutes." } ```

### 2. The Law (Open Policy Agent) Before the control plane can even consider acting on an inference, it must submit the proposed action to Open Policy Agent (OPA). OPA evaluates declarative Rego policies using a strict fail-closed (default-deny) model.

Policies enforce inviolable operational guardrails: * *Never restart a stateful database node automatically.* * *Never perform more than one remediation per namespace in a 30-minute window.* * *Require confidence score $\ge 0.90$ for autonomous queueing; otherwise, route to on-call engineer for manual approval.* * *During business peak hours (08:00–17:00 EAT), require manual multi-factor authorization for all mutations.*

```rego package aegis.policy

default allow = false

allow { input.action == "restart_stuck_worker" input.confidence >= 0.90 not cooldown_active(input.namespace) not is_protected_service(input.target_service) } ```

### 3. The Will (Task Orchestrator) The NestJS central API acts as the authoritative coordinator. If OPA approves the action, the Will does not push commands into the remote cluster. Pushing requires opening inbound firewall ports on the target infrastructure, which violates zero-trust principles.

Instead, the Will writes an encrypted, signed task document into PostgreSQL and Redis, tagged with the destination `cluster_id`.

Pull-Based Execution with Cryptographic Verification

Within each Kubernetes cluster, an Aegis Executor daemon runs with minimal, strictly scoped RBAC permissions limited to its local namespace.

The Executor communicates strictly outbound over HTTPS, polling the Will for pending assignments: `GET /api/v1/executor/tasks?cluster_id=prod-eu-west-1`

To prevent man-in-the-middle tampering and replay attacks: 1. Every task payload contains a timestamp, a unique nonce, and an HMAC-SHA256 signature generated using a pre-shared cluster secret. 2. The Executor rejects any task whose timestamp deviates by more than 5 seconds from system time. 3. Upon completing the bounded operation (e.g., executing a rolling restart on a specific deployment), the Executor posts the execution logs and exit code back to the central API.

```typescript function verifyTaskSignature(task: ExecutorTask, sharedSecret: string): boolean { const message = `${task.id}|${task.clusterId}|${task.action}|${task.createdAt}`; const expectedSig = crypto .createHmac("sha256", sharedSecret) .update(message) .digest("hex"); return crypto.timingSafeEqual( Buffer.from(task.signature), Buffer.from(expectedSig) ); } ```

Shadow Mode: Building Trust Before Automation

No sane infrastructure team enables autonomous remediation on day one. Aegis solves this adoption barrier through Shadow Mode.

In Shadow Mode: * The Brain continuously monitors telemetry and generates recommendations. * The Law evaluates policies against the proposed actions. * The Will records the full audit trail into the database as a "Simulated Remediation" without actually enqueueing the task for the Executor. * Engineers inspect dashboard metrics to verify: *"Did the AI accurately identify the issue? Would the policy have allowed a safe fix?"*

Only after teams observe weeks of accurate, benign simulated actions in Shadow Mode do they toggle individual namespaces into Autonomous Mode. Reliability engineering requires earned trust through transparent, auditable verification.

Frequently asked questions

What engineering problem does this guide tackle?
When production infrastructure encounters anomalies—such as memory leaks, stuck worker queues, or pod crashes—engineers are typically alerted after user-facing degradation has already occurred. However, naive automated self-healing scripts introduce severe operational hazards: an AI or script executing destructive remediation commands directly against production can trigger cascading outages, restart loops, or data loss if it misdiagnoses the root cause.
How is the system architected?
Aegis decouples reasoning from execution through a Zero-Knowledge, three-tier architecture: the Brain, the Will, and the Law. The Brain (FastAPI AI engine) ingests OpenTelemetry metrics and logs to perform anomaly inference, outputting structured recommendations with confidence scores. The Brain has zero access to cluster credentials. The Will (NestJS control plane) receives recommendations, checks system health, and consults the Law (Open Pol…
What measurable outcomes can you expect?
A resilient, policy-gated infrastructure guardrail platform. Prevents unaudited autonomous execution by requiring confidence thresholds and OPA policy evaluation. Introduces Shadow Mode to log theoretical remediation decisions against live telemetry for verification before enabling automated task queuing.