AI ENGINEERING
IN DEVELOPMENT
Multi-agent AI SRE
An incident-response system that reads alerts, diagnoses them with an LLM against past incidents, and either fixes the server itself or asks a human on Telegram — decided by a confidence gate rather than by hope.
- Python
- FastAPI
- PostgreSQL
- pgvector
- Redis
- Auto-execute confidence gate
- 0.90
- Rate limit per service
- 3/hour
- Unreachable detection
- 90s
Auto-execute confidence gate
Rate limit per service
Unreachable detection
The constraint
Most alerting tells you something is wrong and stops there. Most auto-remediation acts with no idea whether it should. The interesting problem is in between: let a model diagnose an incident, then constrain what it is permitted to do about it, so that being confidently wrong is survivable.
How an alert moves
ALERT PATH
A psutil agent on each managed server posting metrics every 30 seconds to a token-authenticated ingress.
- DISCOVERY
- SSH scanning enumerates systemd services, Docker containers, Kubernetes pods and open ports, so the system knows what it is responsible for rather than being told.
- INCIDENT MEMORY
- Past incidents are embedded and stored in pgvector, and similarity search puts the closest ones in front of the model before it diagnoses a new alert.
- PREDICTION
- Hourly trend analysis flags risk windows and scaling suggestions ahead of the threshold being crossed, rather than reacting once it has.
- HUMAN IN THE LOOP
- A Telegram bot carries status, scans, incidents and predictions, with inline approve and deny buttons and natural-language chat for anything the commands do not cover.
- PROVIDER-AGNOSTIC
- Ollama, OpenAI, Anthropic and OpenRouter sit behind one provider interface, so the model is a configuration choice rather than an architectural commitment.
- OBSERVABILITY
- Prometheus metrics, Alertmanager webhooks, Grafana dashboards and Loki logs — the system that watches servers is itself watched.
The decision gate
The hard part of automated remediation is not deciding what to do. It is deciding when the system is allowed to act without a human, and making that answer conservative by construction rather than by policy.
- WHITELISTED ONLY
- Actions come from a fixed list — restart a service, container or pod, scale a deployment, clear a cache. Schema changes, data deletion, firewall edits and config rewrites are matched as forbidden patterns and blocked outright.
- CONFIDENCE ≥ 0.90
- Below the threshold the action does not execute; it becomes a Telegram approval request instead. The number is a constant in the executor, not a prompt instruction the model can talk itself past.
- THREE PER SERVICE PER HOUR
- A Redis counter caps automated actions per service. It is what stops a confident-but-wrong diagnosis from restarting the same service in a loop.
- TIMEOUTS ESCALATE, THEN DECIDE
- An unanswered approval is re-pinged at five minutes and auto-executes at fifteen — but only if confidence is at or above 0.85. Silence resolves toward acting, and only for decisions that were nearly good enough to act on alone.