DevOps Agent Manual

1. Overview

The DevOps Agent runs as a persistent background service alongside your stack. It checks every target on a cron schedule, learns normal behaviour over time, and acts — or alerts you — when something deviates. Unlike one-off health checks, it accumulates context: each check informs the next alert, and incident history shapes future runbooks.

2. Target types

HTTP monitoring

HTTP targets are checked at the configured interval. The agent measures total response time (TCP + TLS + TTFB) and tracks the SSL certificate expiry date. A target is marked degraded if response time exceeds 3× the baseline p95, and down if the request fails or returns a non-2xx status.

SSL certificate warnings fire when fewer than 14 days remain. You can override this threshold via the DevOps policy field on hire.

Cloud provider health checks

Railway, Vercel, Fly, Netlify, and Render are checked via their respective REST APIs. The agent queries the latest deployment and flags down for failure conclusions (failure, timed_out, startup_failure) and degraded for in-progress or queued states that have been waiting too long.

You can add multiple targets of different types. Monitoring 2 Railway services and 1 HTTP health check counts as 3 targets toward your tier limit.

3. Alerts and downtime detection

Alert suppression

The agent fires exactly one alert per incident episode. If a target goes down and stays down across multiple check cycles, you receive one downtime email — not one per check. Alerts resume only when the target recovers and then goes down again.

The same rule applies to degradation: one alert per degradation episode, regardless of how long it lasts.

Real outage vs. transient blip

A real outage is defined as 2 or more consecutive failed checks. A single failed check is treated as a transient blip and does not trigger a postmortem (though downtime alerts still fire on the first failure).

Alert delivery

4. Baseline learning and anomaly scoring

Every time an HTTP target responds successfully, the agent records the response time. It maintains a rolling window of up to 100 samples per target and computes p50, p95, and p99 percentiles.

Once 20 samples are collected (the minimum for a statistically meaningful baseline), the agent begins anomaly scoring. A response is flagged as anomalous if it exceeds p95 × 1.5. The anomaly score is the percentage above the p95 threshold.

- API Gateway: p50=142ms, p95=380ms, p99=820ms (87 samples)
- Homepage: p50=210ms, p95=590ms, p99=1240ms (100 samples)
- Payments API: p50=88ms, p95=215ms, p99=410ms (100 samples)

Anomalies are written to the knowledge strand as informational records — they do not send emails by default. You can query the chat panel: “Show me recent anomalies” to see what was flagged.

The hourly baseline snapshot persists the current p50/p95/p99 per target to the knowledge strand so you can ask the agent to compare performance week-over-week.

5. GitHub Actions CI/CD monitoring

For github_actions targets, the agent fetches the last 100 workflow runs every 2 hours and runs two analyses:

Flaky workflow detection

A workflow is flagged as flaky if it has 3 or more failures in the last 7 days. The agent records the failure count, failure rate, and average duration for each workflow in the repo. Incidents are written to the knowledge strand with a 24-hour cooldown — you won't be flooded with the same alert every 2 hours.

Blocked build detection

A build is considered “blocked” if a workflow on main or master has been in a failed state for more than 2 hours, or has been running (in_progress) for more than 2 hours. Feature branch builds are excluded to avoid false positives.

Weekly CI report

Every Sunday at 09:00 UTC, the agent generates a markdown CI report covering the previous 7 days: workflow success rates, average build duration, deploy frequency, flaky workflows, and blocked builds. The report is stored in the knowledge strand and can be requested at any time via chat.

6. Postmortems and runbooks

Automatic postmortem generation

When a target that has been down for 2+ consecutive checks finally recovers, the agent automatically:

1. Creates an incident summary with start time, resolution time, and duration
2. Generates a 5-section SRE postmortem via LLM (incident summary, timeline, root cause, impact, action items)
3. Stores the postmortem in the knowledge strand
4. Emails the postmortem to your alert email address

Postmortem generation is asynchronous — the monitoring loop is never blocked. The email typically arrives within 30–60 seconds of recovery.

Requesting postmortems manually

Type any of the following in the chat panel to generate a postmortem for the most recent resolved incident:

postmortem
generate postmortem
incident report
write postmortem

Runbook generation

A runbook is a response playbook for a specific service — what to check first, what commands to run, what escalation paths to follow. The agent generates runbooks from patterns in your incident history.

runbook
generate runbook
runbook Payments API   ← generates runbook scoped to that target

If you have no incident history for a target yet, the agent generates a starter runbook template based on the service type.

7. Auto-remediation

Auto-remediation is available on the Pro and Studio tiers. When enabled, the agent can restart crashed services automatically without waiting for you to respond.

8. Custom tool registry

You can register any external API, webhook, or OpenAPI spec as a tool that the agent can call during conversations and incident response.

Register a single tool

{
  "name": "Restart worker pool",
  "description": "Restarts the background job worker pool on the staging cluster",
  "webhook_url": "https://api.example.com/ops/restart-workers",
  "method": "POST",
  "api_key": "sk_ops_…",
  "body_template": "{ \"env\": \"{{env}}\" }"
}

Ingest an OpenAPI spec

Pass an OpenAPI 3 spec JSON and the agent will automatically register each path+method combination as a separate tool, using the operationId as the name.

{
  "openapi_spec": { ...your OpenAPI 3 spec JSON... },
  "api_key": "sk_…"   // optional — applied to all endpoints
}

How tool dispatch works

When you send a message, the agent's system prompt is augmented with descriptions of all your registered tools. If the LLM decides a tool should be called, it emits a structured tool call in its response. The agent parses it, calls the webhook, and formats the result back to you — all in the same response.

User: Restart the worker pool on staging
Agent: Calling "Restart worker pool" with env=staging…
Result (HTTP 200): { "status": "restarted", "workers": 8 }

Template substitution

Both webhook_url and body_template support {{variable}} placeholders. The agent substitutes values from the tool call arguments at runtime.

9. Chat commands

DevOps policy (ASPR)

Set custom instructions on hire (or update them later in the dashboard). These instructions are injected into every LLM call and action check. Examples:

"Never auto-restart the payments-api service — always request approval."
"Alert on SSL expiry 30 days out, not 14."
"Ignore 502s on /health between 03:00–04:00 UTC (maintenance window)."
"When generating runbooks, include the on-call Slack handle: @ops-oncall."

10. Feature tiers