# Events and monitoring

> See the swarm as one system with the console, the event journal, reconnectable watch streams, webhooks and read-only diagnostics.

Source: https://docs.getrunstate.com/guides/observability/

When fifty agents run on ten machines, fifty sets of logs don't tell you who owns what. Everything agents do through runstate lands in one place, and you can read it three ways: the **console** for people, **events** for code that reacts to changes, and **diagnostics** for a point-in-time picture of a run.

## The console

The console at [app.getrunstate.com](https://app.getrunstate.com) is organized by space:

| Page | What it shows |
| --- | --- |
| **Tasks** | Every task with its state, filterable by state and key prefix. A task's page shows its outcome, subscribers and each admission attempt (who took it, when, with which resources). You can cancel a task or detach a subscriber from there. |
| **Resources** | Concurrency pools, shared quotas and budgets with their current levels, plus agents that are waiting for capacity. Quotas can be paused (a cooldown) from here. |
| **Groups** | Task groups, their conditions, member verdicts and decisions. |
| **Activity** | The event journal as it happens. |
| **Usage** | Operations and retained data over time. |
| **API keys** | Create and revoke keys and choose their permissions. |

Holder names appear throughout the console, so give workers stable, meaningful holder names (see [Configuration & auth](https://docs.getrunstate.com/configuration/#worker-identity)).

## Events

Every state change is written to an ordered event journal: tasks created, admitted, succeeded and failed; quota units consumed and granted; budget reservations settled; groups finalized; runs cancelled. Each event has:

| Field | Meaning |
| --- | --- |
| `eventId` | Position in the journal. Increases monotonically; use it to deduplicate. |
| `type` | For example `task.created`, `task.admitted`, `task.succeeded`, `task.failed`, `task.cancelled`, `task.expired`, `task.joined`, `message.ready`, `allowance.consumed`, `allowance.cooldown`, `budget.reserved`, `budget.settled`, `budget.voided`, `group.finalized`, `scope.cancelled`, `timer.fired`, `barrier.released`. |
| `aggregateType`, `aggregateId` | The resource the event is about, such as `task` and the task id. |
| `aggregateVersion` | That resource's version after the change. |
| `payload` | Event details; task events include the `scopeId` of the task's run. |
| `createdAt` | Timestamp. |

### Watch a run live

`watch()` is a long-poll stream you can resume. Filter by run (which includes its child runs), event types or one resource.

**TypeScript**

```ts
const controller = new AbortController();
process.on('SIGINT', () => controller.abort());

for await (const event of rs.events.watch({
  scopeId: run.id,
  types: ['task.succeeded', 'task.failed', 'group.finalized'],
  waitSeconds: 20,
  cursor: loadCursor(), // undefined on the first run
  signal: controller.signal,
})) {
  console.log(event.eventId, event.type, event.aggregateId, event.payload);
  saveCursor(event.cursor);
}
```

**Python**

```python
async for event in rs.events.watch(
    scope_id=run.id,
    types=["task.succeeded", "task.failed", "group.finalized"],
    wait_seconds=20,
    cursor=load_cursor(),  # None on the first run
):
    print(event["eventId"], event["type"], event["aggregateId"], event["payload"])
    save_cursor(event["cursor"])
```

Pass `stop_event=` (an `asyncio.Event`) to end the stream from elsewhere. With the blocking client, `rs.watch(...)` takes the same keyword arguments and returns a regular iterator.

How watch behaves:

- **At least once, in order.** Events arrive ordered by `eventId`. After a reconnect you may see an event again, and the SDK doesn't deduplicate for you: skip `eventId`s you've already handled.
- **You own the cursor.** Each yielded event carries a `cursor`; pass the last one back as `cursor` to resume. All events from one long-poll response share the same cursor (the position after that response), so persist it once you've handled every event that carries it.
- **Long poll.** `waitSeconds` (0 to 30) is how long the server holds a request open when there are no new events. The stream keeps polling until you abort it.
- **Recovery.** Watch is for noticing changes quickly. For the authoritative state of a task after a gap, read it with `ticket.status()`.

Filters: `types` (list of event types), `aggregate` (`{ type, id }`, for example `{ type: 'task', id: taskId }`), `scopeId` / `scope_id` (the run and its descendants), and `limit`.

For a one-off page instead of a stream, `rs.events.list({ cursor, limit })` returns `{ events, nextCursor }` (up to 500 events per page).

### Webhooks

To push events to a server instead of holding a connection, register an HTTPS endpoint. Neither SDK wraps this yet, so call the API directly:

```bash
curl -X POST "$RUNSTATE_BASE_URL/v1/spaces/$RUNSTATE_SPACE_ID/destinations" \
  -H "Authorization: Bearer $RUNSTATE_API_KEY" \
  -H "Idempotency-Key: $(uuidgen)" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://hooks.example.com/runstate", "eventTypes": ["task.failed", "group.finalized"]}'
```

The response includes a `signingSecret`, returned only once. Each delivery is a JSON `POST` of the event (`eventId`, `type`, `aggregateType`, `aggregateId`, `payload`, `createdAt`) with these headers:

| Header | Value |
| --- | --- |
| `x-runstate-event-id` | The event id. Use it to deduplicate. |
| `x-runstate-timestamp` | Unix time in seconds when the request was signed. |
| `x-runstate-signature` | `v1=` followed by the hex HMAC-SHA256 of `<timestamp>.<raw body>`, keyed with the signing secret. |

Every delivery also carries the same three values under their original names, `x-kernel-event-id`, `x-kernel-timestamp` and `x-kernel-signature`, so receivers written against those keep working. New receivers should read the `x-runstate-*` headers.

Verify the signature against the raw request body before trusting the payload, and reject old timestamps. Any `2xx` response counts as delivered. `GET /v1/spaces/{spaceId}/deliveries` lists recent delivery attempts and their results.

## Diagnostics

Diagnostics are read-only, recent-window views for people and tooling. They are observations, not a source of truth: use them to understand a stuck run, not to decide who owns something.

**TypeScript**

```ts
const view = await rs.diagnostics.run(run.id);
for (const worker of view.workers) {
  console.log(worker.holder, worker.keys, worker.earliestLeaseExpiry);
}
for (const wait of view.waiting) {
  console.log(`${wait.scopeId} waits for ${wait.units} of ${wait.permit}, blocked by ${wait.blockedBy.join(', ')}`);
}

const waits = await rs.diagnostics.waits(); // space-wide: waiters, held grants, held claims
console.log(waits.waiters.length, waits.grants.length, waits.claims.length);
```

**Python**

```python
view = await rs.diagnostics.run(run.id)
for worker in view["workers"]:
    print(worker["holder"], worker["keys"], worker["earliestLeaseExpiry"])
for wait in view["waiting"]:
    print(f"{wait['scopeId']} waits for {wait['units']} of {wait['permit']}, blocked by {wait['blockedBy']}")

waits = await rs.diagnostics.waits()  # space-wide: waiters, held grants, held claims
print(len(waits["waiters"]), len(waits["grants"]), len(waits["claims"]))
```

`diagnostics.run(runId)` covers the run and its child runs:

| Field | Contents |
| --- | --- |
| `scopes` | The runs in the subtree and their states. |
| `workers` | Holders with live leases, the keys they hold, and their earliest lease expiry. `diagnostics.workers(runId)` returns just this list. |
| `waiting` | Pool waiters and what they're blocked by. |
| `expiredRecent` | Claims and pool leases that expired recently, with the holder that lost them. A good first place to look for crash takeovers. |
| `messages` | Per work queue: messages by state and recent dead letters. |
| `deliveriesFailed` | Recent failed webhook deliveries. |
| `timeline` | Recent events for the run. |

`diagnostics.waits()` returns the space's wait-for edges (who waits for which pool, who holds grants and claims). It shows what each piece of work is waiting for; it doesn't resolve anything.

## Related

- [Troubleshooting](https://docs.getrunstate.com/troubleshooting/): common symptoms and where to look.
- API reference: [Events, watch & webhooks](https://docs.getrunstate.com/api/events/), [Diagnostics](https://docs.getrunstate.com/api/diagnostics/).