Events and monitoring

When fifty agents run on ten machines, fifty sets of logs don’t tell you who is doing what. Everything agents do through runstate lands in one place, and you can read it three ways: the console for people, events for code that reacts to changes, and status for a point-in-time picture of a run.

The console

The console at app.getrunstate.com is organized by space:

Page	What it shows
Tasks	Every queue job with its state, filterable by state and key. A job’s page shows its result and each attempt: which worker took it, when, and with which resources. You can cancel a job from there.
Resources	Limits, slots and budgets with their current levels, plus callers waiting for them.
Groups	gather groups, their inputs, which results were accepted, and the decision.
Activity	The event stream as it happens.
Usage	Operations and retained data over time.
API keys	Create and revoke keys and choose their permissions.

Worker labels appear throughout the console. They default to <hostname>-<pid>; set worker to a stable name such as the pod name to make them easier to read (see Configuration & auth).

Events

Every change in a run is written to an ordered event stream: jobs created, started, succeeded and failed; budget reservations and charges; groups decided; the run stopping. Each event has a type and data with the details.

Common types:

Type	When
task.created	A queue job was submitted.
task.succeeded	A job recorded its result.
task.failed	A job used up its attempts.
task.cancelled	A job was cancelled, for example by stop() or a finished gather.
budget.reserved, budget.settled	A call reserved budget, and was charged.
budget.overrun	A call cost more than its reservation and the budget couldn’t cover the difference.
group.finalized	A gather group was decided.
timer.fired	A sleep finished.

Stream a run’s events

TypeScript

import runstate from 'runstate-sdk';

const controller = new AbortController();
process.on('SIGINT', () => controller.abort());

for await (const event of runstate.events({
  types: ['task.succeeded', 'task.failed', 'budget.overrun'],
  signal: controller.signal,
})) {
  console.log(event.type, event.data);
}

Python

import runstate

for event in runstate.events(types=["task.succeeded", "task.failed", "budget.overrun"]):
    print(event.type, event.data)

# in async code
async for event in runstate.aevents(types=["task.failed"]):
    ...

How the stream behaves:

• It covers the current run. Use a client from runstate.run(name) to watch a specific run.
• In order, at least once. Events arrive in order. After a network error the SDK reconnects and resumes where it left off, so you may see an event again. Make your handling safe to repeat.
• It runs until you stop it. Abort the signal (TypeScript) or break out of the loop.
• Leave out types to receive every event.

Status

status() returns a snapshot of the run: its state and deadline, the workers currently holding work, and what is waiting.

TypeScript

const { run, workers, waiting } = await runstate.status();
console.log(run, `${workers.length} workers busy`, `${waiting.length} waiting`);

Python

status = runstate.status()  # await runstate.astatus()
print(status.run, len(status.workers), "workers busy", len(status.waiting), "waiting")

Status is an observation for people and tooling, not a source of truth. Use it to understand a stuck run, for example which worker holds a slot or what a job is waiting for.

Webhooks

To push events to a server instead of holding a connection, register an HTTPS endpoint. The SDKs don’t wrap this, so call the HTTP API directly:

curl -X POST "https://api.getrunstate.com/v1/spaces/$RUNSTATE_SPACE_ID/destinations" \
  -H "Authorization: Bearer $RUNSTATE_API_KEY" \
  -H "Idempotency-Key: $(uuidgen)" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://hooks.example.com/runstate", "eventTypes": ["task.failed", "group.finalized"]}'

The response includes a signingSecret, returned only once. Each delivery is a JSON POST of the event with these headers:

Header	Value
x-runstate-event-id	The event id. Use it to deduplicate.
x-runstate-timestamp	Unix time in seconds when the request was signed.
x-runstate-signature	v1= followed by the hex HMAC-SHA256 of <timestamp>.<raw body>, keyed with the signing secret.

Every delivery also carries the same three values as x-kernel-event-id, x-kernel-timestamp and x-kernel-signature, so older receivers keep working. New receivers should read the x-runstate-* headers.

Verify the signature against the raw request body before trusting the payload, and reject old timestamps. Any 2xx response counts as delivered. GET /v1/spaces/{spaceId}/deliveries lists recent delivery attempts and their results.

Related

• Troubleshooting: common symptoms and where to look.
• API reference: Events and webhooks.