Troubleshooting

Start with the error code (every SDK error has one) and the error table. For a stuck run, rs.diagnostics.run(runId) and the console’s Resources and Tasks pages show who holds what and what is waiting.

Connecting

Calls fail with UnavailableError after about 30 seconds

The SDK can’t reach the API. Most often RUNSTATE_BASE_URL isn’t set, so the client is trying http://localhost:8080. Set it to https://api.getrunstate.com. Each attempt times out after requestTimeoutMs (10 s) and is retried twice, which is where the 30 seconds come from.

ConfigError: missing apiKey or missing spaceId

RUNSTATE_API_KEY or RUNSTATE_SPACE_ID isn’t set in the environment of the process that constructs the client, and wasn’t passed as an option.

FORBIDDEN on some calls but not others

The key lacks a permission; the error message names it (credential lacks resource_config). Creating queues, pools, quotas, budgets, barriers, work limits or webhook destinations, and changing space limits, need resource_config. Diagnostics, usage and limits reads need usage_read. Runtime calls (runs, claims, messages, tasks, taking quota and pool units, reserving budget) need coordination_write. See Configuration & auth. FORBIDDEN without that message means the key isn’t allowed for this space.

RATE_LIMITED with only a few workers

A space allows its plan’s requests per minute (600 on Free) unless its requestsPerMinute was set lower. Current SDKs long-poll in consume() (waitMs, default 20 s), so an idle worker makes about three requests a minute and its empty waits, like lease renewals, don’t count toward the rate. SDK versions before 0.2.0 poll every pollMs (250 ms), up to 240 requests a minute per idle worker, and those empty polls do count: upgrade the SDK, or raise pollMs. If you call receive() or admit() in your own loop, pass waitMs / wait_ms instead of sleeping between calls (request rate).

Workers don’t get work

The worker runs but never receives tasks

• Different run. Workers only receive tasks submitted under the exact run they attach to (not its parent or children). Check that rs.scope(runId) uses the id the producer submitted to.
• Task group members are delivered on the group’s own run: use (await group.status()).scopeId.
• Different queue name. Queue names are per space and case-sensitive.
• Admission. Tasks submitted with requirements or workLimit are still visible to receive()/consume(), which bypass admission. If you use admission, workers should call admit().

NotFoundError: mailbox "work" not found

The queue doesn’t exist in this space yet. Call rs.mailboxes.ensure(name) before submitting, consuming or creating a task group on it. The same applies to pools, quotas, budgets and work limits.

A task stays PENDING

• No worker is consuming that queue under that run.
• With admission: a required pool or quota is short, the work limit is full, or a quota has a cooldown. admit() returns null until everything is available; check Resources in the console.
• The run was cancelled or its deadline passed. rs.scope(runId).status() shows effective: 'INACTIVE'.

Leases and ownership

LeaseLostError, or a lease’s signal aborts, during long work

The SDK couldn’t renew in time, so it gave the lease up. Common causes:

• The event loop was blocked. Renewal runs on timers in your process. CPU-heavy synchronous work in Node.js, or blocking calls inside an async def in Python, stop renewals. Move heavy work off the event loop (worker threads, asyncio.to_thread) or use a longer leaseSeconds.
• The lease is too short for your network or pauses. Try 60 to 120 seconds for long model calls.
• The run was cancelled. Renewals are refused after cancellation, so every held lease in the run is lost on its next renewal. That is the intended stop signal.

CLAIM_HELD but nobody is working on the key

The previous owner probably crashed and its lease hasn’t expired yet; it will within leaseSeconds of its last renewal. rs.diagnostics.run(runId) lists holders and their earliest lease expiry, and expiredRecent shows leases that just expired. Use wait: true to queue behind the owner instead of failing.

A pool stays full after agents finish

• Leases from crashed agents return only when they expire.
• Pool units taken through admit() return when the delivery is settled (complete(), retry(), reject()) or its lease runs out. If units stay held, a worker is holding an unsettled delivery, or a cancelled task’s delivery is still inside its lease. See admission.
• The organization may be at its “agents working at once” limit (CONCURRENCY_LIMITED); acquisitions wait through it.

Tasks and results

The task’s attempt keeps increasing

The work is being retried. Either the handler throws (check onError / on_error), or the work outlasts its lease: receive() and admit() don’t renew, so a delivery held past leaseSeconds is handed to another worker. After 5 attempts the task becomes FAILED.

IDEMPOTENCY_CONFLICT when submitting

A task with that key already exists in the run with different input. Inputs are compared by content fingerprint, so a timestamp or random id inside the input makes every retry “different”. Keep volatile values out of the input, or use a different key.

submit() returns an old result instead of doing the work again

Task keys are permanent within a run: submitting an existing key with the same input returns the existing task, even if it finished long ago. Use a new key or a new run to redo work.

result() throws WaitTimeoutError

Only your wait ended. The task is still running on the server. Call result() again later, or re-attach with rs.scope(runId).task(taskId).

A worker keeps polling after the run’s deadline

After a deadline passes, requests fail with DEADLINE_EXCEEDED, which consume() treats as a temporary error and keeps polling through. Stop the worker with its signal / stop_event when your run ends, or cancel the run (cancellation stops consume()).

Capacity and completion

quota.take() waits much longer than the window

• A cooldown is active: check cooldownUntil in quota.status().
• The queue is strict FIFO, so a waiter asking for many units holds back smaller requests behind it.
• Waiters left by crashed processes stay queued until their time-to-live and can be granted units first.

BACKLOG_FULL when sending or submitting

The space has more queued and in-progress messages than its maxBacklog (1,000 by default). Add workers, or raise maxBacklog (per-space limits).

An N_ACCEPTED group never finalizes

A group stays open until you call group.close(), expectedMembers members have joined, or its deadline passes. Only a closed group can fail with threshold_unreachable, so a group that is still open and has too few members that can succeed waits. Call close() once you’ve added the members, or create the group with expectedMembers or a deadline. See close the group.

Python

RuntimeWarning: coroutine 'Delivery.complete' was never awaited

The handler is a plain def, but delivery methods are coroutines. Make the handler async def and await delivery.complete(...). This applies to the blocking Runstate client too.

The blocking Runstate client hangs inside a handler

An async def handler runs on the blocking client’s own event loop, so calling a blocking method from it waits on itself. Inside async def handlers, await the async objects you were given; call blocking methods only from plain def handlers or outside handlers. See the blocking client.

Webhooks

Signatures don’t verify

Compute the HMAC over the raw request body exactly as received, prefixed by the x-runstate-timestamp value and a dot, using the signing secret from destination creation. Re-serializing parsed JSON changes the bytes. See Webhooks.