Troubleshooting

Start with the error class and its code (every SDK error has one) and the error table. For a stuck run, runstate.status() and the console’s Resources and Tasks pages show who holds what and what is waiting.

Connecting

Calls fail with UnavailableError

The SDK can’t reach the API. It connects to https://api.getrunstate.com unless RUNSTATE_BASE_URL or baseUrl / base_url says otherwise, so check that variable first: a leftover http://localhost:8080 from local development is the usual cause. Each request times out after timeout (30 seconds) and is retried up to 3 times before the error is raised.

ConfigError on the first call

RUNSTATE_API_KEY or RUNSTATE_SPACE_ID isn’t set in the environment of the process, and wasn’t passed to configure() or Runstate. The SDK reads configuration on the first call, not at import, so the error appears there.

ConfigError is also raised for arguments once can’t hash (functions, class instances, dates), an anonymous function without name, a bad duration string, or an amount with more decimal places than the budget’s scale.

AuthError on some calls but not others

The key lacks a permission; the error message names it. Creating a limit, slots, budget or queue the first time code declares it needs resource_config. Doing the work needs coordination_write. See Configuration & auth. AuthError without a permission in the message means the key isn’t allowed for this space, the space is suspended, or a plan limit was reached.

Calls are slow with only a few workers

The SDK waits and retries when your organization or space reaches its request rate, so a rate limit shows up as slow calls rather than errors. A space allows its plan’s requests per minute (600 on Free) unless its requestsPerMinute was set lower. See request rate.

once

Two processes both ran the function

• Different runs. Results belong to a run. Check that both processes use the same RUNSTATE_RUN (or the same runstate.run(name)) and the same space.
• Different arguments. The key is a hash of the arguments. { id: 1, at: Date.now() } is different every time. Keep volatile values out of the arguments, or pass key.
• Different names. The key starts with the function name. Two functions with different names never share results. Pass the same name to both.
• A takeover. If the first process stalled longer than its lease, another caller took the work over. The first one’s result is rejected with LeaseLostError. Use a longer lease, and don’t block the event loop (see below).

The function doesn’t run again

That is the point of once: within a run, the same arguments return the recorded result. Start a fresh run to run it again: call runstate.stop() or runstate.finish() and start a new process, or use runstate.run(name) with a new client.

StoppedError on every call after stop() or finish()

A client that has seen its run end keeps raising StoppedError, so a loop in a stopped worker can’t quietly start the next run. To work in the next run, start a new process, or make a fresh client with runstate.run(name) (the next call with that name starts a fresh run).

OnceFailedError

The function threw attempts times (3 by default). The error carries the last error message. Every caller in the run gets the same error until the run ends.

Queues and workers

A job never starts

• No worker is serving. A worker process has to import the module that defines the queue and call runstate.serve().
• Different run. Workers take jobs from the run they joined and from the runs under it (child runs and gathers). Check RUNSTATE_RUN on both sides.
• Different queue name. Names are per space and case-sensitive.
• Missing resources. Every limit, slots and budget a queue names must exist. Declare them with setup(), with limit, slots or budget in code, or in the console.
• Waiting for resources. A job starts only when all of its slots, limit units and budget are available. Check Resources in the console.
• The run was stopped or its deadline passed. runstate.status() shows the run’s state.

queue X not found from a producer-only queue

A producer-only queue (runstate.queue('review') with no handler) uses the queue a worker declares. Until a worker that defines the queue has started, calls fail with a RunstateError (code NOT_FOUND) and the message queue review not found. Start the workers first and wait until they are serving, or pass the same attempts the workers use so the producer declares the queue itself.

The same job runs several times

The handler throws, or it runs longer than its lease and the lease can’t be renewed. A job is retried with backoff up to attempts (5 by default), then callers get TaskFailedError. Check context().attempt in the handler and log errors.

Submitting returns an old result

Jobs are keyed by their input within a run. Submitting the same input again returns the existing job, even if it finished long ago. Use a new run to redo the work, or pass key.

A result wait throws TimeoutError

Only your wait ended. The job is still running on the server. Call it again with the same input, or call job.result() again.

Leases

LeaseLostError, or context().signal aborts, during long work

The SDK couldn’t renew in time, so it gave the work up. For once and slots, the call raises LeaseLostError. For a queue job, the worker’s result is dropped quietly and the caller gets the result of the worker that finished the job. Common causes:

• The event loop was blocked. Renewal runs in your process. CPU-heavy synchronous work in Node.js, or blocking calls inside an async def in Python, stop renewals. Move heavy work off the event loop (worker threads, asyncio.to_thread) or use a longer lease.
• The lease is too short for your network or pauses. Try 60 to 120 seconds for long model calls.
• The run was stopped. Every held lease in the run ends. That is the intended stop signal.

Limits, slots and budgets

ConflictError when declaring a resource

The max, per or limit in code doesn’t match the resource that already exists in the space. Make the code match, change the resource in the console, or leave the sizes out of the code to use whatever is configured.

The same error comes from a queue whose attempts differ between processes, for example a worker with attempts: 5 and a producer with attempts: 3. Use the same value everywhere, or leave attempts out of producer-only queues.

A limit waits much longer than its window

• The queue is fair, so a caller asking for many units holds back smaller requests behind it.
• Callers left waiting by crashed processes stay queued for a while and can be served first.

Slots stay full after agents finish

• Slots held by crashed processes return only when their lease lapses.
• The organization may be at its “agents working at once” limit. The SDK waits through it.

BudgetExceededError although the budget looks free

Reservations count against the budget until the call finishes. Many calls in flight, each holding reserve, can fill it. Lower reserve if it is far above the usual cost, or raise the budget’s limit.

Python

An async function returns a coroutine instead of a result

Decorating an async def keeps it async. await it. The same goes for module functions: inside async def, use the a forms such as await runstate.atake(...) and await runstate.aserve(). See Sync and async.

Webhooks

Signatures don’t verify

Compute the HMAC over the raw request body exactly as received, prefixed by the x-runstate-timestamp value and a dot, using the signing secret from destination creation. Re-serializing parsed JSON changes the bytes. See Webhooks.