GitHub

durable_queue

API reference for durable_queue.

Work a workflow hands to persisted background workers (Effect's effect/workflow/DurableQueue)::

Emails = durable_queue.make(
    name="emails",
    payload=S.Struct(Email),
    idempotency_key=lambda email: email["id"],
)

# in a workflow: offer the item and suspend until a worker is done
yield from Emails.process({"id": "welcome-ada", "to": "ada@example.com"})

# in a worker process
EmailWorker = Emails.worker(send_email, concurrency=5)

process offers the item to the PersistedQueue named DurableQueue/<name> with a DurableDeferred token, then suspends; a worker takes it, runs the handler and records the handler's Exit through the token, which resumes the workflow with the typed success or error.

Delivery is at-least-once: a crash between the handler's success and the acknowledgement delivers the item again, so handlers must be idempotent. An item out of attempts is dead-lettered: its deferred never resolves and the workflow stays parked until the item is requeued out of band (the id-based de-duplication keeps replays from resurrecting it). Run persisted_queue.layer_cleanup in one instance to prune old items.

A worker killed while it holds an item stalls that item until its lock expires: 120 seconds by default with persisted_queue.layer_store_sql and 90 with layer_store_redis. Set it with lock_expiration= on the store.

Values

AttributeTypeDescription
DEFAULT_RETRY_SCHEDULESchedule[Any, Any, Any, Any]

How process retries a failed offer: exponential from 500 ms, at most a minute apart.

Classes

Functions

make

make[P, A = None, E = Never](*, name, payload, idempotency_key, success=None, error=None) -> DurableQueue[P, A, E]

A queue named name of payload items; idempotency_key names an item within its workflow execution. success defaults to None, error to none.

ParameterTypeDescription
namestr
payloadCodec[P, Any, Never, Never]
idempotency_keyCallable[[P], str]
successCodec[A, Any, Never, Never] | None = None
errorCodec[E, Any, Never, Never] | None = None

Returns DurableQueue[P, A, E]

def make[P, A = None, E = Never](
    *,
    name: str,
    payload: Codec[P, Any, Never, Never],
    idempotency_key: Callable[[P], str],
    success: Codec[A, Any, Never, Never] | None = None,
    error: Codec[E, Any, Never, Never] | None = None,
) -> DurableQueue[P, A, E]:
    """A queue named ``name`` of ``payload`` items; ``idempotency_key``
    names an item within its workflow execution. ``success`` defaults to
    ``None``, ``error`` to none."""
    return DurableQueue(
        name=name,
        payload_schema=payload,
        idempotency_key=idempotency_key,
        deferred=make_deferred(f"DurableQueue/{name}", success=success, error=error),
    )