Work a workflow hands to persisted background workers (Effect's
effect/workflow/DurableQueue)::
Emails = durable_queue.make(
name="emails",
payload=S.Struct(Email),
idempotency_key=lambda email: email["id"],
)
# in a workflow: offer the item and suspend until a worker is done
yield from Emails.process({"id": "welcome-ada", "to": "ada@example.com"})
# in a worker process
EmailWorker = Emails.worker(send_email, concurrency=5)process offers the item to the PersistedQueue named
DurableQueue/<name> with a DurableDeferred token, then suspends;
a worker takes it, runs the handler and records the handler's Exit
through the token, which resumes the workflow with the typed success or
error.
Delivery is at-least-once: a crash between the handler's success and the
acknowledgement delivers the item again, so handlers must be idempotent.
An item out of attempts is dead-lettered: its deferred never resolves and
the workflow stays parked until the item is requeued out of band (the
id-based de-duplication keeps replays from resurrecting it). Run
persisted_queue.layer_cleanup in one instance to prune old items.
A worker killed while it holds an item stalls that item until its lock
expires: 120 seconds by default with persisted_queue.layer_store_sql and
90 with layer_store_redis. Set it with lock_expiration= on the store.
Values#
| Attribute | Type | Description |
|---|---|---|
DEFAULT_RETRY_SCHEDULE | Schedule[Any, Any, Any, Any] | How |
Classes#
Functions#
make#
make[P, A = None, E = Never](*, name, payload, idempotency_key, success=None, error=None) -> DurableQueue[P, A, E]A queue named name of payload items; idempotency_key
names an item within its workflow execution. success defaults to
None, error to none.
| Parameter | Type | Description |
|---|---|---|
name | str | |
payload | Codec[P, Any, Never, Never] | |
idempotency_key | Callable[[P], str] | |
success | Codec[A, Any, Never, Never] | None = None | |
error | Codec[E, Any, Never, Never] | None = None |
Returns DurableQueue[P, A, E]
def make[P, A = None, E = Never](
*,
name: str,
payload: Codec[P, Any, Never, Never],
idempotency_key: Callable[[P], str],
success: Codec[A, Any, Never, Never] | None = None,
error: Codec[E, Any, Never, Never] | None = None,
) -> DurableQueue[P, A, E]:
"""A queue named ``name`` of ``payload`` items; ``idempotency_key``
names an item within its workflow execution. ``success`` defaults to
``None``, ``error`` to none."""
return DurableQueue(
name=name,
payload_schema=payload,
idempotency_key=idempotency_key,
deferred=make_deferred(f"DurableQueue/{name}", success=success, error=error),
)