GitHub

PersistedQueue

API reference for PersistedQueue.

class PersistedQueue[A, R = Never]

A durable queue of A values, stored as their schema's JSON.

Delivery is at-least-once: a crash between a handler's success and its acknowledgement delivers the element again, so handlers must be idempotent. Make one with make.

Functions

__init__

__init__(store, *, name, schema, max_attempts, retry_schedule) -> None
ParameterTypeDescription
storePersistedQueueStoreService
namestr
schemaCodec[A, Any, Any, Any]
max_attemptsint
retry_scheduleRetrySchedule
def __init__(
    self,
    store: PersistedQueueStoreService,
    *,
    name: str,
    schema: Codec[A, Any, Any, Any],
    max_attempts: int,
    retry_schedule: RetrySchedule,
) -> None:
    json_schema = to_codec_json(schema)
    self._encode = encode_unknown_effect(
        cast("Codec[object, object, object, Any]", json_schema)
    )
    self._decode = decode_unknown_effect(cast("Codec[A, object, Any, object]", json_schema))
    self._store = store
    self._name = name
    self._take_options = TakeOptions(
        name=name,
        max_attempts=max_attempts,
        retry_delay=lambda attempts: retry_delay(retry_schedule, attempts),
    )

offer

offer(value, *, id=None, delay=None, deduplicate='untilRemoved') -> Effect[str, PersistedQueueError | SchemaError, R]

Add value and give its id: id, or a new UUID. An id the queue already has adds nothing, and that offer's delay is ignored. deduplicate sets how long a custom id stays reserved:

  • "untilRemoved" (default): until layer_cleanup removes the completed or failed element. The id works as an idempotency key.
  • "untilExecuted": until the element completes or fails its final attempt. Offers while it is pending, processing or waiting to retry are ignored; a later offer enqueues a new element behind queued work, with its own delay. Handlers should read current state rather than the offered value: an offer arriving after the handler read state but before the element completes is ignored. A re-offered failed id may lose its earlier failure record.

The mode is chosen per offer and not stored with the id, so an "untilExecuted" offer can re-open an id other producers offer with "untilRemoved": don't share an idempotency key between them.

delay postpones the first delivery, measured by the store's clock (SQL: the database's, rounded up to whole seconds); zero or negative is none. An infinite delay stores nothing: the id is given back, and may be offered again.

ParameterTypeDescription
valueA
idstr | None = None
delayDurationInput | None = None
deduplicateDeduplicate = 'untilRemoved'

Returns Effect[str, PersistedQueueError | SchemaError, R]

def offer(
    self,
    value: A,
    *,
    id: str | None = None,
    delay: DurationInput | None = None,
    deduplicate: Deduplicate = "untilRemoved",
) -> Effect[str, PersistedQueueError | SchemaError, R]:
    """Add ``value`` and give its id: ``id``, or a new UUID. An id the
    queue already has adds nothing, and that offer's ``delay`` is
    ignored. ``deduplicate`` sets how long a custom ``id`` stays
    reserved:

    - ``"untilRemoved"`` (default): until ``layer_cleanup`` removes the
      completed or failed element. The id works as an idempotency key.
    - ``"untilExecuted"``: until the element completes or fails its final
      attempt. Offers while it is pending, processing or waiting to retry
      are ignored; a later offer enqueues a new element behind queued
      work, with its own ``delay``. Handlers should read current state
      rather than the offered value: an offer arriving after the handler
      read state but before the element completes is ignored. A
      re-offered failed id may lose its earlier failure record.

    The mode is chosen per offer and not stored with the id, so an
    ``"untilExecuted"`` offer can re-open an id other producers offer
    with ``"untilRemoved"``: don't share an idempotency key between them.

    ``delay`` postpones the first delivery, measured by the store's
    clock (SQL: the database's, rounded up to whole seconds); zero or
    negative is none. An infinite delay stores nothing: the id is
    given back, and may be offered again."""

    def body() -> Effect[str, PersistedQueueError | SchemaError, R]:
        element_id = id if id is not None else str(uuid.uuid4())
        seconds = positive_delay(delay)
        if is_infinite(seconds):
            return succeed(element_id)
        return self._encode(value).flat_map(
            lambda element: self._store.offer(
                name=self._name,
                id=element_id,
                element=element,
                is_custom_id=id is not None,
                delay=seconds,
                deduplicate=deduplicate,
            ).as_(element_id)
        )

    return suspend(body)

take

take[XA, XE = Never, XR = Never](f) -> Effect[XA, XE | PersistedQueueError, R | XR]

Run f on the next element, waiting while there is none. If f succeeds the element is done; if it fails it is retried on the queue's retry schedule, and marked failed once its attempts run out; if it is interrupted it is released for another take, without counting the attempt. An element its schema cannot decode is marked failed and the next one taken.

ParameterTypeDescription
fCallable[[A, Metadata], Effect[XA, XE, XR]]

Returns Effect[XA, XE | PersistedQueueError, R | XR]

def take[XA, XE = Never, XR = Never](
    self, f: Callable[[A, Metadata], Effect[XA, XE, XR]]
) -> Effect[XA, XE | PersistedQueueError, R | XR]:
    """Run ``f`` on the next element, waiting while there is none. If
    ``f`` succeeds the element is done; if it fails it is retried on
    the queue's retry schedule, and marked failed once its attempts run
    out; if it is interrupted it is released for another take, without
    counting the attempt. An element its schema cannot decode is marked
    failed and the next one taken."""

    def decode_failed(cause: Cause[SchemaError]) -> Effect[Never, Any]:
        if cause.has_interrupts_only:
            return fail_cause(cause)
        return fail(DeadLetter(cause))

    def attempt(scope: CloseableScope) -> Effect[XA, Any, Any]:
        return (
            self._store.take(self._take_options)
            .pipe(provide_service(Scope, scope))
            .flat_map(
                lambda item: (
                    self._decode(item.element)
                    .catch_cause(decode_failed)
                    .flat_map(lambda value: f(value, Metadata(item.id, item.attempts)))
                )
            )
        )

    return suspend(lambda: _take_loop(attempt))