GitHub

persisted_queue

API reference for persisted_queue.

Durable queues of schema-typed values (Effect's effect/persistence/PersistedQueue)::

queue = yield from persisted_queue.make(name="emails", schema=Email)
yield from queue.offer(Email(to="ada@example.com"), id="welcome-ada")
yield from queue.take(lambda email, meta: send(email))

offer stores a value (an id the queue already has is not stored again); take runs a handler on the next value: success completes it, failure retries it on the queue's schedule until its attempts run out (then it is failed, the dead letter), interruption releases it. Delivery is at-least-once: a crash between a handler's success and its acknowledgement delivers the value again, so handlers must be idempotent.

Queues come from PersistedQueueFactory (layer) over a PersistedQueueStore: layer_store_memory, layer_store_sql (any SqlClient) or layer_store_redis. layer_cleanup removes old completed (and, optionally, failed) elements.

Values

AttributeTypeDescription
layer_store_memoryLayer[PersistedQueueStore]

Queues in process memory, fresh each time the layer is built: failed takes are retried on the queue's schedule until its attempts run out.

DEFAULT_MAX_ATTEMPTSLiteral[10] = 10
DEFAULT_RETRY_SCHEDULERetrySchedule

Exponential from 1 second, capped at 5 minutes.

layerLayer[PersistedQueueFactory, Never, PersistedQueueStore]

The factory over the current store.

make_factoryEffect[PersistedQueueFactoryService, Never, PersistedQueueStore]

A factory over the current store.

Types

TypeDefinitionDescription
RetryScheduletype RetrySchedule = Schedule[object, int]

A queue's retry schedule: its input is the attempt number, its delay how long a failed element waits before it is visible again.

Deduplicatetype Deduplicate = Literal['untilRemoved', 'untilExecuted']

How long an offered custom id stays reserved: "untilRemoved" until cleanup removes its completed or failed element (an idempotency key); "untilExecuted" until its element completes or fails its final attempt.

Classes

Functions

is_dead_letter_cause

is_dead_letter_cause(cause) -> bool

Whether a take scope failed because the queue could not decode the element's stored payload. Such an element can never be processed, so a custom store's take finalizer marks it failed at once, whatever attempts remain (which also releases its id for an "untilExecuted" offer).

ParameterTypeDescription
causeCause[Any]

Returns bool

def is_dead_letter_cause(cause: Cause[Any]) -> bool:
    """Whether a take scope failed because the queue could not decode the
    element's stored payload. Such an element can never be processed, so a
    custom store's take finalizer marks it failed at once, whatever attempts
    remain (which also releases its id for an ``"untilExecuted"`` offer)."""
    return dead_letter_from_cause(cause) is not None

make_store_memory

make_store_memory(clock) -> PersistedQueueStoreService

A store whose queues live in dicts, timed by clock. Delayed entries are scanned on every take, so it is for development and tests rather than large delayed backlogs.

ParameterTypeDescription
clockClockService

Returns PersistedQueueStoreService

def make_store_memory(clock: ClockService) -> PersistedQueueStoreService:
    """A store whose queues live in dicts, timed by ``clock``. Delayed
    entries are scanned on every take, so it is for development and tests
    rather than large delayed backlogs."""
    store = _MemoryStore(clock)
    return PersistedQueueStoreService(offer=store.offer, take=store.take, cleanup=store.cleanup)

layer_cleanup

layer_cleanup(*, interval='1 hour', time_to_live='30 days', failed_time_to_live=None) -> Layer[Never, Never, PersistedQueueStore]

Run the store's cleanup every interval (default an hour), logging a failure. Completed elements are kept time_to_live (default 30 days) so their ids stay de-duplicated; failed ones are the dead letters, kept unless failed_time_to_live is given.

Run it in one instance of a deployment: racing ones are harmless but redundant.

ParameterTypeDescription
intervalDurationInput = '1 hour'
time_to_liveDurationInput = '30 days'
failed_time_to_liveDurationInput | None = None

Returns Layer[Never, Never, PersistedQueueStore]

def layer_cleanup(
    *,
    interval: DurationInput = "1 hour",
    time_to_live: DurationInput = "30 days",
    failed_time_to_live: DurationInput | None = None,
) -> Layer[Never, Never, PersistedQueueStore]:
    """Run the store's ``cleanup`` every ``interval`` (default an hour),
    logging a failure. Completed elements are kept ``time_to_live``
    (default 30 days) so their ids stay de-duplicated; failed ones are the
    dead letters, kept unless ``failed_time_to_live`` is given.

    Run it in one instance of a deployment: racing ones are harmless but
    redundant."""
    ttl = from_input_unsafe(time_to_live)
    failed_ttl = None if failed_time_to_live is None else from_input_unsafe(failed_time_to_live)
    every = from_input_unsafe(interval)

    def start(store: PersistedQueueStoreService) -> Effect[object, Never, Scope]:
        return (
            store.cleanup(time_to_live=ttl, failed_time_to_live=failed_ttl)
            .catch_cause(lambda cause: log_warning("Failed to clean up persisted queue", cause))
            .repeat(spaced(every))
            .annotate_logs({"module": "effect/persistence/PersistedQueue", "fiber": "cleanup"})
            .interruptible()
            .fork_scoped()
        )

    return effect_discard(PersistedQueueStore().flat_map(start))

make

make[A, RR = Never](*, name, schema, max_attempts=DEFAULT_MAX_ATTEMPTS, retry_schedule=DEFAULT_RETRY_SCHEDULE) -> Effect[PersistedQueue[A, RR], Never, PersistedQueueFactory]

The queue name of schema's values, from the factory.

An element failing max_attempts times is marked failed. After each failure it waits the retry_schedule's delay for its attempt: the schedule is replayed up to the element's stored attempt count (with time simulated from the summed delays, so up_to(duration=) caps the summed delays, not real time).

ParameterTypeDescription
namestr
schemaCodec[A, Any, RR, RR]
max_attemptsint = DEFAULT_MAX_ATTEMPTS
retry_scheduleRetrySchedule = DEFAULT_RETRY_SCHEDULE

Returns Effect[PersistedQueue[A, RR], Never, PersistedQueueFactory]

def make[A, RR = Never](
    *,
    name: str,
    schema: Codec[A, Any, RR, RR],
    max_attempts: int = DEFAULT_MAX_ATTEMPTS,
    retry_schedule: RetrySchedule = DEFAULT_RETRY_SCHEDULE,
) -> Effect[PersistedQueue[A, RR], Never, PersistedQueueFactory]:
    """The queue ``name`` of ``schema``'s values, from the factory.

    An element failing ``max_attempts`` times is marked failed. After each
    failure it waits the ``retry_schedule``'s delay for its attempt: the
    schedule is replayed up to the element's stored attempt count (with
    time simulated from the summed delays, so ``up_to(duration=)`` caps
    the summed delays, not real time)."""
    return PersistedQueueFactory().flat_map(
        lambda factory: factory.make(
            name=name, schema=schema, max_attempts=max_attempts, retry_schedule=retry_schedule
        )
    )

layer_store_redis

layer_store_redis(*, prefix='effectq:', poll_interval=_DEFAULT_POLL_INTERVAL, lock_refresh_interval=_DEFAULT_LOCK_REFRESH_INTERVAL, lock_expiration=_DEFAULT_LOCK_EXPIRATION) -> Layer[PersistedQueueStore, Never, Redis]

The store of make_store_redis.

ParameterTypeDescription
prefixstr = 'effectq:'
poll_intervalDurationInput = _DEFAULT_POLL_INTERVAL
lock_refresh_intervalDurationInput = _DEFAULT_LOCK_REFRESH_INTERVAL
lock_expirationDurationInput = _DEFAULT_LOCK_EXPIRATION

Returns Layer[PersistedQueueStore, Never, Redis]

def layer_store_redis(
    *,
    prefix: str = "effectq:",
    poll_interval: DurationInput = _DEFAULT_POLL_INTERVAL,
    lock_refresh_interval: DurationInput = _DEFAULT_LOCK_REFRESH_INTERVAL,
    lock_expiration: DurationInput = _DEFAULT_LOCK_EXPIRATION,
) -> Layer[PersistedQueueStore, Never, Redis]:
    """The store of ``make_store_redis``."""
    return layer_effect(
        PersistedQueueStore,
        make_store_redis(
            prefix=prefix,
            poll_interval=poll_interval,
            lock_refresh_interval=lock_refresh_interval,
            lock_expiration=lock_expiration,
        ),
    )

make_store_redis

make_store_redis(*, prefix='effectq:', poll_interval=_DEFAULT_POLL_INTERVAL, lock_refresh_interval=_DEFAULT_LOCK_REFRESH_INTERVAL, lock_expiration=_DEFAULT_LOCK_EXPIRATION) -> Effect[PersistedQueueStoreService, Never, Redis | Scope]

A store on Redis. Each queue name keeps its elements under <prefix><name> (a list, with :pending, :attempts, :ids, :delayed and :failed beside it). A claimed element is locked for lock_expiration, renewed every lock_refresh_interval while it is processed; an element whose lock lapsed (its worker died) goes back to the queue, or to :failed if that was its last attempt. Retried and delayed elements wait in :delayed, scored by the offering process's clock and released by the polling one's, so clock skew between hosts shifts them. Each acquisition locks under a key of its own, recorded in :owners; settling checks it, so a handler whose claim a reset recovered cannot settle a newer claim of the same id (until a reset recovers it, a handler whose lock lapsed can still settle or renew). Older workers using one lock per id must be stopped, and their locks left to expire, before workers of this format start on the same prefix; keep the Redis data.

ParameterTypeDescription
prefixstr = 'effectq:'
poll_intervalDurationInput = _DEFAULT_POLL_INTERVAL
lock_refresh_intervalDurationInput = _DEFAULT_LOCK_REFRESH_INTERVAL
lock_expirationDurationInput = _DEFAULT_LOCK_EXPIRATION

Returns Effect[PersistedQueueStoreService, Never, Redis | Scope]

def make_store_redis(
    *,
    prefix: str = "effectq:",
    poll_interval: DurationInput = _DEFAULT_POLL_INTERVAL,
    lock_refresh_interval: DurationInput = _DEFAULT_LOCK_REFRESH_INTERVAL,
    lock_expiration: DurationInput = _DEFAULT_LOCK_EXPIRATION,
) -> Effect[PersistedQueueStoreService, Never, Redis | Scope]:
    """A store on Redis. Each queue ``name`` keeps its elements under
    ``<prefix><name>`` (a list, with ``:pending``, ``:attempts``, ``:ids``,
    ``:delayed`` and ``:failed`` beside it). A claimed element is locked
    for ``lock_expiration``, renewed every ``lock_refresh_interval`` while
    it is processed; an element whose lock lapsed (its worker died) goes
    back to the queue, or to ``:failed`` if that was its last attempt.
    Retried and delayed elements wait in ``:delayed``, scored by the
    offering process's clock and released by the polling one's, so clock
    skew between hosts shifts them. Each acquisition locks under a key of
    its own, recorded in ``:owners``; settling checks it, so a handler
    whose claim a reset recovered cannot settle a newer claim of the same
    id (until a reset recovers it, a handler whose lock lapsed can still
    settle or renew). Older workers using one lock per id must be stopped,
    and their locks left to expire, before workers of this format start on
    the same prefix; keep the Redis data."""
    lock_expiration_seconds = store_interval(lock_expiration, _DEFAULT_LOCK_EXPIRATION)
    settings = _Settings(
        prefix=prefix,
        poll_interval=store_interval(poll_interval, _DEFAULT_POLL_INTERVAL),
        lock_refresh=store_interval(lock_refresh_interval, _DEFAULT_LOCK_REFRESH_INTERVAL),
        lock_expiration_millis=to_millis(lock_expiration_seconds),
        ack_schedule=ack_retry_schedule(lock_expiration_seconds),
    )

    def build(redis: RedisService, clock: ClockService) -> Generator[Any, Any, Any]:
        run = _Scripts(redis)
        states = QueueStates()
        active_locks: dict[str, scripts.Claim] = {}
        mailboxes = yield from RcMap.make(
            _lookup(settings, run, states, clock), idle_time_to_live="30 seconds"
        )

        def refresh_locks() -> Generator[Any, Any]:
            while True:
                yield from sleep(settings.lock_refresh)
                yield from run.expire_all(
                    dict(active_locks), settings.lock_expiration_millis
                ).ignore()

        yield from (
            gen(refresh_locks)
            .annotate_logs({"module": "effect/persistence/PersistedQueue", "fiber": "refreshLocks"})
            .interruptible()
            .fork_scoped()
        )
        store = _RedisStore(settings, redis, run, states, clock, mailboxes, active_locks)
        return PersistedQueueStoreService(offer=store.offer, take=store.take, cleanup=store.cleanup)

    return Redis().flat_map(
        lambda redis: clock_with(lambda clock: gen(lambda: build(redis, clock)))
    )

layer_store_sql

layer_store_sql(*, table_name='effect_queue', poll_interval=_DEFAULT_POLL_INTERVAL, lock_refresh_interval=_DEFAULT_LOCK_REFRESH_INTERVAL, lock_expiration=_DEFAULT_LOCK_EXPIRATION) -> Layer[PersistedQueueStore, SqlError, SqlClient]

The store of make_store_sql.

ParameterTypeDescription
table_namestr = 'effect_queue'
poll_intervalDurationInput = _DEFAULT_POLL_INTERVAL
lock_refresh_intervalDurationInput = _DEFAULT_LOCK_REFRESH_INTERVAL
lock_expirationDurationInput = _DEFAULT_LOCK_EXPIRATION

Returns Layer[PersistedQueueStore, SqlError, SqlClient]

def layer_store_sql(
    *,
    table_name: str = "effect_queue",
    poll_interval: DurationInput = _DEFAULT_POLL_INTERVAL,
    lock_refresh_interval: DurationInput = _DEFAULT_LOCK_REFRESH_INTERVAL,
    lock_expiration: DurationInput = _DEFAULT_LOCK_EXPIRATION,
) -> Layer[PersistedQueueStore, SqlError, SqlClient]:
    """The store of ``make_store_sql``."""
    return layer_effect(
        PersistedQueueStore,
        make_store_sql(
            table_name=table_name,
            poll_interval=poll_interval,
            lock_refresh_interval=lock_refresh_interval,
            lock_expiration=lock_expiration,
        ),
    )

make_store_sql

make_store_sql(*, table_name='effect_queue', poll_interval=_DEFAULT_POLL_INTERVAL, lock_refresh_interval=_DEFAULT_LOCK_REFRESH_INTERVAL, lock_expiration=_DEFAULT_LOCK_EXPIRATION) -> Effect[PersistedQueueStoreService, SqlError, SqlClient | Scope]

A store in the table table_name (made and migrated if needed) of the SqlClient. Each poll claims rows under a lock of its own, renewed every lock_refresh_interval while they are processed; a row whose lock is older than lock_expiration (its worker died) is claimed again, or marked failed if that was its last attempt. A settlement or renewal only applies to the claim a handler received, so a handler whose lock lapsed still settles its row until another poll claims it. Delays use the database's clock, rounded up to whole seconds.

ParameterTypeDescription
table_namestr = 'effect_queue'
poll_intervalDurationInput = _DEFAULT_POLL_INTERVAL
lock_refresh_intervalDurationInput = _DEFAULT_LOCK_REFRESH_INTERVAL
lock_expirationDurationInput = _DEFAULT_LOCK_EXPIRATION

Returns Effect[PersistedQueueStoreService, SqlError, SqlClient | Scope]

def make_store_sql(
    *,
    table_name: str = "effect_queue",
    poll_interval: DurationInput = _DEFAULT_POLL_INTERVAL,
    lock_refresh_interval: DurationInput = _DEFAULT_LOCK_REFRESH_INTERVAL,
    lock_expiration: DurationInput = _DEFAULT_LOCK_EXPIRATION,
) -> Effect[PersistedQueueStoreService, SqlError, SqlClient | Scope]:
    """A store in the table ``table_name`` (made and migrated if needed) of
    the ``SqlClient``. Each poll claims rows under a lock of its own,
    renewed every ``lock_refresh_interval`` while they are processed; a row
    whose lock is older than ``lock_expiration`` (its worker died) is
    claimed again, or marked failed if that was its last attempt. A
    settlement or renewal only applies to the claim a handler received, so
    a handler whose lock lapsed still settles its row until another poll
    claims it. Delays use the database's clock, rounded up to whole
    seconds."""
    poll_every = store_interval(poll_interval, _DEFAULT_POLL_INTERVAL)
    lock_refresh = store_interval(lock_refresh_interval, _DEFAULT_LOCK_REFRESH_INTERVAL)
    lock_expiration_seconds = store_interval(lock_expiration, _DEFAULT_LOCK_EXPIRATION)

    def build() -> Generator[Any, Any, PersistedQueueStoreService]:
        sql = (yield from SqlClient()).without_transforms()
        yield from migrator.make()(
            loader=migrations(table_name), table=f"{table_name}_migrations"
        ).or_die()
        dialect = Dialect(sql)
        expires_at = dialect.seconds_ago(lock_expiration_seconds)
        settlements = Settlements(dialect, table_name, ack_retry_schedule(lock_expiration_seconds))
        yield from (
            suspend(settlements.refresh_locks)
            .tap_cause(log_warning)
            .retry(spaced("500 millis"))
            .schedule(fixed(lock_refresh))
            .annotate_logs(
                {
                    "package": "@effect/sql",
                    "module": "SqlPersistedQueue",
                    "fiber": "refreshLocks",
                }
            )
            .fork_scoped()
        )
        states = QueueStates()
        mailboxes = yield from RcMap.make(
            _lookup(dialect, table_name, expires_at, poll_every, lock_refresh, states, settlements),
            idle_time_to_live="30 seconds",
        )
        store = _SqlStore(
            dialect,
            offer_statement(dialect, table_name),
            replace_executed_statement(dialect, table_name),
            cleanup_statement(dialect, table_name),
            settlements,
            states,
            mailboxes,
        )
        return PersistedQueueStoreService(offer=store.offer, take=store.take, cleanup=store.cleanup)

    return gen(build)