Durable queues of schema-typed values (Effect's
effect/persistence/PersistedQueue)::
queue = yield from persisted_queue.make(name="emails", schema=Email)
yield from queue.offer(Email(to="ada@example.com"), id="welcome-ada")
yield from queue.take(lambda email, meta: send(email))offer stores a value (an id the queue already has is not stored
again); take runs a handler on the next value: success completes it,
failure retries it on the queue's schedule until its attempts run out
(then it is failed, the dead letter), interruption releases it. Delivery
is at-least-once: a crash between a handler's success and its
acknowledgement delivers the value again, so handlers must be idempotent.
Queues come from PersistedQueueFactory (layer) over a
PersistedQueueStore: layer_store_memory, layer_store_sql (any
SqlClient) or layer_store_redis. layer_cleanup removes old
completed (and, optionally, failed) elements.
Values#
| Attribute | Type | Description |
|---|---|---|
layer_store_memory | Layer[PersistedQueueStore] | Queues in process memory, fresh each time the layer is built: failed takes are retried on the queue's schedule until its attempts run out. |
DEFAULT_MAX_ATTEMPTS | Literal[10] = 10 | |
DEFAULT_RETRY_SCHEDULE | RetrySchedule | Exponential from 1 second, capped at 5 minutes. |
layer | Layer[PersistedQueueFactory, Never, PersistedQueueStore] | The factory over the current store. |
make_factory | Effect[PersistedQueueFactoryService, Never, PersistedQueueStore] | A factory over the current store. |
Types#
| Type | Definition | Description |
|---|---|---|
RetrySchedule | type RetrySchedule = Schedule[object, int] | A queue's retry schedule: its input is the attempt number, its delay how long a failed element waits before it is visible again. |
Deduplicate | type Deduplicate = Literal['untilRemoved', 'untilExecuted'] | How long an offered custom id stays reserved: |
Classes#
Functions#
is_dead_letter_cause#
is_dead_letter_cause(cause) -> boolWhether a take scope failed because the queue could not decode the
element's stored payload. Such an element can never be processed, so a
custom store's take finalizer marks it failed at once, whatever attempts
remain (which also releases its id for an "untilExecuted" offer).
| Parameter | Type | Description |
|---|---|---|
cause | Cause[Any] |
Returns bool
def is_dead_letter_cause(cause: Cause[Any]) -> bool:
"""Whether a take scope failed because the queue could not decode the
element's stored payload. Such an element can never be processed, so a
custom store's take finalizer marks it failed at once, whatever attempts
remain (which also releases its id for an ``"untilExecuted"`` offer)."""
return dead_letter_from_cause(cause) is not Nonemake_store_memory#
make_store_memory(clock) -> PersistedQueueStoreServiceA store whose queues live in dicts, timed by clock. Delayed
entries are scanned on every take, so it is for development and tests
rather than large delayed backlogs.
| Parameter | Type | Description |
|---|---|---|
clock | ClockService |
Returns PersistedQueueStoreService
def make_store_memory(clock: ClockService) -> PersistedQueueStoreService:
"""A store whose queues live in dicts, timed by ``clock``. Delayed
entries are scanned on every take, so it is for development and tests
rather than large delayed backlogs."""
store = _MemoryStore(clock)
return PersistedQueueStoreService(offer=store.offer, take=store.take, cleanup=store.cleanup)layer_cleanup#
layer_cleanup(*, interval='1 hour', time_to_live='30 days', failed_time_to_live=None) -> Layer[Never, Never, PersistedQueueStore]Run the store's cleanup every interval (default an hour),
logging a failure. Completed elements are kept time_to_live
(default 30 days) so their ids stay de-duplicated; failed ones are the
dead letters, kept unless failed_time_to_live is given.
Run it in one instance of a deployment: racing ones are harmless but redundant.
| Parameter | Type | Description |
|---|---|---|
interval | DurationInput = '1 hour' | |
time_to_live | DurationInput = '30 days' | |
failed_time_to_live | DurationInput | None = None |
Returns Layer[Never, Never, PersistedQueueStore]
def layer_cleanup(
*,
interval: DurationInput = "1 hour",
time_to_live: DurationInput = "30 days",
failed_time_to_live: DurationInput | None = None,
) -> Layer[Never, Never, PersistedQueueStore]:
"""Run the store's ``cleanup`` every ``interval`` (default an hour),
logging a failure. Completed elements are kept ``time_to_live``
(default 30 days) so their ids stay de-duplicated; failed ones are the
dead letters, kept unless ``failed_time_to_live`` is given.
Run it in one instance of a deployment: racing ones are harmless but
redundant."""
ttl = from_input_unsafe(time_to_live)
failed_ttl = None if failed_time_to_live is None else from_input_unsafe(failed_time_to_live)
every = from_input_unsafe(interval)
def start(store: PersistedQueueStoreService) -> Effect[object, Never, Scope]:
return (
store.cleanup(time_to_live=ttl, failed_time_to_live=failed_ttl)
.catch_cause(lambda cause: log_warning("Failed to clean up persisted queue", cause))
.repeat(spaced(every))
.annotate_logs({"module": "effect/persistence/PersistedQueue", "fiber": "cleanup"})
.interruptible()
.fork_scoped()
)
return effect_discard(PersistedQueueStore().flat_map(start))make#
make[A, RR = Never](*, name, schema, max_attempts=DEFAULT_MAX_ATTEMPTS, retry_schedule=DEFAULT_RETRY_SCHEDULE) -> Effect[PersistedQueue[A, RR], Never, PersistedQueueFactory]The queue name of schema's values, from the factory.
An element failing max_attempts times is marked failed. After each
failure it waits the retry_schedule's delay for its attempt: the
schedule is replayed up to the element's stored attempt count (with
time simulated from the summed delays, so up_to(duration=) caps
the summed delays, not real time).
| Parameter | Type | Description |
|---|---|---|
name | str | |
schema | Codec[A, Any, RR, RR] | |
max_attempts | int = DEFAULT_MAX_ATTEMPTS | |
retry_schedule | RetrySchedule = DEFAULT_RETRY_SCHEDULE |
Returns Effect[PersistedQueue[A, RR], Never, PersistedQueueFactory]
def make[A, RR = Never](
*,
name: str,
schema: Codec[A, Any, RR, RR],
max_attempts: int = DEFAULT_MAX_ATTEMPTS,
retry_schedule: RetrySchedule = DEFAULT_RETRY_SCHEDULE,
) -> Effect[PersistedQueue[A, RR], Never, PersistedQueueFactory]:
"""The queue ``name`` of ``schema``'s values, from the factory.
An element failing ``max_attempts`` times is marked failed. After each
failure it waits the ``retry_schedule``'s delay for its attempt: the
schedule is replayed up to the element's stored attempt count (with
time simulated from the summed delays, so ``up_to(duration=)`` caps
the summed delays, not real time)."""
return PersistedQueueFactory().flat_map(
lambda factory: factory.make(
name=name, schema=schema, max_attempts=max_attempts, retry_schedule=retry_schedule
)
)layer_store_redis#
layer_store_redis(*, prefix='effectq:', poll_interval=_DEFAULT_POLL_INTERVAL, lock_refresh_interval=_DEFAULT_LOCK_REFRESH_INTERVAL, lock_expiration=_DEFAULT_LOCK_EXPIRATION) -> Layer[PersistedQueueStore, Never, Redis]The store of make_store_redis.
| Parameter | Type | Description |
|---|---|---|
prefix | str = 'effectq:' | |
poll_interval | DurationInput = _DEFAULT_POLL_INTERVAL | |
lock_refresh_interval | DurationInput = _DEFAULT_LOCK_REFRESH_INTERVAL | |
lock_expiration | DurationInput = _DEFAULT_LOCK_EXPIRATION |
Returns Layer[PersistedQueueStore, Never, Redis]
def layer_store_redis(
*,
prefix: str = "effectq:",
poll_interval: DurationInput = _DEFAULT_POLL_INTERVAL,
lock_refresh_interval: DurationInput = _DEFAULT_LOCK_REFRESH_INTERVAL,
lock_expiration: DurationInput = _DEFAULT_LOCK_EXPIRATION,
) -> Layer[PersistedQueueStore, Never, Redis]:
"""The store of ``make_store_redis``."""
return layer_effect(
PersistedQueueStore,
make_store_redis(
prefix=prefix,
poll_interval=poll_interval,
lock_refresh_interval=lock_refresh_interval,
lock_expiration=lock_expiration,
),
)make_store_redis#
make_store_redis(*, prefix='effectq:', poll_interval=_DEFAULT_POLL_INTERVAL, lock_refresh_interval=_DEFAULT_LOCK_REFRESH_INTERVAL, lock_expiration=_DEFAULT_LOCK_EXPIRATION) -> Effect[PersistedQueueStoreService, Never, Redis | Scope]A store on Redis. Each queue name keeps its elements under
<prefix><name> (a list, with :pending, :attempts, :ids,
:delayed and :failed beside it). A claimed element is locked
for lock_expiration, renewed every lock_refresh_interval while
it is processed; an element whose lock lapsed (its worker died) goes
back to the queue, or to :failed if that was its last attempt.
Retried and delayed elements wait in :delayed, scored by the
offering process's clock and released by the polling one's, so clock
skew between hosts shifts them. Each acquisition locks under a key of
its own, recorded in :owners; settling checks it, so a handler
whose claim a reset recovered cannot settle a newer claim of the same
id (until a reset recovers it, a handler whose lock lapsed can still
settle or renew). Older workers using one lock per id must be stopped,
and their locks left to expire, before workers of this format start on
the same prefix; keep the Redis data.
| Parameter | Type | Description |
|---|---|---|
prefix | str = 'effectq:' | |
poll_interval | DurationInput = _DEFAULT_POLL_INTERVAL | |
lock_refresh_interval | DurationInput = _DEFAULT_LOCK_REFRESH_INTERVAL | |
lock_expiration | DurationInput = _DEFAULT_LOCK_EXPIRATION |
Returns Effect[PersistedQueueStoreService, Never, Redis | Scope]
def make_store_redis(
*,
prefix: str = "effectq:",
poll_interval: DurationInput = _DEFAULT_POLL_INTERVAL,
lock_refresh_interval: DurationInput = _DEFAULT_LOCK_REFRESH_INTERVAL,
lock_expiration: DurationInput = _DEFAULT_LOCK_EXPIRATION,
) -> Effect[PersistedQueueStoreService, Never, Redis | Scope]:
"""A store on Redis. Each queue ``name`` keeps its elements under
``<prefix><name>`` (a list, with ``:pending``, ``:attempts``, ``:ids``,
``:delayed`` and ``:failed`` beside it). A claimed element is locked
for ``lock_expiration``, renewed every ``lock_refresh_interval`` while
it is processed; an element whose lock lapsed (its worker died) goes
back to the queue, or to ``:failed`` if that was its last attempt.
Retried and delayed elements wait in ``:delayed``, scored by the
offering process's clock and released by the polling one's, so clock
skew between hosts shifts them. Each acquisition locks under a key of
its own, recorded in ``:owners``; settling checks it, so a handler
whose claim a reset recovered cannot settle a newer claim of the same
id (until a reset recovers it, a handler whose lock lapsed can still
settle or renew). Older workers using one lock per id must be stopped,
and their locks left to expire, before workers of this format start on
the same prefix; keep the Redis data."""
lock_expiration_seconds = store_interval(lock_expiration, _DEFAULT_LOCK_EXPIRATION)
settings = _Settings(
prefix=prefix,
poll_interval=store_interval(poll_interval, _DEFAULT_POLL_INTERVAL),
lock_refresh=store_interval(lock_refresh_interval, _DEFAULT_LOCK_REFRESH_INTERVAL),
lock_expiration_millis=to_millis(lock_expiration_seconds),
ack_schedule=ack_retry_schedule(lock_expiration_seconds),
)
def build(redis: RedisService, clock: ClockService) -> Generator[Any, Any, Any]:
run = _Scripts(redis)
states = QueueStates()
active_locks: dict[str, scripts.Claim] = {}
mailboxes = yield from RcMap.make(
_lookup(settings, run, states, clock), idle_time_to_live="30 seconds"
)
def refresh_locks() -> Generator[Any, Any]:
while True:
yield from sleep(settings.lock_refresh)
yield from run.expire_all(
dict(active_locks), settings.lock_expiration_millis
).ignore()
yield from (
gen(refresh_locks)
.annotate_logs({"module": "effect/persistence/PersistedQueue", "fiber": "refreshLocks"})
.interruptible()
.fork_scoped()
)
store = _RedisStore(settings, redis, run, states, clock, mailboxes, active_locks)
return PersistedQueueStoreService(offer=store.offer, take=store.take, cleanup=store.cleanup)
return Redis().flat_map(
lambda redis: clock_with(lambda clock: gen(lambda: build(redis, clock)))
)layer_store_sql#
layer_store_sql(*, table_name='effect_queue', poll_interval=_DEFAULT_POLL_INTERVAL, lock_refresh_interval=_DEFAULT_LOCK_REFRESH_INTERVAL, lock_expiration=_DEFAULT_LOCK_EXPIRATION) -> Layer[PersistedQueueStore, SqlError, SqlClient]The store of make_store_sql.
| Parameter | Type | Description |
|---|---|---|
table_name | str = 'effect_queue' | |
poll_interval | DurationInput = _DEFAULT_POLL_INTERVAL | |
lock_refresh_interval | DurationInput = _DEFAULT_LOCK_REFRESH_INTERVAL | |
lock_expiration | DurationInput = _DEFAULT_LOCK_EXPIRATION |
Returns Layer[PersistedQueueStore, SqlError, SqlClient]
def layer_store_sql(
*,
table_name: str = "effect_queue",
poll_interval: DurationInput = _DEFAULT_POLL_INTERVAL,
lock_refresh_interval: DurationInput = _DEFAULT_LOCK_REFRESH_INTERVAL,
lock_expiration: DurationInput = _DEFAULT_LOCK_EXPIRATION,
) -> Layer[PersistedQueueStore, SqlError, SqlClient]:
"""The store of ``make_store_sql``."""
return layer_effect(
PersistedQueueStore,
make_store_sql(
table_name=table_name,
poll_interval=poll_interval,
lock_refresh_interval=lock_refresh_interval,
lock_expiration=lock_expiration,
),
)make_store_sql#
make_store_sql(*, table_name='effect_queue', poll_interval=_DEFAULT_POLL_INTERVAL, lock_refresh_interval=_DEFAULT_LOCK_REFRESH_INTERVAL, lock_expiration=_DEFAULT_LOCK_EXPIRATION) -> Effect[PersistedQueueStoreService, SqlError, SqlClient | Scope]A store in the table table_name (made and migrated if needed) of
the SqlClient. Each poll claims rows under a lock of its own,
renewed every lock_refresh_interval while they are processed; a row
whose lock is older than lock_expiration (its worker died) is
claimed again, or marked failed if that was its last attempt. A
settlement or renewal only applies to the claim a handler received, so
a handler whose lock lapsed still settles its row until another poll
claims it. Delays use the database's clock, rounded up to whole
seconds.
| Parameter | Type | Description |
|---|---|---|
table_name | str = 'effect_queue' | |
poll_interval | DurationInput = _DEFAULT_POLL_INTERVAL | |
lock_refresh_interval | DurationInput = _DEFAULT_LOCK_REFRESH_INTERVAL | |
lock_expiration | DurationInput = _DEFAULT_LOCK_EXPIRATION |
Returns Effect[PersistedQueueStoreService, SqlError, SqlClient | Scope]
def make_store_sql(
*,
table_name: str = "effect_queue",
poll_interval: DurationInput = _DEFAULT_POLL_INTERVAL,
lock_refresh_interval: DurationInput = _DEFAULT_LOCK_REFRESH_INTERVAL,
lock_expiration: DurationInput = _DEFAULT_LOCK_EXPIRATION,
) -> Effect[PersistedQueueStoreService, SqlError, SqlClient | Scope]:
"""A store in the table ``table_name`` (made and migrated if needed) of
the ``SqlClient``. Each poll claims rows under a lock of its own,
renewed every ``lock_refresh_interval`` while they are processed; a row
whose lock is older than ``lock_expiration`` (its worker died) is
claimed again, or marked failed if that was its last attempt. A
settlement or renewal only applies to the claim a handler received, so
a handler whose lock lapsed still settles its row until another poll
claims it. Delays use the database's clock, rounded up to whole
seconds."""
poll_every = store_interval(poll_interval, _DEFAULT_POLL_INTERVAL)
lock_refresh = store_interval(lock_refresh_interval, _DEFAULT_LOCK_REFRESH_INTERVAL)
lock_expiration_seconds = store_interval(lock_expiration, _DEFAULT_LOCK_EXPIRATION)
def build() -> Generator[Any, Any, PersistedQueueStoreService]:
sql = (yield from SqlClient()).without_transforms()
yield from migrator.make()(
loader=migrations(table_name), table=f"{table_name}_migrations"
).or_die()
dialect = Dialect(sql)
expires_at = dialect.seconds_ago(lock_expiration_seconds)
settlements = Settlements(dialect, table_name, ack_retry_schedule(lock_expiration_seconds))
yield from (
suspend(settlements.refresh_locks)
.tap_cause(log_warning)
.retry(spaced("500 millis"))
.schedule(fixed(lock_refresh))
.annotate_logs(
{
"package": "@effect/sql",
"module": "SqlPersistedQueue",
"fiber": "refreshLocks",
}
)
.fork_scoped()
)
states = QueueStates()
mailboxes = yield from RcMap.make(
_lookup(dialect, table_name, expires_at, poll_every, lock_refresh, states, settlements),
idle_time_to_live="30 seconds",
)
store = _SqlStore(
dialect,
offer_statement(dialect, table_name),
replace_executed_statement(dialect, table_name),
cleanup_statement(dialect, table_name),
settlements,
states,
mailboxes,
)
return PersistedQueueStoreService(offer=store.offer, take=store.take, cleanup=store.cleanup)
return gen(build)