Limit how often each key is used and stop calling what keeps failing, in memory or shared through Redis.
RateLimiter and CircuitBreaker guard calls by key: a user id, an API, a host. Their
counters live in a store, in memory for one process or in Redis for every process that
uses it, so a limit of 100 requests a minute holds across a whole fleet.
Rate limits#
limiter.consume(key=..., limit=..., window=...) charges a token to key, allowed limit
per window, a duration such as "1 minute" (a bare number is milliseconds). The "fixed-window" algorithm (the default) counts tokens per
window; "token-bucket" refills them steadily. Over the limit, it fails with a
RateLimiterError whose reason is RateLimitExceeded (with its retry_after), or,
with on_exceeded="delay", succeeds with the delay to wait. A limit of zero admits
no request for tokens: it fails, or waits forever, without touching the store.
rate_limiter.make_with_rate_limiter gives a function for .pipe that limits any
effect, waiting out the delay when there is one. adaptive_consume and
adaptive_feedback learn an upstream server's limit from its 429 responses.
Circuit breakers#
breaker.execute(key=...) runs an effect through key's circuit. While it is closed,
calls run and count toward the trip policy: circuit_breaker.consecutive(n) failures in
a row, or a share of the last calls failing, with circuit_breaker.rate(threshold=...).
Once it trips, calls fail at once with CircuitOpen for open_duration (or a schedule of
longer openings), and then one probe call decides whether it closes again. classify
decides how a call counts, so a 404 can be a success and a 503 a failure. It gets the
call's Exit and a ClassifyContext whose probe says whether the call is the probe, so
a rate-limited response ignored while closed can still reopen the circuit.
breaker.transitions is a stream of the changes.
A call that must always run, a write to a ledger say, passes on_rejection="allow". When
the circuit would admit it, it is recorded like any call and can be the probe; when the
circuit would refuse it, it runs unrecorded instead of failing with CircuitOpen.
import pyfect as fx
from pyfect import layer
from pyfect.persistence import (
CircuitBreaker,
CircuitBreakerError,
RateLimiter,
RateLimiterError,
circuit_breaker,
rate_limiter,
)
def limited(error: RateLimiterError) -> fx.Effect[str]:
return fx.succeed(f"limited: {type(error.reason).__name__}")
@fx.fn
def rate_limits():
limiter = yield from RateLimiter()
for _ in range(3):
consumed = limiter.consume(key="user:1", limit=2, window="1 minute")
allowed = consumed.map(lambda r: f"allowed, {r.remaining} left")
print((yield from allowed.pipe(fx.catch_tag(RateLimiterError, limited))))
# with_rate_limiter limits any effect: here, waiting instead of failing.
with_rate_limiter = yield from rate_limiter.make_with_rate_limiter
reply = yield from fx.succeed("pong").pipe(
with_rate_limiter(key="ping", limit=10, window="1 second", on_exceeded="delay")
)
print(reply)
statuses = iter(["503", "503", "200"])
def call_api() -> fx.Effect[str, str]:
status = next(statuses)
return fx.succeed(status) if status == "200" else fx.fail(status)
@fx.fn
def breaker():
api = yield from CircuitBreaker()
def call():
return (
fx.suspend(call_api)
.pipe(api.execute(key="api"))
.pipe(fx.catch_tag(CircuitBreakerError, lambda e: fx.succeed(type(e.reason).__name__)))
.catch_all(lambda status: fx.succeed(f"failed: {status}"))
)
for _ in range(3):
print((yield from call()))
print(type((yield from api.state("api"))).__name__)
yield from fx.sleep("100 millis") # the open time passes: the next call is the probe
print((yield from call()))
print(type((yield from api.state("api"))).__name__)
# Counters and circuit states in memory; the Redis stores share them between processes.
Limits = layer.merge(
rate_limiter.layer.pipe(layer.provide(rate_limiter.layer_store_memory)),
circuit_breaker.layer(
trip_policy=circuit_breaker.consecutive(2), open_duration="100 millis"
).pipe(layer.provide(circuit_breaker.layer_store_memory())),
)
fx.run_main(rate_limits().and_then(breaker()).pipe(fx.provide(Limits)))HTTP clients#
client.with_rate_limiter(limiter=..., key=..., limit=..., window=...) sends each request
of an HttpClient through a limiter. It follows the server's
ratelimit-* and x-ratelimit-* headers, holds requests once a reported budget runs
out, and retries a 429 after its Retry-After. client.with_circuit_breaker(breaker=...)
gives each origin its own circuit, and circuit_breaker_classifier counts timeouts,
transport errors and 408, 500, 502, 503 and 504 responses as failures and leaves 429s
to the limiter. Apply the breaker last, after retry_transient, so it sees each
request's final outcome.
Sharing them through Redis#
The pyfect-dev[redis] extra adds redis_py.layer(url), the Redis service on redis-py.
Over it, rate_limiter.layer_store_redis() and circuit_breaker.layer_store_redis()
keep counters and circuit states in Redis, updated atomically by Lua scripts, and
persistence.layer_redis and persisted_queue.layer_store_redis() do the same for the
persistence services. Swap the memory stores for these and nothing
else changes. The memory rate limiter store drops a key once its counter is back to where
an unseen key starts, sweeping idle ones every minute; the Redis one lets them expire.
import pyfect as fx
from pyfect import layer
from pyfect.persistence import (
CircuitBreaker,
Persistence,
RateLimiter,
RedisError,
circuit_breaker,
persisted_queue,
persistence,
rate_limiter,
redis_py,
)
from pyfect.persistence.persisted_queue import PersistedQueueFactory
# One redis-py client, connected when the layer is built and closed with its scope.
RedisLive = redis_py.layer("redis://localhost:6379")
# The same services as in memory, now shared by every process using this Redis.
Live: fx.Layer[RateLimiter | CircuitBreaker | Persistence | PersistedQueueFactory, RedisError] = (
layer.merge(
rate_limiter.layer.pipe(layer.provide(rate_limiter.layer_store_redis())),
circuit_breaker.layer(
trip_policy=circuit_breaker.consecutive(5), open_duration="30 seconds"
).pipe(layer.provide(circuit_breaker.layer_store_redis())),
persistence.layer_redis,
persisted_queue.layer.pipe(layer.provide(persisted_queue.layer_store_redis())),
).pipe(layer.provide(RedisLive))
)The layer connects when it is built and fails with a RedisError if the server can't be
reached. redis_py.layer_config reads its options from config.
Reference: rate_limiter,
circuit_breaker,
redis,
redis_py.