GitHub

Rate limiting and circuit breakers

Limit how often each key is used and stop calling what keeps failing, in memory or shared through Redis.

RateLimiter and CircuitBreaker guard calls by key: a user id, an API, a host. Their counters live in a store, in memory for one process or in Redis for every process that uses it, so a limit of 100 requests a minute holds across a whole fleet.

Rate limits

limiter.consume(key=..., limit=..., window=...) charges a token to key, allowed limit per window, a duration such as "1 minute" (a bare number is milliseconds). The "fixed-window" algorithm (the default) counts tokens per window; "token-bucket" refills them steadily. Over the limit, it fails with a RateLimiterError whose reason is RateLimitExceeded (with its retry_after), or, with on_exceeded="delay", succeeds with the delay to wait. A limit of zero admits no request for tokens: it fails, or waits forever, without touching the store.

rate_limiter.make_with_rate_limiter gives a function for .pipe that limits any effect, waiting out the delay when there is one. adaptive_consume and adaptive_feedback learn an upstream server's limit from its 429 responses.

Circuit breakers

breaker.execute(key=...) runs an effect through key's circuit. While it is closed, calls run and count toward the trip policy: circuit_breaker.consecutive(n) failures in a row, or a share of the last calls failing, with circuit_breaker.rate(threshold=...). Once it trips, calls fail at once with CircuitOpen for open_duration (or a schedule of longer openings), and then one probe call decides whether it closes again. classify decides how a call counts, so a 404 can be a success and a 503 a failure. It gets the call's Exit and a ClassifyContext whose probe says whether the call is the probe, so a rate-limited response ignored while closed can still reopen the circuit. breaker.transitions is a stream of the changes.

A call that must always run, a write to a ledger say, passes on_rejection="allow". When the circuit would admit it, it is recorded like any call and can be the probe; when the circuit would refuse it, it runs unrecorded instead of failing with CircuitOpen.

trip policy: too many failuresopen_duration passesprobe succeedsprobe failsClosed: calls run and arecountedOpen: calls fail with CircuitOpenHalfOpen: one probe runs
trip policy: too many failuresopen_duration passesprobe succeedsprobe failsClosed: calls run and arecountedOpen: calls fail with CircuitOpenHalfOpen: one probe runs
limits.py
import pyfect as fx
from pyfect import layer
from pyfect.persistence import (
    CircuitBreaker,
    CircuitBreakerError,
    RateLimiter,
    RateLimiterError,
    circuit_breaker,
    rate_limiter,
)


def limited(error: RateLimiterError) -> fx.Effect[str]:
    return fx.succeed(f"limited: {type(error.reason).__name__}")


@fx.fn
def rate_limits():
    limiter = yield from RateLimiter()
    for _ in range(3):
        consumed = limiter.consume(key="user:1", limit=2, window="1 minute")
        allowed = consumed.map(lambda r: f"allowed, {r.remaining} left")
        print((yield from allowed.pipe(fx.catch_tag(RateLimiterError, limited))))

    # with_rate_limiter limits any effect: here, waiting instead of failing.
    with_rate_limiter = yield from rate_limiter.make_with_rate_limiter
    reply = yield from fx.succeed("pong").pipe(
        with_rate_limiter(key="ping", limit=10, window="1 second", on_exceeded="delay")
    )
    print(reply)


statuses = iter(["503", "503", "200"])


def call_api() -> fx.Effect[str, str]:
    status = next(statuses)
    return fx.succeed(status) if status == "200" else fx.fail(status)


@fx.fn
def breaker():
    api = yield from CircuitBreaker()

    def call():
        return (
            fx.suspend(call_api)
            .pipe(api.execute(key="api"))
            .pipe(fx.catch_tag(CircuitBreakerError, lambda e: fx.succeed(type(e.reason).__name__)))
            .catch_all(lambda status: fx.succeed(f"failed: {status}"))
        )

    for _ in range(3):
        print((yield from call()))
    print(type((yield from api.state("api"))).__name__)
    yield from fx.sleep("100 millis")  # the open time passes: the next call is the probe
    print((yield from call()))
    print(type((yield from api.state("api"))).__name__)


# Counters and circuit states in memory; the Redis stores share them between processes.
Limits = layer.merge(
    rate_limiter.layer.pipe(layer.provide(rate_limiter.layer_store_memory)),
    circuit_breaker.layer(
        trip_policy=circuit_breaker.consecutive(2), open_duration="100 millis"
    ).pipe(layer.provide(circuit_breaker.layer_store_memory())),
)
fx.run_main(rate_limits().and_then(breaker()).pipe(fx.provide(Limits)))

HTTP clients

client.with_rate_limiter(limiter=..., key=..., limit=..., window=...) sends each request of an HttpClient through a limiter. It follows the server's ratelimit-* and x-ratelimit-* headers, holds requests once a reported budget runs out, and retries a 429 after its Retry-After. client.with_circuit_breaker(breaker=...) gives each origin its own circuit, and circuit_breaker_classifier counts timeouts, transport errors and 408, 500, 502, 503 and 504 responses as failures and leaves 429s to the limiter. Apply the breaker last, after retry_transient, so it sees each request's final outcome.

Sharing them through Redis

The pyfect-dev[redis] extra adds redis_py.layer(url), the Redis service on redis-py. Over it, rate_limiter.layer_store_redis() and circuit_breaker.layer_store_redis() keep counters and circuit states in Redis, updated atomically by Lua scripts, and persistence.layer_redis and persisted_queue.layer_store_redis() do the same for the persistence services. Swap the memory stores for these and nothing else changes. The memory rate limiter store drops a key once its counter is back to where an unseen key starts, sweeping idle ones every minute; the Redis one lets them expire.

redis_live.py
import pyfect as fx
from pyfect import layer
from pyfect.persistence import (
    CircuitBreaker,
    Persistence,
    RateLimiter,
    RedisError,
    circuit_breaker,
    persisted_queue,
    persistence,
    rate_limiter,
    redis_py,
)
from pyfect.persistence.persisted_queue import PersistedQueueFactory

# One redis-py client, connected when the layer is built and closed with its scope.
RedisLive = redis_py.layer("redis://localhost:6379")

# The same services as in memory, now shared by every process using this Redis.
Live: fx.Layer[RateLimiter | CircuitBreaker | Persistence | PersistedQueueFactory, RedisError] = (
    layer.merge(
        rate_limiter.layer.pipe(layer.provide(rate_limiter.layer_store_redis())),
        circuit_breaker.layer(
            trip_policy=circuit_breaker.consecutive(5), open_duration="30 seconds"
        ).pipe(layer.provide(circuit_breaker.layer_store_redis())),
        persistence.layer_redis,
        persisted_queue.layer.pipe(layer.provide(persisted_queue.layer_store_redis())),
    ).pipe(layer.provide(RedisLive))
)

The layer connects when it is built and fails with a RedisError if the server can't be reached. redis_py.layer_config reads its options from config.

Reference: rate_limiter, circuit_breaker, redis, redis_py.