GitHub

Request batching

Collect lookups made at the same time into one call with Request and RequestResolver.

Code that fetches one thing at a time, such as get_user(id) for each order on a page, makes one query per item: the N+1 problem. A request describes one lookup, and a resolver answers many of them at once. Requests made through the same resolver in the same scheduler tick go to it as one batch, so N concurrent lookups become one call, while the code that makes them still asks for one item.

Requests and resolvers

Subclass fx.Request[A, E] as a frozen dataclass: A is what the request succeeds with, E what it fails with (Never by default), and a third parameter names the services it needs. Frozen, equal requests compare equal, which caching and grouping rely on.

fx.RequestResolver(run_all) gets every Entry of a batch, in the order the requests were made, and completes each one: entry.request is the request, and entry.succeed(value), entry.fail(error), entry.complete(exit) are effects that complete it (complete_unsafe(exit) does it from plain code). fx.request(req, resolver) makes the request and waits for its outcome; its type comes from the request.

users.py
from collections.abc import Sequence
from dataclasses import dataclass

import pyfect as fx
from pyfect.batching import Entry

USERS = {1: "ada", 2: "grace", 3: "edsger"}
queries: list[list[int]] = []


@dataclass
class UserNotFound(fx.TaggedError):
    id: int


# A request names what it succeeds and fails with. Frozen, so equal requests
# compare equal.
@dataclass(frozen=True)
class GetUser(fx.Request[str, UserNotFound]):
    id: int


def users_by_id(ids: list[int]) -> dict[int, str]:
    # Stands in for `SELECT ... WHERE id IN (...)`: one query for the whole batch.
    queries.append(ids)
    return {i: USERS[i] for i in ids if i in USERS}


@fx.fn
def run_all(entries: Sequence[Entry[GetUser]]):
    found = users_by_id(sorted({entry.request.id for entry in entries}))
    for entry in entries:
        name = found.get(entry.request.id)
        if name is None:
            yield from entry.fail(UserNotFound(entry.request.id))
        else:
            yield from entry.succeed(name)


users = fx.RequestResolver(run_all)


def get_user(id: int) -> fx.Effect[str, UserNotFound]:
    return fx.request(GetUser(id), users)


@fx.fn
def program():
    # Made concurrently, the three requests are one batch and one query.
    names = yield from fx.for_each([1, 2, 3], get_user, concurrency="unbounded")
    print(names, queries)

    # One after another, each request is a batch of its own.
    queries.clear()
    yield from fx.for_each([1, 2], get_user)
    print(queries)

    # An entry the resolver fails fails only its own request.
    queries.clear()
    found, missing = yield from fx.all(
        get_user(1).result(), get_user(9).result(), concurrency="unbounded"
    )
    print(found, missing, queries)


fx.run_blocking(program().or_die())

Batching needs the requests to be made concurrently: for_each and all with a concurrency option, forked fibers, or anything else that runs them side by side. There is no switch to turn it on. Run one after another, each request is a batch of one, which still works but saves nothing. Equal requests in a batch are separate entries, so run_all dedupes the keys it queries, as above.

Interrupting a caller takes its request out of the batch, and a batch nobody waits for any more does not run.

Failures

An entry failed with entry.fail fails only its own request. run_all itself can fail too: name that error as the resolver's second type parameter, RequestResolver[GetPrice, DatabaseDown](run_all), and every request the run has not completed fails with it. A request's error type is its own error and the resolver's together. A request the run finishes without completing dies with UncompletedRequestError, since that is a bug in the resolver.

failures.py
from collections.abc import Sequence
from dataclasses import dataclass

import pyfect as fx
from pyfect.batching import Entry, UncompletedRequestError


@dataclass
class DatabaseDown(fx.TaggedError):
    pass


@dataclass(frozen=True)
class GetPrice(fx.Request[int]):
    sku: str


PRICES = {"apple": 30, "pear": 45}


@dataclass
class Database:
    up: bool = True


database = Database()


@fx.fn
def run_all(entries: Sequence[Entry[GetPrice]]):
    if not database.up:
        # The run fails: every request it has not completed fails with this.
        return (yield from DatabaseDown())
    for entry in entries:
        if entry.request.sku in PRICES:
            yield from entry.succeed(PRICES[entry.request.sku])
        # Any other SKU is left uncompleted.


prices = fx.RequestResolver[GetPrice, DatabaseDown](run_all)


def price(sku: str) -> fx.Effect[int, DatabaseDown]:
    return fx.request(GetPrice(sku), prices)


@fx.fn
def program():
    both = fx.all(price("apple"), price("pear"), concurrency="unbounded")
    assert (yield from both) == (30, 45)

    database.up = False
    failed = yield from both.result()
    assert isinstance(failed, fx.Err) and isinstance(failed.error, DatabaseDown)

    # A request the run leaves uncompleted dies: that is a bug in the resolver.
    database.up = True
    lost = yield from price("plum").exit()
    assert isinstance(lost, fx.Failure)
    [defect] = lost.cause.defects
    assert isinstance(defect, UncompletedRequestError)


fx.run_blocking(program().or_die())

Shaping batches

Other constructors cover the common shapes. RequestResolver.from_function_batched(f) answers a batch with f(entries), one value per entry in order; from_function(f) answers each entry alone; from_effect(f) runs an effect per entry, all at once, and from_effect_tagged({GetUser: ..., GetTeam: ...}) splits a batch by request class.

Methods derive new resolvers. batch_n(n) runs a batch as soon as it holds n requests, for an API with a page size. grouped(key) (or make_grouped) splits requests into batches by a key, such as the shard or tenant. set_delay("10 millis") collects each batch for longer than one yield. around(before, after) wraps each batch, and with_span(name) traces it with its size and links to the spans that made the requests.

sizes.py
from collections.abc import Sequence
from dataclasses import dataclass

import pyfect as fx
from pyfect.batching import Entry

sizes: list[int] = []


@dataclass(frozen=True)
class Square(fx.Request[int]):
    n: int


def squares(entries: Sequence[Entry[Square]]) -> list[int]:
    sizes.append(len(entries))
    return [entry.request.n**2 for entry in entries]


# One value per entry, in order. Batches run once they hold 3 requests.
resolver = fx.RequestResolver.from_function_batched(squares).batch_n(3)


@fx.fn
def program():
    results = yield from fx.for_each(
        range(7), lambda n: fx.request(Square(n), resolver), concurrency="unbounded"
    )
    print(results, sizes)


fx.run_blocking(program())

Caching

A resolver doesn't remember anything by default. resolver.with_cache(capacity=...) is an effect giving one that does, by request equality: an equal request made while one is pending waits for it rather than joining the batch, and a later one is answered at once without a batch. The least recently used outcomes go first (strategy="fifo": the oldest), and interrupted outcomes are not kept. Make the cached resolver once and share it, as each call to with_cache starts an empty cache.

cache.py
from collections.abc import Sequence
from dataclasses import dataclass

import pyfect as fx
from pyfect.batching import Entry

batches: list[list[str]] = []


@dataclass(frozen=True)
class GetStock(fx.Request[int]):
    sku: str


def run_all(entries: Sequence[Entry[GetStock]]) -> fx.Effect[None]:
    def answer() -> None:
        batches.append([entry.request.sku for entry in entries])
        for entry in entries:
            entry.complete_unsafe(fx.Success(len(entry.request.sku)))

    return fx.sync(answer)


@fx.fn
def program():
    stock = fx.RequestResolver(run_all)

    def lookup(skus: list[str]) -> fx.Effect[list[int]]:
        return fx.for_each(skus, lambda sku: fx.request(GetStock(sku), stock), concurrency=4)

    # Without a cache, equal requests are separate entries in the batch.
    yield from lookup(["apple", "apple", "pear"])
    print(batches)

    # with_cache remembers outcomes by request: an equal request made while one is
    # pending waits for it, and a later one is answered without a batch.
    batches.clear()
    cached = yield from stock.with_cache(capacity=100)

    def cached_lookup(skus: list[str]) -> fx.Effect[list[int]]:
        return fx.for_each(skus, lambda sku: fx.request(GetStock(sku), cached), concurrency=4)

    yield from cached_lookup(["apple", "apple", "pear"])
    yield from cached_lookup(["pear", "plum"])
    print(batches)


fx.run_blocking(program())

resolver.as_cache(capacity=..., time_to_live=...) gives a Cache whose lookups are requests, for outcomes that expire. resolver.persisted(store_id=...) keeps outcomes in a persistence store, so they outlive the process.

Reference: pyfect.batching.