GitHub

Threads and throughput

All of a run's fibers share one thread: what that means for CPU-heavy work, free-threaded Python and the throughput to expect.

Fibers are cheap and many of them can be waiting at once, but they don't run in parallel. Every fiber of a run executes on that run's thread: the thread that called fx.run_blocking (until it returns; see below), or the asyncio event loop's thread for run_async and a ManagedRuntime used from an asyncio app. A fiber runs until it waits (a sleep, a queue, a socket, fx.blocking) or uses up its step budget, then the next one runs, as in Effect on Node's single thread.

one_thread.py
import threading

import pyfect as fx


def thread_of(_: int) -> fx.Effect[int]:
    # Each fiber yields once, so they interleave, and reports the thread it ran on.
    return fx.yield_now().map(lambda _: threading.get_ident())


@fx.gen
def program():
    fibers = yield from fx.for_each(range(100), thread_of, concurrency="unbounded")
    elsewhere = yield from fx.blocking(threading.get_ident)
    here = threading.get_ident()
    print("threads the 100 fibers ran on:", len(set(fibers)))
    print("the run's thread:", set(fibers) == {here})
    print("fx.blocking ran elsewhere:", elsewhere != here)


fx.run_blocking(program)

Free-threaded Python

Free-threaded CPython (3.14t) removes the GIL, but it doesn't make fibers faster: they still take turns on one thread. What it speeds up is work pyfect hands to other threads: fx.blocking and fx.try_blocking calls, the live sockets' event loop (one per process, on a thread of its own), DNS lookups and ThreadWorkers, each of which runs a runtime of its own. On the GIL build, Python code on those threads takes turns holding the GIL with the run's thread (waiting on I/O or in C code that releases the GIL doesn't); on 3.14t it runs at the same time.

Sharing state between runs

A program often makes several runs at once: fx.run_blocking from each thread of a thread pool or a threaded web server, ThreadWorkers, a ManagedRuntime per thread. Each run's fibers take turns on its own thread, but runs on different threads go at the same time: on 3.14t in parallel, on the GIL build switching at any bytecode. pyfect's shared-state types can be shared between them, at module level or handed from one run to another, and each operation stays atomic, as it is in Effect, where one thread runs everything:

  • refs: Ref, SynchronizedRef, SubscriptionRef
  • transactions: TxRef and every Tx* collection (see Transactions)
  • coordination: Deferred, Latch, Semaphore, PartitionedSemaphore, Queue, PubSub
  • resources: Pool, Cache, ScopedCache, fx.cached, RcRef, RcMap, ScopedRef, LayerMap, a request resolver's with_cache
  • fibers and scopes: FiberSet, FiberMap, FiberHandle, a Fiber's await_exit, join and interrupt, a scope's finalizers
  • process-wide state: metrics, Reference defaults, MutableHashMap, snowflake generators, the in-memory rate limiter store

A fiber waiting on one of them in one run (a take, a wait, a tx_retry) is woken by a change another run makes, through its own run, even when the other run ends right after.

shared_state.py
import threading

import pyfect as fx

requests = fx.Ref(0)
stock = fx.TxRef(1000)
sold = fx.TxRef(0)


def sell_one(left: int) -> fx.Effect[None]:
    if left == 0:
        return fx.void
    return stock.set(left - 1).and_then(sold.update(lambda n: n + 1))


def handle_requests() -> None:
    # Each call is a run of its own on this thread, as in a threaded web server.
    for _ in range(500):
        fx.run_blocking(requests.update(lambda n: n + 1))
        fx.run_blocking(fx.tx(stock.get().flat_map(sell_one)))


threads = [threading.Thread(target=handle_requests) for _ in range(4)]
for thread in threads:
    thread.start()
for thread in threads:
    thread.join()
print("requests:", fx.run_blocking(requests.get()))
print("sold:", fx.run_blocking(sold.get()), "left:", fx.run_blocking(stock.get()))

What a run starts goes on after its caller returns, as on Effect's scheduler, which is always alive. When fx.run_blocking, fx.run_sync or a ManagedRuntime's sync run returns, the fibers it left waiting move to one background thread that pyfect starts when first needed: a fork, a Cache or RcRef lookup other callers wait on, a FiberSet member, a Pool's refill, an idle timer. Interrupts and scope closes still reach them. So a handler can make a lookup that outlives it, and a ManagedRuntime's sync runs on different threads each return as soon as their own effect is done.

The background thread is one thread for the whole process, so fibers on it take turns as a run's fibers do: keep CPU-heavy work in fx.blocking there too. It is a daemon thread, so the process does not wait for it at exit; to wait for background work, wait for it in a run (join the fiber, await_empty a FiberSet). An asyncio loop runs its own fibers and stops with the loop: a fiber left on a loop that has closed never runs again, and waking it raises LoopClosedError where the wake comes from.

Two limits remain. A plain Python value your own code changes inside fx.sync, a dict or a counter, is not guarded by anything: keep shared values in a Ref. And Ref.update's function runs while the ref is locked, so runs on other threads updating the same ref wait for it: keep it pure and quick, as in Effect.

CPU-heavy work

A fiber that computes for 200 ms holds up every other fiber of the run for 200 ms: timers fire late, sockets aren't read, subscribers fall behind. In an asyncio app it also holds up the app's own coroutines, since they share the loop's thread. Move such work off the run's thread:

  • fx.blocking(f) (or fx.try_blocking) runs f on a pool of daemon threads, up to min(32, cpus + 4) of them, and the fiber waits without blocking the others. Pure-Python work there still takes turns with the run's thread on the GIL build, but the run keeps serving its fibers between turns; on 3.14t it runs in parallel. Interrupting the fiber abandons the thread, it doesn't stop it.
  • Separate processes give real parallelism on either build: a ProcessWorker from workers, or several copies of the program behind a load balancer.
cpu_work.py
import time

import pyfect as fx


def crunch() -> int:
    # Stands in for CPU-heavy work: parsing, hashing, a model, pure-Python loops.
    end = time.perf_counter() + 0.3
    steps = 0
    while time.perf_counter() < end:
        steps += 1
    return steps


@fx.fn
def ticks_while(work: fx.Effect[int]):
    """How often a fiber that wakes every 10 ms got to run while ``work`` ran."""
    ticks: list[float] = []
    heartbeat = fx.sleep("10 millis").tap(
        lambda _: fx.sync(lambda: ticks.append(time.perf_counter()))
    )
    beating = yield from heartbeat.forever().fork()
    yield from fx.sleep("20 millis")
    start = time.perf_counter()
    yield from work
    end = time.perf_counter()
    yield from beating.interrupt()
    return sum(start < tick < end for tick in ticks)


# On the run's thread, nothing else runs until the work returns.
print("inline:", fx.run_blocking(ticks_while(fx.sync(crunch))), "ticks")
# On a worker thread, the run keeps serving its other fibers.
print("fx.blocking: more than 5 ticks:", fx.run_blocking(ticks_while(fx.blocking(crunch))) > 5)

What throughput to expect

Every fiber step is several Python calls, about 15 times what the same step costs in Effect under V8. benchmarks/streams.py in the repository runs the pipelines below; these are its best-of-five numbers on CPython 3.14, with free-threaded 3.14t in brackets and Effect 4.1.2 under Node 24 on the same machine (from docs/PARITY.md):

PipelinepyfectEffect
map + filter, chunks of 409616.5M items/s122M
group_by_key + flat_map, chunks of 2563.68M (3.63M)7.3M
the same, each group cut by grouped_within(100, "1 second")1.97M (1.71M)3.9M
group_by_key + flat_map, chunks of one285k (290k)1.18M
the same with grouped_within195k (206k)1.02M
Queue offer and take, one item at a time528k (387k)5.7M
forking 20k fibers that yield once175k forks/s (220k)2.5M

So plan for millions of items a second where operators work on whole chunks, and a few hundred thousand where each item is its own effect, queue hop or fiber, per run and per core. The free-threaded build is no faster here. To go past one core, split the work across processes.

Rechunk before grouping

Most of what group_by and group_by_key cost is paid per chunk, not per item, so items that arrive in chunks of one pay it for every item. A dogfood app's ingest pipeline measured about 59k items/s with one-item chunks, 238k/s at 5 items a chunk and about 1M/s at 64. Rechunk items that arrive one at a time before grouping them: rechunk(n) holds items until it has n, and grouped_within(n, duration) followed by flatten_iterable() also lets a partial batch go once duration has passed, for a live source that may trickle.

rechunk.py
import pyfect as fx
from pyfect import stream
from pyfect.stream import Stream


def per_key(items: Stream[int]) -> fx.Effect[int]:
    """Groups ``items`` into four keys and counts what the groups read back."""
    groups = items.group_by_key(lambda n: n % 4)
    return groups.flat_map(lambda group: group[1], concurrency="unbounded").run_count()


def chunks(items: Stream[int]) -> int:
    return fx.run_blocking(items.chunks().run_count())


# Items that arrive one at a time, as from a queue fed one item at a time.
arriving = stream.range(1, 1000, chunk_size=1)
print("chunks as they arrive:", chunks(arriving))
print("after rechunk(64):", chunks(arriving.rechunk(64)))
# Same result, but group_by_key now pays its per-chunk cost 16 times instead of 1000.
print(
    "items grouped:",
    fx.run_blocking(per_key(arriving)),
    fx.run_blocking(per_key(arriving.rechunk(64))),
)
# On a live source, rechunk holds items until it has 64; grouped_within also lets a
# partial batch go after 10 ms.
live = arriving.grouped_within(64, "10 millis").flatten_iterable()
print("grouped_within(64, ...):", chunks(live))

Spotting saturation

A run is saturated when its one thread is busy all the time: the process shows one core at 100% while the others idle, whatever the build. The signs inside the program:

  • Subscribers with a "sliding" or "dropping" strategy miss items: a load run of a dogfood app pushing windows to 300 clients lost about 62% of them at fast subscribers, with one core saturated, and none at 30 clients. Count what a strategy drops, as in Publishing to every subscriber, or look for gaps in sequence numbers.
  • Bounded queues stay full and producers spend their time waiting to offer.
  • Timers fire late. A probe fiber that sleeps and measures how late it woke is a cheap gauge to export as a metric:
lag.py
import time

import pyfect as fx


@fx.fn
def worst_lag(load: fx.Effect[None]):
    """The latest a 50 ms timer fired, over ten wakes, while ``load`` runs."""
    lags: list[float] = []

    @fx.gen
    def probe():
        for _ in range(10):
            start = time.perf_counter()
            yield from fx.sleep("50 millis")
            lags.append(time.perf_counter() - start - 0.05)

    loading = yield from load.forever().fork()
    yield from probe
    yield from loading.interrupt()
    return max(lags)


def busy() -> None:
    # 200 ms of CPU on the run's thread between yields.
    end = time.perf_counter() + 0.2
    while time.perf_counter() < end:
        pass


idle = fx.run_blocking(worst_lag(fx.yield_now()))
saturated = fx.run_blocking(worst_lag(fx.sync(busy).and_then(fx.yield_now())))
print("idle: under 50 ms late:", idle < 0.05)
print("saturated: over 100 ms late:", saturated > 0.1)

When a run is saturated, profile for work to batch (chunked operators instead of per-item effects), move CPU-heavy steps to fx.blocking or a worker process, or run more processes.