All of a run's fibers share one thread: what that means for CPU-heavy work, free-threaded Python and the throughput to expect.
Fibers are cheap and many of them can be waiting at once, but they don't run in parallel. Every
fiber of a run executes on that run's thread: the thread that called fx.run_blocking (until
it returns; see below), or the asyncio event loop's thread for
run_async and a ManagedRuntime used from an asyncio app.
A fiber runs until it waits (a sleep, a queue, a socket, fx.blocking) or uses up its step
budget, then the next one runs, as in Effect on Node's single thread.
import threading
import pyfect as fx
def thread_of(_: int) -> fx.Effect[int]:
# Each fiber yields once, so they interleave, and reports the thread it ran on.
return fx.yield_now().map(lambda _: threading.get_ident())
@fx.gen
def program():
fibers = yield from fx.for_each(range(100), thread_of, concurrency="unbounded")
elsewhere = yield from fx.blocking(threading.get_ident)
here = threading.get_ident()
print("threads the 100 fibers ran on:", len(set(fibers)))
print("the run's thread:", set(fibers) == {here})
print("fx.blocking ran elsewhere:", elsewhere != here)
fx.run_blocking(program)Free-threaded Python#
Free-threaded CPython (3.14t) removes the GIL, but it doesn't make fibers faster: they still
take turns on one thread. What it speeds up is work pyfect hands to other threads:
fx.blocking and fx.try_blocking calls, the live sockets' event loop (one per process, on a
thread of its own), DNS lookups and ThreadWorkers, each of which runs a runtime of its own. On
the GIL build, Python code on those threads takes turns holding the GIL with the run's thread
(waiting on I/O or in C code that releases the GIL doesn't); on 3.14t it runs at the same time.
Sharing state between runs#
A program often makes several runs at once: fx.run_blocking from each thread of a thread
pool or a threaded web server, ThreadWorkers, a ManagedRuntime per thread. Each run's
fibers take turns on its own thread, but runs on different threads go at the same time: on
3.14t in parallel, on the GIL build switching at any bytecode. pyfect's shared-state types
can be shared between them, at module level or handed from one run to another, and each
operation stays atomic, as it is in Effect, where one thread runs everything:
- refs:
Ref,SynchronizedRef,SubscriptionRef - transactions:
TxRefand everyTx*collection (see Transactions) - coordination:
Deferred,Latch,Semaphore,PartitionedSemaphore,Queue,PubSub - resources:
Pool,Cache,ScopedCache,fx.cached,RcRef,RcMap,ScopedRef,LayerMap, a request resolver'swith_cache - fibers and scopes:
FiberSet,FiberMap,FiberHandle, aFiber'sawait_exit,joinandinterrupt, a scope's finalizers - process-wide state: metrics,
Referencedefaults,MutableHashMap, snowflake generators, the in-memory rate limiter store
A fiber waiting on one of them in one run (a take, a wait, a tx_retry) is woken by a
change another run makes, through its own run, even when the other run ends right after.
import threading
import pyfect as fx
requests = fx.Ref(0)
stock = fx.TxRef(1000)
sold = fx.TxRef(0)
def sell_one(left: int) -> fx.Effect[None]:
if left == 0:
return fx.void
return stock.set(left - 1).and_then(sold.update(lambda n: n + 1))
def handle_requests() -> None:
# Each call is a run of its own on this thread, as in a threaded web server.
for _ in range(500):
fx.run_blocking(requests.update(lambda n: n + 1))
fx.run_blocking(fx.tx(stock.get().flat_map(sell_one)))
threads = [threading.Thread(target=handle_requests) for _ in range(4)]
for thread in threads:
thread.start()
for thread in threads:
thread.join()
print("requests:", fx.run_blocking(requests.get()))
print("sold:", fx.run_blocking(sold.get()), "left:", fx.run_blocking(stock.get()))What a run starts goes on after its caller returns, as on Effect's scheduler, which is always
alive. When fx.run_blocking, fx.run_sync or a ManagedRuntime's sync run returns, the
fibers it left waiting move to one background thread that pyfect starts when first needed: a
fork, a Cache or RcRef lookup other callers wait on, a FiberSet member, a Pool's
refill, an idle timer. Interrupts and scope closes still reach them. So a handler can make a
lookup that outlives it, and a ManagedRuntime's sync runs on different threads each return
as soon as their own effect is done.
The background thread is one thread for the whole process, so fibers on it take turns as a
run's fibers do: keep CPU-heavy work in fx.blocking there too. It is a daemon thread, so the
process does not wait for it at exit; to wait for background work, wait for it in a run (join
the fiber, await_empty a FiberSet). An asyncio loop runs its own fibers and stops with the
loop: a fiber left on a loop that has closed never runs again, and waking it raises
LoopClosedError where the wake comes from.
Two limits remain. A plain Python value your own code changes inside fx.sync, a dict or a counter,
is not guarded by anything: keep shared values in a Ref. And Ref.update's function runs
while the ref is locked, so runs on other threads updating the same ref wait for it: keep it
pure and quick, as in Effect.
CPU-heavy work#
A fiber that computes for 200 ms holds up every other fiber of the run for 200 ms: timers fire late, sockets aren't read, subscribers fall behind. In an asyncio app it also holds up the app's own coroutines, since they share the loop's thread. Move such work off the run's thread:
fx.blocking(f)(orfx.try_blocking) runsfon a pool of daemon threads, up tomin(32, cpus + 4)of them, and the fiber waits without blocking the others. Pure-Python work there still takes turns with the run's thread on the GIL build, but the run keeps serving its fibers between turns; on 3.14t it runs in parallel. Interrupting the fiber abandons the thread, it doesn't stop it.- Separate processes give real parallelism on either build: a
ProcessWorkerfrom workers, or several copies of the program behind a load balancer.
import time
import pyfect as fx
def crunch() -> int:
# Stands in for CPU-heavy work: parsing, hashing, a model, pure-Python loops.
end = time.perf_counter() + 0.3
steps = 0
while time.perf_counter() < end:
steps += 1
return steps
@fx.fn
def ticks_while(work: fx.Effect[int]):
"""How often a fiber that wakes every 10 ms got to run while ``work`` ran."""
ticks: list[float] = []
heartbeat = fx.sleep("10 millis").tap(
lambda _: fx.sync(lambda: ticks.append(time.perf_counter()))
)
beating = yield from heartbeat.forever().fork()
yield from fx.sleep("20 millis")
start = time.perf_counter()
yield from work
end = time.perf_counter()
yield from beating.interrupt()
return sum(start < tick < end for tick in ticks)
# On the run's thread, nothing else runs until the work returns.
print("inline:", fx.run_blocking(ticks_while(fx.sync(crunch))), "ticks")
# On a worker thread, the run keeps serving its other fibers.
print("fx.blocking: more than 5 ticks:", fx.run_blocking(ticks_while(fx.blocking(crunch))) > 5)What throughput to expect#
Every fiber step is several Python calls, about 15 times what the same step costs in Effect
under V8. benchmarks/streams.py in the repository runs the pipelines below; these are its
best-of-five numbers on CPython 3.14, with free-threaded 3.14t in brackets and Effect 4.1.2
under Node 24 on the same machine (from docs/PARITY.md):
| Pipeline | pyfect | Effect |
|---|---|---|
map + filter, chunks of 4096 | 16.5M items/s | 122M |
group_by_key + flat_map, chunks of 256 | 3.68M (3.63M) | 7.3M |
the same, each group cut by grouped_within(100, "1 second") | 1.97M (1.71M) | 3.9M |
group_by_key + flat_map, chunks of one | 285k (290k) | 1.18M |
the same with grouped_within | 195k (206k) | 1.02M |
Queue offer and take, one item at a time | 528k (387k) | 5.7M |
| forking 20k fibers that yield once | 175k forks/s (220k) | 2.5M |
So plan for millions of items a second where operators work on whole chunks, and a few hundred thousand where each item is its own effect, queue hop or fiber, per run and per core. The free-threaded build is no faster here. To go past one core, split the work across processes.
Rechunk before grouping#
Most of what group_by and group_by_key cost is paid per chunk, not per item, so items that
arrive in chunks of one pay it for every item. A dogfood app's ingest
pipeline measured about 59k items/s with one-item chunks, 238k/s at 5 items a chunk and about
1M/s at 64. Rechunk items that arrive one at a time before grouping them: rechunk(n) holds
items until it has n, and grouped_within(n, duration) followed by flatten_iterable() also
lets a partial batch go once duration has passed, for a live source that may trickle.
import pyfect as fx
from pyfect import stream
from pyfect.stream import Stream
def per_key(items: Stream[int]) -> fx.Effect[int]:
"""Groups ``items`` into four keys and counts what the groups read back."""
groups = items.group_by_key(lambda n: n % 4)
return groups.flat_map(lambda group: group[1], concurrency="unbounded").run_count()
def chunks(items: Stream[int]) -> int:
return fx.run_blocking(items.chunks().run_count())
# Items that arrive one at a time, as from a queue fed one item at a time.
arriving = stream.range(1, 1000, chunk_size=1)
print("chunks as they arrive:", chunks(arriving))
print("after rechunk(64):", chunks(arriving.rechunk(64)))
# Same result, but group_by_key now pays its per-chunk cost 16 times instead of 1000.
print(
"items grouped:",
fx.run_blocking(per_key(arriving)),
fx.run_blocking(per_key(arriving.rechunk(64))),
)
# On a live source, rechunk holds items until it has 64; grouped_within also lets a
# partial batch go after 10 ms.
live = arriving.grouped_within(64, "10 millis").flatten_iterable()
print("grouped_within(64, ...):", chunks(live))Spotting saturation#
A run is saturated when its one thread is busy all the time: the process shows one core at 100% while the others idle, whatever the build. The signs inside the program:
- Subscribers with a
"sliding"or"dropping"strategy miss items: a load run of a dogfood app pushing windows to 300 clients lost about 62% of them at fast subscribers, with one core saturated, and none at 30 clients. Count what a strategy drops, as in Publishing to every subscriber, or look for gaps in sequence numbers. - Bounded queues stay full and producers spend their time waiting to offer.
- Timers fire late. A probe fiber that sleeps and measures how late it woke is a cheap gauge to export as a metric:
import time
import pyfect as fx
@fx.fn
def worst_lag(load: fx.Effect[None]):
"""The latest a 50 ms timer fired, over ten wakes, while ``load`` runs."""
lags: list[float] = []
@fx.gen
def probe():
for _ in range(10):
start = time.perf_counter()
yield from fx.sleep("50 millis")
lags.append(time.perf_counter() - start - 0.05)
loading = yield from load.forever().fork()
yield from probe
yield from loading.interrupt()
return max(lags)
def busy() -> None:
# 200 ms of CPU on the run's thread between yields.
end = time.perf_counter() + 0.2
while time.perf_counter() < end:
pass
idle = fx.run_blocking(worst_lag(fx.yield_now()))
saturated = fx.run_blocking(worst_lag(fx.sync(busy).and_then(fx.yield_now())))
print("idle: under 50 ms late:", idle < 0.05)
print("saturated: over 100 ms late:", saturated > 0.1)When a run is saturated, profile for work to batch (chunked operators instead of per-item
effects), move CPU-heavy steps to fx.blocking or a worker process, or run more processes.