Run code on other threads or in child processes and talk to it with messages, or with typed RPC calls.
A worker is code running elsewhere, on a thread or in a child process, that the parent
talks to only by messages. Child processes run Python code in parallel with the parent,
which is what CPU-bound work needs. pyfect.workers gives both sides as services:
WorkerPlatform spawns workers in the parent, and WorkerRunnerPlatform starts the
worker's end inside it.
Workers and runners#
python_worker.layer(spawn) provides the parent's platform, with spawn giving the
worker for each id: a ThreadWorker(function) or a ProcessWorker(args) running
python <args>. platform.spawn(id) gives a Worker: send posts it a message and
run(handler) starts it and handles its replies until the scope closes. Inside the
worker, python_worker_runner.layer provides the runner, which handles the parent's
messages with run and answers with send.
import pyfect as fx
from pyfect.queue import Queue
from pyfect.workers import (
ThreadWorker,
WorkerPlatform,
WorkerRunnerPlatform,
python_worker,
python_worker_runner,
)
# The worker's side: answer each message from the parent with its square.
@fx.fn
def serve():
runner = yield from WorkerRunnerPlatform().flat_map(lambda platform: platform.start())
yield from runner.run(lambda port, n: runner.send(port, n * n))
def worker_main() -> None:
fx.run_blocking(serve().pipe(fx.provide(python_worker_runner.layer)).or_die())
# The parent's side: spawn a worker, send it messages, handle its replies.
@fx.fn
def program():
worker = yield from WorkerPlatform().flat_map(lambda platform: platform.spawn(0))
replies = yield from Queue[int].unbounded()
yield from worker.run(replies.offer).fork_scoped() # runs until the scope closes
for n in [2, 3, 4]:
yield from worker.send(n)
print((yield from replies.take_n(3)))
# ThreadWorker runs a function on a thread; ProcessWorker runs `python <args>`.
Workers = python_worker.layer(lambda _id: ThreadWorker(worker_main))
fx.run_main(fx.scoped(program()).pipe(fx.provide(Workers)))A worker that fails to start or exits fails run, and a message that can't be posted
fails send (or run, if it was sent before the worker was ready), with a
WorkerError whose reason is WorkerSpawnError, WorkerSendError,
WorkerReceiveError or WorkerUnknownError. Messages to a process worker are pickled.
RPC over workers#
Most programs want typed calls, not raw messages: serve an RPC group in the
worker and call it from the parent. In the worker, rpc_server.layer(Group) over
rpc_server.layer_protocol_worker_runner serves the handlers. In the parent,
rpc_client.layer_protocol_worker(size=n) is a ClientProtocol over a pool of n
workers, so rpc_client.make(Group) gives the same client as over HTTP, and its calls
are spread over the pool.
import sys
import pyfect as fx
from pyfect import layer
from pyfect import schema as S
from pyfect.rpc import RpcGroup, RpcRequest, make
from pyfect.rpc import client as rpc_client
from pyfect.rpc import server as rpc_server
from pyfect.workers import ProcessWorker, python_worker, python_worker_runner
class Primes(RpcGroup):
count_primes = make("CountPrimes", payload=S.Int, success=S.Int)
def count_primes(request: RpcRequest[int]) -> fx.Effect[int]:
"""CPU-bound work: it runs in a child process, so it doesn't hold this one's GIL."""
below = request.payload
sieve = bytearray([1]) * below
sieve[:2] = b"\0\0"
for i in range(2, int(below**0.5) + 1):
if sieve[i]:
sieve[i * i :: i] = bytes(len(sieve[i * i :: i]))
return fx.succeed(sum(sieve))
PrimesLive = Primes.to_layer(Primes.handlers().handle(Primes.count_primes)(count_primes))
# The worker's program: serve the group to the parent until it closes the worker.
WorkerMain = (
rpc_server.layer(Primes)
.pipe(layer.provide(PrimesLive))
.pipe(layer.provide(rpc_server.layer_protocol_worker_runner))
.pipe(layer.provide(python_worker_runner.layer))
)
@fx.fn
def program():
primes = yield from rpc_client.make(Primes)
counts = yield from fx.for_each(
[10, 1_000, 100_000, 1_000_000], primes.count_primes, concurrency=2
)
print(counts)
# A pool of two worker processes, each running this file with the argument "worker".
Workers = rpc_client.layer_protocol_worker(size=2).pipe(
layer.provide(python_worker.layer(lambda _id: ProcessWorker([__file__, "worker"])))
)
if sys.argv[1:] == ["worker"]:
fx.run_blocking_exit(WorkerMain.launch())
else:
fx.run_main(fx.scoped(program()).pipe(fx.provide(Workers)))A pool can also grow and shrink: min_size, max_size and time_to_live in place of
size. rpc.worker.layer_initial_message(schema, build) sends each worker a value before
any request, such as its settings, which the worker reads with
rpc.worker.initial_message(schema).
Reference: pyfect.workers,
python_worker,
pyfect.rpc.worker.