GitHub

Workers

Run code on other threads or in child processes and talk to it with messages, or with typed RPC calls.

A worker is code running elsewhere, on a thread or in a child process, that the parent talks to only by messages. Child processes run Python code in parallel with the parent, which is what CPU-bound work needs. pyfect.workers gives both sides as services: WorkerPlatform spawns workers in the parent, and WorkerRunnerPlatform starts the worker's end inside it.

Workers and runners

python_worker.layer(spawn) provides the parent's platform, with spawn giving the worker for each id: a ThreadWorker(function) or a ProcessWorker(args) running python <args>. platform.spawn(id) gives a Worker: send posts it a message and run(handler) starts it and handles its replies until the scope closes. Inside the worker, python_worker_runner.layer provides the runner, which handles the parent's messages with run and answers with send.

squares.py
import pyfect as fx
from pyfect.queue import Queue
from pyfect.workers import (
    ThreadWorker,
    WorkerPlatform,
    WorkerRunnerPlatform,
    python_worker,
    python_worker_runner,
)


# The worker's side: answer each message from the parent with its square.
@fx.fn
def serve():
    runner = yield from WorkerRunnerPlatform().flat_map(lambda platform: platform.start())
    yield from runner.run(lambda port, n: runner.send(port, n * n))


def worker_main() -> None:
    fx.run_blocking(serve().pipe(fx.provide(python_worker_runner.layer)).or_die())


# The parent's side: spawn a worker, send it messages, handle its replies.
@fx.fn
def program():
    worker = yield from WorkerPlatform().flat_map(lambda platform: platform.spawn(0))
    replies = yield from Queue[int].unbounded()
    yield from worker.run(replies.offer).fork_scoped()  # runs until the scope closes
    for n in [2, 3, 4]:
        yield from worker.send(n)
    print((yield from replies.take_n(3)))


# ThreadWorker runs a function on a thread; ProcessWorker runs `python <args>`.
Workers = python_worker.layer(lambda _id: ThreadWorker(worker_main))
fx.run_main(fx.scoped(program()).pipe(fx.provide(Workers)))

A worker that fails to start or exits fails run, and a message that can't be posted fails send (or run, if it was sent before the worker was ready), with a WorkerError whose reason is WorkerSpawnError, WorkerSendError, WorkerReceiveError or WorkerUnknownError. Messages to a process worker are pickled.

RPC over workers

Most programs want typed calls, not raw messages: serve an RPC group in the worker and call it from the parent. In the worker, rpc_server.layer(Group) over rpc_server.layer_protocol_worker_runner serves the handlers. In the parent, rpc_client.layer_protocol_worker(size=n) is a ClientProtocol over a pool of n workers, so rpc_client.make(Group) gives the same client as over HTTP, and its calls are spread over the pool.

RequestmessagemessageExitExitrpc_client.make(Primes)layer_protocol_worker: a pool ofworkersworker process 1: Primeshandlersworker process 2: Primeshandlers
RequestmessagemessageExitExitrpc_client.make(Primes)layer_protocol_worker: a pool ofworkersworker process 1: Primeshandlersworker process 2: Primeshandlers
primes.py
import sys

import pyfect as fx
from pyfect import layer
from pyfect import schema as S
from pyfect.rpc import RpcGroup, RpcRequest, make
from pyfect.rpc import client as rpc_client
from pyfect.rpc import server as rpc_server
from pyfect.workers import ProcessWorker, python_worker, python_worker_runner


class Primes(RpcGroup):
    count_primes = make("CountPrimes", payload=S.Int, success=S.Int)


def count_primes(request: RpcRequest[int]) -> fx.Effect[int]:
    """CPU-bound work: it runs in a child process, so it doesn't hold this one's GIL."""
    below = request.payload
    sieve = bytearray([1]) * below
    sieve[:2] = b"\0\0"
    for i in range(2, int(below**0.5) + 1):
        if sieve[i]:
            sieve[i * i :: i] = bytes(len(sieve[i * i :: i]))
    return fx.succeed(sum(sieve))


PrimesLive = Primes.to_layer(Primes.handlers().handle(Primes.count_primes)(count_primes))

# The worker's program: serve the group to the parent until it closes the worker.
WorkerMain = (
    rpc_server.layer(Primes)
    .pipe(layer.provide(PrimesLive))
    .pipe(layer.provide(rpc_server.layer_protocol_worker_runner))
    .pipe(layer.provide(python_worker_runner.layer))
)


@fx.fn
def program():
    primes = yield from rpc_client.make(Primes)
    counts = yield from fx.for_each(
        [10, 1_000, 100_000, 1_000_000], primes.count_primes, concurrency=2
    )
    print(counts)


# A pool of two worker processes, each running this file with the argument "worker".
Workers = rpc_client.layer_protocol_worker(size=2).pipe(
    layer.provide(python_worker.layer(lambda _id: ProcessWorker([__file__, "worker"])))
)

if sys.argv[1:] == ["worker"]:
    fx.run_blocking_exit(WorkerMain.launch())
else:
    fx.run_main(fx.scoped(program()).pipe(fx.provide(Workers)))

A pool can also grow and shrink: min_size, max_size and time_to_live in place of size. rpc.worker.layer_initial_message(schema, build) sends each worker a value before any request, such as its settings, which the worker reads with rpc.worker.initial_message(schema).

Reference: pyfect.workers, python_worker, pyfect.rpc.worker.