GitHub

Cluster

Entities sharded over runners that talk over sockets or HTTP, with persisted messages, shard locks in SQL, singletons, cron jobs that run once per cluster and durable workflows.

Effect's cluster runs entities, small stateful services addressed by a type and an id, across a group of processes called runners. Each entity id hashes to one of a fixed number of shards, and each shard belongs to one runner at a time, so a message for an entity reaches the one runner that holds it. Messages can be persisted, so a request survives a runner restarting, and runners take shards over from each other when one leaves. pyfect.cluster follows Effect's effect/cluster; import it as a submodule.

Entities

An entity type is a subclass of Entity whose attributes are RPCs. Its handlers are built when an entity id gets its first message, and that instance keeps running, holding its state, until it has been idle for max_idle_time. An entity handles one message at a time unless to_layer is given a concurrency. Counter.client() is an effect giving a function from an entity id to a client whose methods are the RPCs, with payload, result and error typed from their schemas; it works the same from any runner. class Counter(Entity[Literal["Increment", "Get"]]) lists its RPCs' tags, so pyright rejects Counter.to_layer(handlers) while one has no handler, as with an RPC group; unlisted, to_layer raises a ValueError for it.

entities.py
from typing import Literal, TypedDict

import pyfect as fx
from pyfect import schema as S
from pyfect.cluster import test_runner
from pyfect.cluster.cluster_schema import Persisted
from pyfect.cluster.entity import CurrentAddress, Entity, Request
from pyfect.rpc import make as make_rpc


class Amount(TypedDict):
    by: int


# An entity type is a class of RPCs; listing their tags lets pyright check that each
# has a handler. Persisted RPCs are stored before they run, so a request survives its
# runner restarting; `get` is volatile.
class Counter(
    Entity[Literal["Increment", "Get"]],
    type="Counter",
    rpc_annotations=fx.Context.make(Persisted, True),
):
    increment = make_rpc("Increment", payload=S.Struct(Amount), success=S.Int)
    get = make_rpc("Get", success=S.Int).annotate(Persisted, False)


# The handlers are built once per entity id, when its first message arrives,
# so state made here belongs to that one counter.
@fx.gen
def counter():
    address = yield from CurrentAddress()
    print("start", address.entity_id)
    total = [0]

    def increment(request: Request[Amount]) -> fx.Effect[int]:
        def add() -> int:
            total[0] += request.payload["by"]
            return total[0]

        return fx.sync(add)

    return (
        Counter.handlers()
        .handle(Counter.increment)(increment)
        .handle(Counter.get)(lambda _: fx.sync(lambda: total[0]))
    )


@fx.gen
def program():
    make_client = yield from Counter.client()
    a, b = make_client("a"), make_client("b")
    yield from a.increment({"by": 2})
    print("a:", (yield from a.increment({"by": 3})))
    print("b:", (yield from b.increment({"by": 10})))
    print("a:", (yield from a.get()))


# `test_runner.layer` is one runner in memory: sharding, runners and message storage.
cluster = Counter.to_layer_effect(counter).pipe(fx.layer.provide_merge(test_runner.layer))
fx.run_main(program.pipe(fx.provide(cluster)))

An RPC annotated Persisted is stored before it runs and its reply is stored when it finishes: a sender waits on the stored reply, a request with the same primary key is answered once, and a request a runner had not finished when it stopped runs again on the runner that takes its shard over. A volatile RPC goes straight to the runner and is not stored: if that runner dies before answering, the sender sends it again to the runner that takes the shard, so its handler can run twice, as in Effect. Keep volatile handlers safe to repeat, or annotate the RPC Persisted. Handlers read their own address from CurrentAddress and their runner's from CurrentRunnerAddress. to_layer takes the handlers directly, to_layer_effect the effect that builds them, and to_layer_queue gives a mailbox to read messages from instead. entity.make_test_client calls an entity's handlers without a cluster, for unit tests. It reads ShardingConfig to place each entity id on a shard, so provide sharding_config.layer(), as Effect's makeTestClient needs it too.

Shards and runners

A ShardId is a shard group and a number. ShardingConfig sets shards_per_group (300, as in Effect) and the groups a runner serves. Keep it the same for the life of a cluster: an entity's shard comes from its id's hash modulo that count. A Runner has a RunnerAddress, its shard groups and a weight. Shards are assigned to runners with pyfect.hash_ring.HashRing, a consistent-hashing ring with Effect's exact hash, so a pyfect ring assigns shards as Effect's would. A runner with weight 2 gets twice the shards.

hash mod shards_per_groupentity idShardId(group, n)HashRing of runner addressesrunner-arunner-c: weight 2shard lock in RunnerStorage
hash mod shards_per_groupentity idShardId(group, n)HashRing of runner addressesrunner-arunner-c: weight 2shard lock in RunnerStorage
shards.py
from collections import Counter

import pyfect as fx
from pyfect.cluster import MachineId, runner_address, sharding_config, snowflake
from pyfect.hash_ring import HashRing

shards = sharding_config.DEFAULTS.shards_per_group
print("shards per group:", shards)

# Shards are spread over runners by consistent hashing, weighted per runner.
ring = HashRing[runner_address.RunnerAddress]()
a, b, c = (runner_address.make(f"runner-{n}", 34431) for n in "abc")
ring.add_many([a, b])
ring.add(c, weight=2)


def assign() -> list[runner_address.RunnerAddress]:
    return ring.get_shards(shards) or []  # None when the ring is empty


before = assign()
print(sorted(Counter(r.host for r in before).items()))

# When a runner leaves, its shards are spread over the others, weight kept.
ring.remove(b)
after = assign()
print(sorted(Counter(r.host for r in after).items()))
print("moved:", sum(x != y for x, y in zip(before, after, strict=True)))


# Snowflakes: ids ordered by time, unique per machine, for messages and requests.
@fx.fn
def ids():
    generator = yield from snowflake.Generator()
    yield from generator.set_machine_id(MachineId(7))
    first, second = generator.next_unsafe(), generator.next_unsafe()
    print(first < second, snowflake.machine_id(first), snowflake.sequence(first))


fx.run_main(ids().pipe(fx.provide(snowflake.layer_generator)))

A HashRing works outside a cluster too. ring.get(key) places a key at the node nearest its hash, so it is only as even as the node keys hash: two nodes named a and b split 600 keys 97 / 503, as Effect's ring does. To split keys evenly, do what the cluster does: hash each key to one of a fixed number of shards and let get_shards place the shards.

hash_ring.py
import hashlib
from collections import Counter
from dataclasses import dataclass

from pyfect.hash_ring import HashRing


@dataclass(frozen=True)
class Node:
    name: str

    def primary_key(self) -> str:
        return self.name


ring = HashRing[Node]()
ring.add_many([Node("a"), Node("b")])
jobs = [f"job-{i}" for i in range(600)]

# `get` places each key at the node nearest its hash: only as even as the
# node names hash. Short names like these split 600 keys very unevenly
# (Effect's HashRing gives the same counts).
by_get = Counter(node.name for job in jobs if (node := ring.get(job)))
print("get:", sorted(by_get.items()))

# Balanced: hash each key to one of a fixed number of shards, and let the ring
# place the shards. This is what the cluster does with its entities.
SHARDS = 300
shards = ring.get_shards(SHARDS) or []


def shard_of(key: str) -> int:
    return int.from_bytes(hashlib.sha256(key.encode()).digest()[:8], "big") % SHARDS


print("shards:", sorted(Counter(node.name for node in shards).items()))
by_shard = Counter(shards[shard_of(job)].name for job in jobs)
print("keys by shard:", sorted(by_shard.items()))

sharding_config.layer(settings) provides settings (start from dataclasses.replace( sharding_config.DEFAULTS, ...)), and layer_from_env() reads them from the environment in CONSTANT_CASE (SHARDS_PER_GROUP, HOST, PORT), with Effect's defaults for the rest. Durations are duration.Duration values.

Running a cluster

Sharding is the service at the center of a runner: it registers entities and singletons, takes shard locks, routes each message to the runner holding its shard and retries when a shard moves. Three layers build it with everything it needs:

  • test_runner.layer: one runner in memory, for tests and examples. It takes its shards in background fibers once the layer is built. A request sent before they ran is retried send_retry_interval (100 milliseconds) later, so under the test clock it waits until the test moves time by that much; a request sent after a yield (fx.yield_now()) or from a forked fiber answers at once. Effect's TestRunner does the same.
  • single_runner.layer(...): one runner whose messages are in SQL, for a single process that should still survive restarts.
  • cluster_socket.layer(...) and cluster_http.layer(transport="http" | "websocket", ...): a runner that listens on its runner_address and talks to the others over TCP, HTTP or WebSockets, with messages and shard locks in SQL (storage="sql", the default), in memory (storage="local") or in storage layers you provide (storage="byo"). Import them on their own (from pyfect.cluster import cluster_socket), as Effect keeps them in platform-node.

single_runner.layer() needs only a SqlClient; sharding_config= updates the settings it reads from the environment, and runner_storage_kind="memory" keeps the shard locks in memory. Here a workflow engine runs on it over a SQLite file, and a second run on the same file reads the stored result:

single_runner.py
import tempfile
from pathlib import Path
from typing import TypedDict

import pyfect as fx
from pyfect import reactivity
from pyfect import schema as S
from pyfect.cluster import cluster_workflow_engine, single_runner
from pyfect.sql import sqlite
from pyfect.sql.client import SqlClient
from pyfect.workflow import activity, workflow


class Greeting(TypedDict):
    name: str


Greet = workflow.make(
    "Greet", payload=S.Struct(Greeting), success=S.String, idempotency_key=lambda g: g["name"]
)


def compose(name: str) -> str:
    print("composing for", name)
    return f"hello, {name}"


@fx.fn
def greet(greeting: Greeting, _execution_id: str):
    return (
        yield from activity.make(
            name="greet", success=S.String, execute=fx.sync(lambda: compose(greeting["name"]))
        )
    )


def app(database: Path):
    sql = fx.layer.effect(
        SqlClient, sqlite.make(sqlite.SqliteClientConfig(filename=str(database)))
    ).pipe(fx.layer.provide(reactivity.layer))
    # One runner whose messages and shard locks are in SQL, so executions survive a restart.
    return (
        Greet.to_layer(greet)
        .pipe(fx.layer.provide_merge(cluster_workflow_engine.layer))
        .pipe(fx.layer.provide_merge(single_runner.layer()))
        .pipe(fx.layer.provide(sql))
        .or_die()
    )


with tempfile.TemporaryDirectory() as directory:
    database = Path(directory) / "cluster.db"
    print(fx.run_blocking(Greet.execute({"name": "ada"}).pipe(fx.provide(app(database)))))
    # A new run on the same database reads the stored result: nothing is composed again.
    print(fx.run_blocking(Greet.execute({"name": "ada"}).pipe(fx.provide(app(database)))))
runner-brunner-brunner-arunner-aCounter b: Incrementsocket or HTTPstore, lockstore, lockclient on runner-aa's shardCounter ab's shardCounter bSQL: messages, shard locks
runner-brunner-brunner-arunner-aCounter b: Incrementsocket or HTTPstore, lockstore, lockclient on runner-aa's shardCounter ab's shardCounter bSQL: messages, shard locks

Every runner of a cluster points at the same database. When a runner joins, the hash ring gives it a share of the shards, and the runners holding them release their locks; when one leaves, or stops renewing its locks, the others take its shards and run its unfinished persisted messages. Here two runners in one process share a SQLite file:

runners.py
import socket
import tempfile
from dataclasses import replace
from pathlib import Path

import pyfect as fx
from pyfect import reactivity
from pyfect import schema as S
from pyfect.cluster import cluster_socket, runner_address
from pyfect.cluster.entity import CurrentRunnerAddress, Entity
from pyfect.cluster.entity_id import EntityId
from pyfect.cluster.sharding import Sharding
from pyfect.cluster.sharding_config import ShardingConfigService
from pyfect.rpc import make as make_rpc
from pyfect.scope import Scope
from pyfect.sql import sqlite
from pyfect.sql.client import SqlClient


class Where(Entity, type="Where"):
    port = make_rpc("Port", success=S.Int)


@fx.gen
def where():
    address = yield from CurrentRunnerAddress()  # the runner this entity lives on
    return Where.handlers().handle(Where.port)(lambda _: fx.succeed(address.port))


def free_ports(count: int) -> list[int]:
    # Hold every probe open until all are bound, so the ports differ.
    probes = [socket.socket() for _ in range(count)]
    for probe in probes:
        probe.bind(("127.0.0.1", 0))
    ports = [probe.getsockname()[1] for probe in probes]
    for probe in probes:
        probe.close()
    return ports


def runner(database: Path, port: int):
    # Each runner listens on its own address and shares one database with the others:
    # messages and shard locks live there.
    def settings(config: ShardingConfigService) -> ShardingConfigService:
        return replace(
            config,
            runner_address=runner_address.make("127.0.0.1", port),
            refresh_assignments_interval=fx.duration.millis(200),
            entity_termination_timeout=fx.duration.zero,
        )

    sql = fx.layer.effect(
        SqlClient, sqlite.make(sqlite.SqliteClientConfig(filename=str(database)))
    ).pipe(fx.layer.provide(reactivity.layer))
    return (
        Where.to_layer_effect(where)
        .pipe(fx.layer.provide_merge(cluster_socket.layer(sharding_config=settings)))
        .pipe(fx.layer.provide(sql))
    )


def owns_shards(context: fx.Context[Sharding]) -> bool:
    sharding = context.get_unsafe(Sharding)
    return any(
        sharding.has_shard_id(sharding.get_shard_id(EntityId(f"probe-{i}"), "default"))
        for i in range(100)
    )


@fx.fn
def program(database: Path):
    ports = free_ports(2)
    scope = yield from Scope()
    # Two runners in one process here; in production, one per process or machine.
    first = yield from runner(database, ports[0]).build_with_scope(scope)
    second = yield from runner(database, ports[1]).build_with_scope(scope)
    while not (owns_shards(first) and owns_shards(second)):
        yield from fx.sleep("100 millis")

    # A client on the first runner reaches every entity, wherever its shard is.
    make_client = yield from Where.client().pipe(fx.provide(first))
    answers = yield from fx.for_each(
        [make_client(f"user-{i}").port() for i in range(40)], lambda call: call
    )
    by_runner = {"first" if port == ports[0] else "second" for port in answers}
    print("runners that answered:", sorted(by_runner))


with tempfile.TemporaryDirectory() as directory:
    # No "Listening on", and no "No healthy runners available" from a runner that syncs
    # while the other shuts down.
    quiet = fx.provide_service(fx.MinimumLogLevel, fx.LogLevel.ERROR)
    fx.run_main(fx.scoped(program(Path(directory) / "cluster.db")).pipe(quiet))

A runner that stops on purpose (its scope closes, as on a deploy) drains before it hands over. It lets the requests its entities are running finish for up to entity_termination_timeout (15 seconds, as in Effect, to fit Kubernetes' grace period), and only then releases their shards. So a job that is running when its runner stops usually finishes on that runner and is not run again elsewhere. A request still running at the timeout is interrupted, and a persisted one then runs again on the runner that takes its shard. With a timeout of 0 the runner hands over at once, as runners.py above does so its example stops quickly. An RPC annotated cluster_schema.InterruptOnTermination is interrupted when the drain starts instead of waited for. A runner that dies (kill -9) does not drain: the others take its shards once they see it is gone, and run its unfinished persisted requests. They see it is gone when its heartbeat and shard locks are older than shard_lock_expiration (35 seconds, as in Effect) and their next assignment check runs, so with the default settings its shards move about 35 to 55 seconds after it dies. A lower shard_lock_expiration (kept well above shard_lock_refresh_interval, 10 seconds) moves them sooner. A runner that was paused (a frozen process, a long GC) may find its shards already taken when it wakes. Before it stores a persisted request's reply it refreshes that shard's lock, and if the lock is gone it leaves the request to the shard's new owner.

When several runners join at once, each join would move shards again. runner_join_debounce (0 by default, as in Effect) waits until the set of healthy runners has stayed the same for that long before giving shards to new runners, so each shard moves once. A runner that becomes unhealthy still loses its shards at once, and joins wait at most four times the debounce.

drain.py
import socket
import tempfile
from dataclasses import replace
from pathlib import Path

import pyfect as fx
from pyfect import exit as Exit
from pyfect import reactivity
from pyfect import schema as S
from pyfect.cluster import cluster_socket, runner_address
from pyfect.cluster.cluster_schema import Persisted
from pyfect.cluster.entity import Entity
from pyfect.cluster.sharding_config import ShardingConfigService
from pyfect.latch import Latch
from pyfect.rpc import make as make_rpc
from pyfect.sql import sqlite
from pyfect.sql.client import SqlClient


class Job(Entity, type="Job", rpc_annotations=fx.Context.make(Persisted, True)):
    run = make_rpc("Run", success=S.String)


def free_port() -> int:
    with socket.socket() as probe:
        probe.bind(("127.0.0.1", 0))
        return probe.getsockname()[1]


def runner(database: Path, termination_timeout: fx.duration.Input, started: Latch, log: list[str]):
    def run(_: object) -> fx.Effect[str]:
        work = fx.sync(started.open_unsafe).and_then(
            fx.sleep("500 millis")
        )  # half a second of work
        return work.and_then(fx.sync(lambda: log.append("job finished"))).as_("done")

    port = free_port()

    def settings(config: ShardingConfigService) -> ShardingConfigService:
        return replace(
            config,
            runner_address=runner_address.make("127.0.0.1", port),
            # How long a stopping runner waits for its entities (15 s by default).
            entity_termination_timeout=fx.duration.from_input_unsafe(termination_timeout),
        )

    sql = fx.layer.effect(
        SqlClient, sqlite.make(sqlite.SqliteClientConfig(filename=str(database)))
    ).pipe(fx.layer.provide(reactivity.layer))
    return (
        Job.to_layer(Job.handlers().handle(Job.run)(run))
        .pipe(fx.layer.provide_merge(cluster_socket.layer(sharding_config=settings)))
        .pipe(fx.layer.provide(sql))
    )


@fx.fn
def stop_during_a_job(database: Path, termination_timeout: fx.duration.Input):
    log: list[str] = []
    started = Latch()
    scope = yield from fx.scope.make()
    context = yield from runner(database, termination_timeout, started, log).build_with_scope(scope)
    make_client = yield from Job.client().pipe(fx.provide(context))
    yield from make_client("report").run().fork()
    yield from started.wait()

    # Stop the runner while the job runs.
    yield from scope.close(Exit.void)
    log.append("runner stopped")
    return log


@fx.fn
def program(directory: Path):
    # The runner drains: the job finishes here before the shard is handed over.
    drained = yield from stop_during_a_job(directory / "drain.db", "5 seconds")
    assert drained == ["job finished", "runner stopped"]
    # With 0 it hands over at once: the job stops here, and its persisted
    # request runs again on the runner that takes the shard.
    handed_over = yield from stop_during_a_job(directory / "handover.db", 0)
    assert handed_over == ["runner stopped"]


with tempfile.TemporaryDirectory() as directory:
    quiet = fx.provide_service(fx.MinimumLogLevel, fx.LogLevel.WARN)  # no "Listening on"
    fx.run_main(program(Path(directory)).pipe(quiet))

sharding_config takes a function that updates the settings read from the environment. client_only=True makes a runner that sends messages but holds no shards, for a web server in front of the cluster. runner_health="k8s" checks whether a runner that stopped answering is still alive by reading its pod from the Kubernetes API, where the default pings it. Messages are SchemaBinary frames, as in Effect; serialization="ndjson" writes newline-delimited JSON instead, and serialization_max_buffer_size limits a frame of either. Lower level pieces are there too: socket_runner and http_runner (the protocols and servers), runner_server, runners and runner_health.

Every runner of a cluster must use the same serialization. A runner can't read the other format, so health pings between the two fail: the runner that holds the cluster's health check marks the runners on the other format healthy = false in cluster_runners, and the hash ring gives them no shards. Nothing is lost or run twice, but they do no entity work, and as in Effect this is logged only at Debug ("Runner is unhealthy"), so it looks like a dead runner. To change serialization, or to upgrade from a pyfect whose default was ndjson, either replace every runner at once, or roll the upgrade out with serialization="ndjson" set on the new runners and switch every runner to binary later, all at once.

entity_proxy exposes an entity as an RPC group or an HTTP API group, and entity_proxy_server serves it. to_http_api_group(name, entity) builds the HTTP group at run time, so its client has no attribute per operation: read one by name with .method("Increment") or .method("IncrementDiscard"), typed Any. For a typed client, declare the operations in a group of your own with entity_proxy.http_endpoints(Entity.rpc), one line per RPC. RPCs you leave out stay off HTTP.

proxy.py
from collections.abc import Callable
from typing import Any, Never, TypedDict

import pyfect as fx
from pyfect import schema as S
from pyfect.cluster import entity_proxy, entity_proxy_server, test_runner
from pyfect.cluster.entity import Entity, Request
from pyfect.cluster.sharding import Sharding
from pyfect.http import server
from pyfect.http_api import get_group
from pyfect.http_api import test as api_test
from pyfect.http_api.api import HttpApi
from pyfect.http_api.group import HttpApiGroup
from pyfect.rpc import make as make_rpc


class Amount(TypedDict):
    by: int


class Counter(Entity, type="Counter"):
    increment = make_rpc("Increment", payload=S.Struct(Amount), success=S.Int)


# One line per RPC, and the client is typed. `POST /increment/:entityId` answers with
# the RPC's success, `POST /increment/:entityId/discard` once the message is sent.
class Counters(HttpApiGroup, identifier="counters"):
    increment, increment_discard = entity_proxy.http_endpoints(Counter.increment)


# Built at run time, so its client reads the same operations by name, typed Any.
Derived = entity_proxy.to_http_api_group("derived", Counter)


def increment(request: Request[Amount]) -> fx.Effect[int]:
    return fx.succeed(request.payload["by"] + 1)


typed_api = HttpApi.make("typed").add(Counters)
derived_api = HttpApi.make("derived").add(Derived)
cluster = Counter.to_layer(Counter.handlers().handle(Counter.increment)(increment)).pipe(
    fx.layer.provide_merge(test_runner.layer)
)


@fx.fn
def typed():
    counters = get_group((yield from api_test.groups(typed_api, [Counters])), Counters)
    print("typed:", (yield from counters.increment(params={"entityId": "a"}, payload={"by": 1})))
    sent = yield from counters.increment_discard(params={"entityId": "a"}, payload={"by": 1})
    print("discard:", sent)


@fx.fn
def by_name():
    derived = get_group((yield from api_test.groups(derived_api, [Derived])), Derived)
    call: Callable[..., fx.Effect[int, Any]] = derived.method("Increment")
    print("by name:", (yield from call(params={"entityId": "a"}, payload={"by": 2})))


def serving(handlers: fx.Layer[Any, Never, Sharding]):
    return handlers.pipe(fx.layer.provide_merge(cluster)).pipe(
        fx.layer.provide_merge(server.layer_services)
    )


typed_handlers = entity_proxy_server.layer_http_api(typed_api, Counters, Counter)
derived_handlers = entity_proxy_server.layer_http_api(derived_api, Derived, Counter)
fx.run_main(fx.scoped(typed()).pipe(fx.provide(serving(typed_handlers))))
fx.run_main(fx.scoped(by_name()).pipe(fx.provide(serving(derived_handlers))))

cluster_metrics reports what a runner is doing. entities and entity_mailbox_size_max are per entity type, as in Effect, so .value() on the gauge itself reads 0 while entities run. Read a type's series with .with_attributes({"type": ...}), or all of them with metric.snapshot.

metrics.py
import pyfect as fx
from pyfect import metric
from pyfect import schema as S
from pyfect.cluster import cluster_metrics, test_runner
from pyfect.cluster.entity import Entity
from pyfect.rpc import make as make_rpc


class Counter(Entity, type="Counter"):
    get = make_rpc("Get", success=S.Int)


cluster = Counter.to_layer(Counter.handlers().handle(Counter.get)(lambda _: fx.succeed(1))).pipe(
    fx.layer.provide_merge(test_runner.layer)
)


@fx.gen
def program():
    make_client = yield from Counter.client()
    yield from make_client("a").get()
    yield from make_client("b").get()

    # The series belong to entity types: the gauge with no attributes stays at 0.
    print("no attributes:", (yield from cluster_metrics.entities.value()).value)
    counters = cluster_metrics.entities.with_attributes({"type": "Counter"})
    print("type=Counter:", (yield from counters.value()).value)

    # metric.snapshot lists every series, for types you don't name in advance.
    for series in (yield from metric.snapshot):
        if series.id == cluster_metrics.entities.id and series.attributes:
            print(series.attributes["type"], series.state)


fx.run_main(program.pipe(fx.provide(cluster)))

Singletons and cron jobs

singleton.make(name, effect) runs an effect on exactly one runner of the cluster, the one holding the singleton's shard, and starts it again on another runner when that shard moves. cluster_cron.make runs a job on a cron schedule once per instant across the whole cluster: each run is a persisted message delivered at its instant, which schedules the next one, so a run is not lost while runners come and go. The job reads its instant from CurrentScheduledTime. With calculate_next_run_from_previous=True one entity runs the instants in order and catches up on missed ones; runs older than skip_if_older_than (a day) are skipped.

cron.py
from dataclasses import replace
from datetime import datetime

import pyfect as fx
from pyfect import cron, reactivity
from pyfect.cluster import cluster_cron, single_runner, singleton
from pyfect.cluster.sharding_config import ShardingConfigService
from pyfect.sql import sqlite
from pyfect.sql.client import SqlClient

runs: list[datetime] = []
two_runs = fx.Deferred[None]()


# A singleton runs on exactly one runner, the one holding its shard, and moves
# with the shard when that runner leaves.
leader = singleton.make("leader", fx.sync(lambda: print("leader elected")))


@fx.gen
def tick():
    scheduled = yield from cluster_cron.CurrentScheduledTime()
    runs.append(scheduled)
    if len(runs) == 2:
        yield from two_runs.succeed(None)


# A cron job runs once per instant across the whole cluster: each run is a
# persisted message, delivered at its time, that schedules the next one.
every_second = cluster_cron.make(
    name="tick", cron=cron.parse_unsafe("* * * * * *", "UTC"), execute=tick
)


@fx.gen
def program():
    yield from two_runs.wait()
    yield from fx.sleep("500 millis")  # let the second run schedule the third before shutting down
    print("on whole seconds:", all(run.microsecond == 0 for run in runs))
    print("seconds apart:", (runs[1] - runs[0]).total_seconds())


def settings(config: ShardingConfigService) -> ShardingConfigService:
    # Storage is checked for messages that are due every 10 seconds by default.
    return replace(config, entity_message_poll_interval=fx.duration.millis(100))


# `single_runner.layer` is one runner with its messages in SQL, here in-memory SQLite.
sql = fx.layer.effect(SqlClient, sqlite.make(sqlite.SqliteClientConfig(filename=":memory:")))
runner = single_runner.layer(sharding_config=settings, runner_storage_kind="memory").pipe(
    fx.layer.provide_merge(sql.pipe(fx.layer.provide(reactivity.layer)))
)
jobs = fx.layer.merge(leader, every_second).pipe(fx.layer.provide_merge(runner))
fx.run_main(program.pipe(fx.provide(jobs)))

Messages due later, like the next cron run or a durable clock's wake-up, are read from storage when they fall due on the runner that stored them, and by every runner every entity_message_poll_interval (10 seconds, as in Effect), which also finds messages another runner stored. Under the test clock each poll, shard-lock refresh and assignment check is a wake-up, so adjusting by days costs real time; adjust by what the test needs.

Workflows

Workflows run on one process with workflow_engine.layer_memory. cluster_workflow_engine.layer runs the same workflows on the cluster: each execution is an entity, its activity results, deferred completions and durable clocks are persisted messages, and an execution suspended on a runner that stops resumes on the runner that takes its shard, replaying its completed activities from storage instead of running them again.

workflows.py
from typing import TypedDict

import pyfect as fx
from pyfect import schema as S
from pyfect.cluster import cluster_workflow_engine, test_runner
from pyfect.workflow import activity, durable_deferred, workflow


class Signup(TypedDict):
    email: str


SignUp = workflow.make(
    "SignUp",
    payload=S.Struct(Signup),
    success=S.String,
    idempotency_key=lambda signup: signup["email"],
)
Confirmed = durable_deferred.make("confirmed", success=S.Boolean)


@fx.fn
def sign_up(signup: Signup, _execution_id: str):
    yield from activity.make(
        name="send-link", execute=fx.sync(lambda: print("link sent to", signup["email"]))
    )
    confirmed = yield from Confirmed.wait()
    return "active" if confirmed else "closed"


# The same workflow, now run by the cluster: each execution is an entity, its
# activities and deferred results are persisted messages, and it moves with its shard.
cluster = (
    SignUp.to_layer(sign_up)
    .pipe(fx.layer.provide_merge(cluster_workflow_engine.layer))
    .pipe(fx.layer.provide_merge(test_runner.layer))
)


@fx.gen
def program():
    signup: Signup = {"email": "ada@example.com"}
    execution_id = yield from SignUp.execute(signup, discard=True)
    token = Confirmed.token_from_execution_id(workflow=SignUp, execution_id=execution_id)
    yield from Confirmed.succeed(token=token, value=True)
    # Executing again with the same idempotency key joins the first execution.
    print("status:", (yield from SignUp.execute(signup)))


fx.run_main(program.pipe(fx.provide(cluster)))

Messages

A message to an entity travels as an envelope: a Request carrying an RPC's payload, an AckChunk acknowledging a streamed reply, or an Interrupt. Replies are WithExit (the final Exit) or Chunk (part of a stream). Request ids are snowflakes: 64-bit ids made of a millisecond timestamp, the machine id and a sequence, ordered by time and unique per machine. Envelopes and replies are stored as Effect's JSON, so the rows pyfect writes match Effect's.

Storage

MessageStorage persists requests and their replies for at-least-once delivery: it dedupes requests by primary key, hands unprocessed messages to the runner that holds their shard, and notifies waiting senders of a final reply after the transaction that stored it commits (stream chunks at once). message_storage.layer_memory keeps them in memory; sql_message_storage.layer uses Effect's tables on a SQL client (SQLite or PostgreSQL).

RunnerStorage registers runners and holds shard locks, so two runners never own one shard. runner_storage.layer_memory is for tests; sql_runner_storage.layer takes PostgreSQL advisory locks on one reserved connection, or leases in a table on SQLite (and with shard_lock_disable_advisory), renewed every shard_lock_refresh_interval.

Kubernetes

Run each runner as a pod (a Deployment or a StatefulSet), every one pointing at the same database, with cluster_socket.layer(...) or cluster_http.layer(...). Their settings come from the environment in CONSTANT_CASE (sharding_config.layer_from_env), so a manifest configures a runner with environment variables.

Addresses. runner_address is the address a runner advertises: the others dial it, and the Kubernetes health check looks the runner's pod up by it, so set its host to the pod's IP with the downward API (HOST from status.podIP, and PORT, 34431 by default). A runner binds to runner_address unless runner_listen_address is set (LISTEN_HOST, and LISTEN_PORT, also 34431 by default). Set it when the advertised address is not one of the pod's own interfaces, such as a Service name or a NAT address; LISTEN_HOST=0.0.0.0 listens on every interface.

Health. runner_health="k8s" decides whether a runner that stopped answering is alive by reading its pod instead of pinging it: the runner is alive while its pod is ready or still initializing. It lists the running pods (runner_health_k8s_namespace and runner_health_k8s_label_selector narrow the list; without a namespace it lists the whole cluster) and caches the list for ten seconds. When the API can't be read, it logs "Failed to fetch pods from Kubernetes API" and counts every runner alive, so a dead runner's shards move only once its shard locks expire, as in Effect.

RBAC. The pod's service account needs list on pods for the health check, and create, get and delete on pods for entity_resource.make_k8s_pod, in the namespaces they use. A health check without a namespace needs a ClusterRole.

CA trust. The client calls https://kubernetes.default.svc/api with the service account's token, read again at most once a minute so a rotated token is picked up. The layers above build their Kubernetes client with layer_k8s_http_client (in cluster_socket and cluster_http), which needs the httpx extra and trusts the CA mounted at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, as Effect's layerK8sHttpClient does. Provide that layer for your own calls too. k8s_http_client.layer alone runs on the HttpClient you give it, which must trust that CA.

Shutdown. Kubernetes sends SIGTERM and kills the pod after terminationGracePeriodSeconds (30 by default). fx.run_main interrupts the program on SIGTERM, and the runner drains for up to entity_termination_timeout (15 seconds), so keep the grace period above it.

runner.yaml
apiVersion: v1
kind: ServiceAccount
metadata: { name: runner, namespace: jobs }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: runner, namespace: jobs }
rules:
  - apiGroups: [""]
    resources: [pods]
    verbs: [list, get, create, delete]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: runner, namespace: jobs }
subjects: [{ kind: ServiceAccount, name: runner, namespace: jobs }]
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: runner }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: runner, namespace: jobs }
spec:
  replicas: 3
  selector: { matchLabels: { app: runner } }
  template:
    metadata: { labels: { app: runner } }
    spec:
      serviceAccountName: runner
      terminationGracePeriodSeconds: 30
      containers:
        - name: runner
          image: registry.example.com/runner:1
          ports: [{ containerPort: 34431 }]
          env:
            - name: HOST
              valueFrom: { fieldRef: { fieldPath: status.podIP } }
            - name: PORT
              value: "34431"
            - name: DATABASE_URL
              valueFrom: { secretKeyRef: { name: runner-db, key: url } }

Entity resources

entity_resource.make(acquire) holds a resource for an entity: a process, a client, a pod. It is acquired on the first get, kept while the entity's handlers restart, and keeps the entity alive while it is held. close releases it, as does idle_time_to_live without a get (never by default); acquire_eagerly=True acquires it when the entity starts. Finalizers acquire adds to entity_resource.CloseScope run only on that release, not when the runner stops, so the resource outlives a runner that hands the entity over.

entity_resource.make_k8s_pod(pod) is such a resource for a Kubernetes pod. It takes a whole k8s_types.Pod manifest, as Effect's makeK8sPod does: metadata.name is required and the namespace defaults to default. It creates the pod (or adopts it when one with that name exists), waits until it is ready, and returns its PodStatus. On release it deletes the pod and waits until it is gone. It needs a K8sHttpClient:

k8s.py
import os

import pyfect as fx
from pyfect import schema as S
from pyfect.cluster import cluster_socket, entity_resource, k8s_types
from pyfect.cluster.entity import CurrentAddress, Entity
from pyfect.rpc import make as make_rpc
from pyfect.sql import pg


class Sandbox(Entity, type="Sandbox"):
    phase = make_rpc("Phase", success=S.String)


def sandbox_pod(entity_id: str) -> k8s_types.Pod:
    # A whole Pod manifest: metadata (name and namespace) and its spec.
    return {
        "metadata": {"name": f"sandbox-{entity_id}", "namespace": "jobs"},
        "spec": {
            "containers": [{"name": "sandbox", "image": "registry.example.com/sandbox:1"}],
            "restartPolicy": "Never",
        },
    }


@fx.gen
def handlers():
    address = yield from CurrentAddress()
    # One pod per entity, created on first use and deleted after 5 idle minutes.
    pod = yield from entity_resource.make_k8s_pod(
        sandbox_pod(str(address.entity_id)), idle_time_to_live="5 minutes"
    )

    def phase(_: object) -> fx.Effect[str]:
        return fx.scoped(pod.get.map(lambda status: status.phase))

    return Sandbox.handlers().handle(Sandbox.phase)(phase)


runner = (
    Sandbox.to_layer_effect(handlers)
    # The Kubernetes API client, trusting the service account's CA.
    .pipe(fx.layer.provide(cluster_socket.layer_k8s_http_client))
    .pipe(
        fx.layer.provide_merge(
            cluster_socket.layer(
                runner_health="k8s",
                runner_health_k8s_namespace="jobs",
                runner_health_k8s_label_selector="app=runner",
            )
        )
    )
    .pipe(fx.layer.provide(pg.layer_config({"url": fx.config.redacted("DATABASE_URL")})))
)

# Kubernetes sets KUBERNETES_SERVICE_HOST in every pod.
if os.environ.get("KUBERNETES_SERVICE_HOST"):
    fx.run_main(runner.launch())

Reference: pyfect.cluster, entity, sharding, cluster_cron, cluster_workflow_engine, pyfect.hash_ring, sharding_config, message_storage, entity_resource, k8s_http_client.