GitHub

Workflows

Long-running processes that survive waiting: steps run once, waits suspend, and failures undo what already happened.

A workflow is an effect the engine can stop and replay. Each step's result is stored, so when a workflow waits for a person, a timer or another system, it suspends; when the answer comes, the engine runs it again from the start, reading back what already happened instead of doing it twice. pyfect.workflow follows Effect's effect/workflow.

Define and run

workflow.make(tag, payload=..., success=..., error=..., idempotency_key=...) declares a workflow: its payload, result and error schemas, and the key that names an execution, so executing the same payload twice joins one execution. to_layer(handler) registers the handler, a function of the payload and the execution id, on a WorkflowEngine. workflow_engine.layer_memory keeps executions in memory, for tests and one-process programs; cluster_workflow_engine.layer persists them on a cluster, where they survive restarts.

execute(payload) runs the workflow and waits for its result, retrying while it is suspended. execute(payload, discard=True) starts it and gives its execution id, which poll, interrupt and resume take. poll gives a Complete with its Exit once the execution has finished, a Suspended while it waits for a deferred or a scheduled wake-up, and None both while a run is in progress and for an id it has no execution for, as in Effect. Inside a workflow, executing another workflow awaits it as a child; a child that is interrupted fails the parent with a defect rather than an interrupt, so the parent can recover with catch_defect or .exit().

yesnonoyesanswer or clock wakes it: replayPlaceOrder.execute(order)run the handleractivity result stored?reuse itrun it, store its Exitdeferred answered?SuspendedComplete, or compensate onfailure
yesnonoyesanswer or clock wakes it: replayPlaceOrder.execute(order)run the handleractivity result stored?reuse itrun it, store its Exitdeferred answered?SuspendedComplete, or compensate onfailure
orders.py
from typing import TypedDict

import pyfect as fx
from pyfect import schema as S
from pyfect.workflow import (
    activity,
    durable_clock,
    durable_deferred,
    workflow,
    workflow_engine,
)


class Order(TypedDict):
    id: str
    amount: float


# A workflow names its payload, result and error, and how to tell executions apart.
PlaceOrder = workflow.make(
    "PlaceOrder",
    payload=S.Struct(Order),
    success=S.String,
    error=S.String,
    idempotency_key=lambda order: order["id"],
)

# A named wait point, completed from outside the workflow by its token.
Approval = durable_deferred.make("approval", success=S.Boolean)


def charge(order: Order) -> fx.Effect[str]:
    def run() -> str:
        print(f"charge {order['id']}")
        return f"ch_{order['id']}"

    return fx.sync(run)


def refund(charge_id: str) -> fx.Effect[None]:
    return fx.sync(lambda: print(f"refund {charge_id}"))


@fx.fn
def place_order(order: Order, _execution_id: str):
    # An activity's result is stored: a replay reads it back instead of charging again.
    charge_id = yield from activity.make(
        name="charge", success=S.String, execute=charge(order)
    ).pipe(PlaceOrder.with_compensation(lambda charge_id, _cause: refund(charge_id)))
    yield from durable_clock.sleep(name="cool-off", duration="10 millis")
    approved = yield from Approval.wait()  # suspends until someone answers
    if not approved:
        return (yield from fx.fail("rejected"))  # runs the compensation: a refund
    return charge_id


Orders = PlaceOrder.to_layer(place_order).pipe(fx.layer.provide_merge(workflow_engine.layer_memory))


@fx.fn
def decide(order: Order, approved: bool):
    running = yield from PlaceOrder.execute(order).exit().fork()
    # The answer is stored durably, so it may arrive before the workflow waits for it.
    token = yield from Approval.token_from_payload(workflow=PlaceOrder, payload=order)
    yield from Approval.succeed(token=token, value=approved)
    print(order["id"], (yield from running.join()))


@fx.fn
def program():
    yield from decide({"id": "a1", "amount": 20}, approved=True)
    yield from decide({"id": "b2", "amount": 35}, approved=False)


fx.run_main(program().pipe(fx.provide(Orders)))

Activities

activity.make(name=..., success=..., execute=...) is a step whose Exit the engine stores, under a name unique in its workflow. pyright checks execute against the schemas. A replay gets the stored result, so code with side effects (a charge, an email) belongs in an activity, and plain code between activities runs again on each replay. An activity interrupted by a replay runs again; activity.retry(times=n) retries a failed one with a fresh attempt, and activity.idempotency_key(name) gives a key, stable across replays, to send to the system it calls.

Durable clocks and deferreds

durable_clock.sleep(name=..., duration=...) waits inside a workflow. A sleep up to in_memory_threshold (60 seconds) is an activity that sleeps, so the execution keeps running (poll gives None) and races including it are live; a longer one schedules a wake-up with the engine and suspends (poll gives Suspended), so nothing waits in memory. Both sleep on the Clock, so the test clock moves them; let the run reach its sleep (a short adjust first) before moving past it.

durable_deferred.make(name, success=..., error=...) is a named wait point. wait() suspends the workflow until something outside completes it with succeed, fail or done, by a token: token_from_payload or token_from_execution_id from outside, token() from inside, to send along with a request for approval. The answer is stored, so it may arrive before the workflow reaches wait(). durable_deferred.race_all and activity.race_all race steps so that a replay gets the same winner.

Compensation

Workflow.with_compensation(compensation), used with .pipe on a step, runs compensation(value, cause) if the whole workflow later fails, with the value the step produced. Compensations run once, in reverse order, after the failure; a replay does not add them twice. Only top-level steps are compensated, not effects inside an activity. In orders.py, the rejected order is refunded.

workflow.add_finalizer(f) runs f(exit) when the execution completes, not each time a run suspends. Each run that registered it runs it then, as in Effect: a workflow that suspended once and then finished calls f twice, the finishing run with the real exit and the earlier run with a successful None exit. Make finalizers safe to repeat, or act only on the exit you care about. durable_queue hands work to PersistedQueue workers and waits for their results. A worker killed while it holds a job stalls that job, and the workflow waiting on it, until the job's lock expires: 120 seconds by default with persisted_queue.layer_store_sql and 90 with layer_store_redis, as in Effect. Set it on the store, layer_store_sql(lock_expiration=...), and keep it above lock_refresh_interval (30 seconds). workflow_proxy and workflow_proxy_server expose workflows as RPC or HttpApi groups that execute, discard and resume them. to_rpc_group and to_http_api_group build the groups at run time, so their clients call operations by name; declare workflow_proxy.rpcs(W) or workflow_proxy.http_endpoints(W) in a group class of your own (submit, submit_discard, submit_resume = workflow_proxy.http_endpoints(SubmitExpense)) to type them, and serve that group with workflow_proxy_server.layer_rpc_handlers or handle_http next to its other endpoints. As in Effect, there is no operation that completes a deferred: expose succeed on a deferred's token through an endpoint of your own. import pyfect leaves them out, so import them as from pyfect.workflow import workflow_proxy.

Reference: pyfect.workflow, workflow, activity, durable_deferred.