Long-running processes that survive waiting: steps run once, waits suspend, and failures undo what already happened.
A workflow is an effect the engine can stop and replay. Each step's result is stored, so
when a workflow waits for a person, a timer or another system, it suspends; when the answer
comes, the engine runs it again from the start, reading back what already happened
instead of doing it twice. pyfect.workflow follows Effect's effect/workflow.
Define and run#
workflow.make(tag, payload=..., success=..., error=..., idempotency_key=...) declares a
workflow: its payload, result and error schemas, and the key that names an execution, so
executing the same payload twice joins one execution. to_layer(handler) registers the
handler, a function of the payload and the execution id, on a WorkflowEngine.
workflow_engine.layer_memory keeps executions in memory, for tests and one-process
programs; cluster_workflow_engine.layer persists them on a
cluster, where they survive restarts.
execute(payload) runs the workflow and waits for its result, retrying while it is
suspended. execute(payload, discard=True) starts it and gives its execution id, which
poll, interrupt and resume take. poll gives a Complete with its Exit once the
execution has finished, a Suspended while it waits for a deferred or a scheduled wake-up,
and None both while a run is in progress and for an id it has no execution for, as in
Effect. Inside a workflow, executing another workflow awaits it as a child; a child that
is interrupted fails the parent with a defect rather than an interrupt, so the parent can
recover with catch_defect or .exit().
from typing import TypedDict
import pyfect as fx
from pyfect import schema as S
from pyfect.workflow import (
activity,
durable_clock,
durable_deferred,
workflow,
workflow_engine,
)
class Order(TypedDict):
id: str
amount: float
# A workflow names its payload, result and error, and how to tell executions apart.
PlaceOrder = workflow.make(
"PlaceOrder",
payload=S.Struct(Order),
success=S.String,
error=S.String,
idempotency_key=lambda order: order["id"],
)
# A named wait point, completed from outside the workflow by its token.
Approval = durable_deferred.make("approval", success=S.Boolean)
def charge(order: Order) -> fx.Effect[str]:
def run() -> str:
print(f"charge {order['id']}")
return f"ch_{order['id']}"
return fx.sync(run)
def refund(charge_id: str) -> fx.Effect[None]:
return fx.sync(lambda: print(f"refund {charge_id}"))
@fx.fn
def place_order(order: Order, _execution_id: str):
# An activity's result is stored: a replay reads it back instead of charging again.
charge_id = yield from activity.make(
name="charge", success=S.String, execute=charge(order)
).pipe(PlaceOrder.with_compensation(lambda charge_id, _cause: refund(charge_id)))
yield from durable_clock.sleep(name="cool-off", duration="10 millis")
approved = yield from Approval.wait() # suspends until someone answers
if not approved:
return (yield from fx.fail("rejected")) # runs the compensation: a refund
return charge_id
Orders = PlaceOrder.to_layer(place_order).pipe(fx.layer.provide_merge(workflow_engine.layer_memory))
@fx.fn
def decide(order: Order, approved: bool):
running = yield from PlaceOrder.execute(order).exit().fork()
# The answer is stored durably, so it may arrive before the workflow waits for it.
token = yield from Approval.token_from_payload(workflow=PlaceOrder, payload=order)
yield from Approval.succeed(token=token, value=approved)
print(order["id"], (yield from running.join()))
@fx.fn
def program():
yield from decide({"id": "a1", "amount": 20}, approved=True)
yield from decide({"id": "b2", "amount": 35}, approved=False)
fx.run_main(program().pipe(fx.provide(Orders)))Activities#
activity.make(name=..., success=..., execute=...) is a step whose Exit the engine
stores, under a name unique in its workflow. pyright checks execute against the
schemas. A replay gets the stored result, so code with side effects (a charge, an email)
belongs in an activity, and plain code between activities runs again on each replay.
An activity interrupted by a replay runs again; activity.retry(times=n) retries a failed
one with a fresh attempt, and activity.idempotency_key(name) gives a key, stable across
replays, to send to the system it calls.
Durable clocks and deferreds#
durable_clock.sleep(name=..., duration=...) waits inside a workflow. A sleep up to
in_memory_threshold (60 seconds) is an activity that sleeps, so the execution keeps
running (poll gives None) and races including it are live; a longer one schedules a
wake-up with the engine and suspends (poll gives Suspended), so nothing waits in memory.
Both sleep on the Clock, so the test clock moves them; let the run reach
its sleep (a short adjust first) before moving past it.
durable_deferred.make(name, success=..., error=...) is a named wait point. wait()
suspends the workflow until something outside completes it with succeed, fail or
done, by a token: token_from_payload or token_from_execution_id from outside,
token() from inside, to send along with a request for approval. The answer is stored,
so it may arrive before the workflow reaches wait(). durable_deferred.race_all and
activity.race_all race steps so that a replay gets the same winner.
Compensation#
Workflow.with_compensation(compensation), used with .pipe on a step, runs
compensation(value, cause) if the whole workflow later fails, with the value the step
produced. Compensations run once, in reverse order, after the failure; a replay does not
add them twice. Only top-level steps are compensated, not effects inside an activity. In
orders.py, the rejected order is refunded.
workflow.add_finalizer(f) runs f(exit) when the execution completes, not each time a
run suspends. Each run that registered it runs it then, as in Effect: a workflow that
suspended once and then finished calls f twice, the finishing run with the real exit and
the earlier run with a successful None exit. Make finalizers safe to repeat, or act only
on the exit you care about. durable_queue hands work to PersistedQueue workers
and waits for their results. A worker killed while it holds a job stalls that job, and the
workflow waiting on it, until the job's lock expires: 120 seconds by default with
persisted_queue.layer_store_sql and 90 with layer_store_redis, as in Effect. Set it on the
store, layer_store_sql(lock_expiration=...), and keep it above lock_refresh_interval
(30 seconds). workflow_proxy and workflow_proxy_server expose
workflows as RPC or HttpApi groups that execute, discard
and resume them. to_rpc_group and to_http_api_group build the groups at run time, so
their clients call operations by name; declare workflow_proxy.rpcs(W) or
workflow_proxy.http_endpoints(W) in a group class of your own
(submit, submit_discard, submit_resume = workflow_proxy.http_endpoints(SubmitExpense)) to
type them, and serve that group with workflow_proxy_server.layer_rpc_handlers or
handle_http next to its other endpoints. As in Effect, there is no operation that completes
a deferred: expose succeed on a deferred's token through an endpoint of your own.
import pyfect leaves them out, so import them as
from pyfect.workflow import workflow_proxy.
Reference: pyfect.workflow,
workflow,
activity,
durable_deferred.