Event sourcing that works offline: write events to a local journal, replay them through handlers, and sync with a server, encrypted end to end if you like.
An event log stores what happened, not the current state. Each write appends an event to a
journal and runs its handler, which updates the read model the program queries. A remote
syncs the journal with a server, so several clients (a phone, a laptop, a browser tab)
apply the same events and converge, even after writing while offline. pyfect.eventlog
follows Effect's effect/eventlog; import it as a submodule, since import pyfect does
not load it.
Events and handlers#
event.make(tag, primary_key=..., payload=..., success=..., error=...) declares an
event: its payload schema and a function giving the key its conflicts are found by.
event_group.empty.add(A).add(B) collects events, and event_log.group(group, lambda h: h.handle(A, fa).handle(B, fb)) registers a handler for each, checked by pyright against
the payload. A handler gets a HandlerInput: the decoded payload, the journal entry
and its conflicts, the stored entries of the same event and key that are newer than it,
which a late entry from another client has to respect.
event_log.layer(event_log.schema(group), handlers) builds the EventLog, which needs an
EventJournal and an Identity, the owner of the log. log.write(event, payload)
appends the event, runs its handler and succeeds with the handler's result;
write_many([...]) writes several under one lock; log.entries reads the journal.
from typing import TypedDict
import pyfect as fx
from pyfect import schema as S
from pyfect.eventlog import event as Ev
from pyfect.eventlog import event_group as G
from pyfect.eventlog import event_journal as J
from pyfect.eventlog import event_log as L
from pyfect.eventlog.event_log_encryption import layer_subtle
class Item(TypedDict):
list: str
name: str
class Cleared(TypedDict):
list: str
# An event: its tag, the key its conflicts are found by, and its payload schema.
ItemAdded = Ev.make("ItemAdded", primary_key=lambda p: p["list"], payload=S.Struct(Item))
ListCleared = Ev.make("ListCleared", primary_key=lambda p: p["list"], payload=S.Struct(Cleared))
Groceries = G.empty.add(ItemAdded).add(ListCleared)
# The read model the handlers keep up to date (a table, in a real program).
lists: dict[str, list[str]] = {}
def add(item: L.HandlerInput[Item]) -> fx.Effect[None]:
return fx.sync(lambda: lists.setdefault(item.payload["list"], []).append(item.payload["name"]))
def clear(cleared: L.HandlerInput[Cleared]) -> fx.Effect[None]:
return fx.sync(lambda: lists.pop(cleared.payload["list"], None)).as_void()
Handlers = L.group(Groceries, lambda h: h.handle(ItemAdded, add).handle(ListCleared, clear))
Log = (
# The log gives the handlers its Registry, so only the journal and identity remain.
L.layer(L.schema(Groceries), Handlers)
.pipe(fx.layer.provide(J.layer_memory))
.pipe(fx.layer.provide(fx.layer.effect(L.Identity, L.make_identity)))
)
@fx.fn
def program():
log = yield from L.EventLog()
yield from log.write(ItemAdded, {"list": "home", "name": "milk"})
yield from log.write(ItemAdded, {"list": "home", "name": "eggs"})
yield from log.write(ItemAdded, {"list": "office", "name": "coffee"})
yield from log.write(ListCleared, {"list": "office"})
print(lists)
# The journal keeps every event, in order, to replay or send to a server.
for entry in (yield from log.entries):
print(entry.event, entry.primary_key)
fx.run_main(program().pipe(fx.provide(Log)).pipe(fx.provide(layer_subtle)))The identity in this example comes from make_identity, which uses
event_log_encryption.layer_subtle: install the pyfect-dev[eventlog] extra
(cryptography and PyNaCl) for it and for anything encrypted. The encryption runs on the
pyfect.crypto services: layer_subtle on their Web Crypto layers, and
event_log_encryption.layer on whatever Aead, Digest and SecureRandom you provide.
Rows#
grocery = event_row.make("grocery", key="id", schema=S.Struct(Grocery), key_type=str, patch_fields=GroceryPatch) is an event that sets, patches or removes one row:
grocery.set(row), grocery.patch(key, fields) and grocery.remove(key) give writes for
write_many. key_type and patch_fields (a total=False TypedDict of the non-key
fields) let pyright check the key and each patch, as Effect derives them from the schema;
make checks patch_fields names exactly the non-key fields, and patch raises
EventRowPatchError for a key field or a field the row does not have.
event_row.handler(row, set=..., patch=..., remove=...) turns three callbacks into its
handler, resolving a late entry against the newer ones already stored, so clients that
apply the same entries in any order end with the same rows. event_row.compaction(group, row) drops entries that newer ones make redundant.
Pass key_type: without it the key is object, so a callback that types its key
(def remove(key: str)) doesn't fit the handler and pyright rejects it.
from typing import NotRequired, TypedDict
import pyfect as fx
from pyfect import schema as S
from pyfect.eventlog import event_group as G
from pyfect.eventlog import event_journal as J
from pyfect.eventlog import event_log as L
from pyfect.eventlog import event_row
from pyfect.eventlog.event_log_encryption import layer_subtle
class Grocery(TypedDict):
id: str
name: str
done: bool
class GroceryPatch(TypedDict): # the non-key fields, each optional
name: NotRequired[str]
done: NotRequired[bool]
# `key_type=str` types the key of `patch`, `remove` and the handler's callbacks.
# Without it the key is `object`, and a callback taking `key: str` doesn't fit.
GroceryRow = event_row.make(
"grocery", key="id", schema=S.Struct(Grocery), key_type=str, patch_fields=GroceryPatch
)
Groceries = G.empty.add(GroceryRow)
table: dict[str, dict[str, object]] = {} # the read model (a SQL table, in a real program)
def set_row(row: Grocery, key: str) -> fx.Effect[None]:
return fx.sync(lambda: table.__setitem__(key, dict(row)))
def patch_row(key: str, fields: GroceryPatch) -> fx.Effect[None]:
return fx.sync(lambda: table[key].update(fields))
def remove_row(key: str) -> fx.Effect[None]:
return fx.sync(lambda: table.pop(key, None)).as_void()
handle = event_row.handler(GroceryRow, set=set_row, patch=patch_row, remove=remove_row)
Handlers = L.group(Groceries, lambda h: h.handle(GroceryRow, handle))
Log = (
L.layer(L.schema(Groceries), Handlers)
.pipe(fx.layer.provide(J.layer_memory))
.pipe(fx.layer.provide(fx.layer.effect(L.Identity, L.make_identity)))
)
@fx.fn
def program():
log = yield from L.EventLog()
yield from log.write_many(
[
GroceryRow.set({"id": "g1", "name": "milk", "done": False}),
GroceryRow.set({"id": "g2", "name": "eggs", "done": False}),
GroceryRow.patch("g1", {"done": True}),
GroceryRow.remove("g2"),
]
)
print(table)
fx.run_main(program().pipe(fx.provide(Log)).pipe(fx.provide(layer_subtle)))Journals#
event_journal.layer_memory keeps entries in memory. sql_event_journal.layer() keeps
them in a SQL database (SQLite or PostgreSQL), with local writes serialized
across processes. Entry ids are UUID v7, so the journal is ordered by time.
Sync with a server#
event_log_remote.layer_encrypted and layer_unencrypted connect the log to a server
over the RPC protocol in event_log_message. A remote sends what the journal
has not sent yet, in chunks, and replays the server's changes through the handlers,
running group_compaction / group_batch_compaction compactors on the way in.
group_reactivity(group, keys) invalidates pyfect.reactivity keys as entries are
handled, so reactive queries rerun.
The encrypted server, event_log_server_encrypted.layer, stores AES-GCM batches it cannot
read: only clients holding the identity's key can. The unencrypted one,
event_log_server_unencrypted.layer(...), runs the handlers itself, maps identities to
stores, checks an EventLogServerAuthorization and can compact the backlog. Both have
memory storage for tests and SQL storage (sql_event_log_server_encrypted,
sql_event_log_server_unencrypted). Clients authenticate by signing a challenge with an
Ed25519 key derived from their identity. A challenge lives on its connection for 30 seconds
and is used once. The server binds the first verified signing key of an identity and rejects
any other; authorize_identity also receives the presented signing_public_key, so a
deployment that records each identity's key at provisioning time
(event_log_session_auth.derive_signing_public_key) can reject every other key, even on
first use.
Here a test server and two devices of one user, a laptop and a phone, sync a note. The
server is event_log_server_encrypted.layer behind rpc_server.layer_protocol_websocket(path),
served by a router; each device's remote is event_log_remote.layer_encrypted over
rpc_client.layer_protocol_socket() on a ws.layer_web_socket(url). Both devices write as
the same Identity: encode_identity_string turns it into text to carry to the other
device, and decode_identity_string reads it back. It needs the eventlog extra for the
encryption and the server extra for uvicorn and WebSockets:
uv add "pyfect-dev[eventlog,server]".
from typing import TypedDict
import pyfect as fx
from pyfect import layer
from pyfect import schema as S
from pyfect.eventlog import event as Ev
from pyfect.eventlog import event_group as G
from pyfect.eventlog import event_journal as J
from pyfect.eventlog import event_log as L
from pyfect.eventlog import event_log_remote as Remote
from pyfect.eventlog import event_log_server_encrypted as Server
from pyfect.eventlog.event_log_encryption import layer_subtle
from pyfect.http import router, uvicorn_server
from pyfect.http import server as http_server
from pyfect.rpc import client as rpc_client
from pyfect.rpc import serialization
from pyfect.rpc import server as rpc_server
from pyfect.socket import ws
class Note(TypedDict):
id: str
text: str
NoteAdded = Ev.make("NoteAdded", primary_key=lambda p: p["id"], payload=S.Struct(Note))
Notes = G.empty.add(NoteAdded)
# The server stores encrypted batches it cannot read, served as RPC over a WebSocket.
protocol = rpc_server.layer_protocol_websocket("/sync").pipe(
layer.provide(serialization.layer_ndjson)
)
sync_server = Server.layer.pipe(layer.provide(layer.merge(protocol, Server.layer_storage_memory)))
served = router.serve(sync_server, disable_logger=True, disable_listen_log=True)
def device(name: str, identity: L.IdentityService, url: str, seen: fx.Queue[str]):
"""One client: its own journal and read model, syncing as ``identity``."""
def added(note: L.HandlerInput[Note]) -> fx.Effect[None]:
return seen.offer(f"{name} has {note.payload['text']!r}").as_void()
handlers = L.group(Notes, lambda h: h.handle(NoteAdded, added))
log = L.layer(L.schema(Notes), handlers).pipe(
layer.provide(layer.merge(J.layer_memory, layer.succeed(L.Identity, identity)))
)
socket = rpc_client.layer_protocol_socket().pipe(
layer.provide(layer.merge(ws.layer_web_socket(url), serialization.layer_ndjson))
)
remote = Remote.layer_encrypted.pipe(layer.provide(socket))
return remote.pipe(layer.provide_merge(log))
@fx.fn
def program():
url = yield from http_server.address_formatted_with(
lambda a: fx.succeed(a.replace("http://", "ws://") + "/sync")
)
# Both devices are the same user: the identity travels as text (a QR code, say).
identity = yield from L.make_identity
shared = L.decode_identity_string(L.encode_identity_string(identity))
seen = yield from fx.Queue[str].unbounded()
# The phone stays connected for the rest of the program.
yield from device("phone", shared, url, seen).build()
@fx.fn
def on_laptop():
log = yield from L.EventLog()
yield from log.write(NoteAdded, {"id": "n1", "text": "buy milk"})
# The laptop handles its own write; the phone gets it through the server.
for _ in range(2):
print((yield from seen.take()))
yield from fx.scoped(on_laptop()).pipe(fx.provide(device("laptop", identity, url, seen)))
fx.run_main(
fx.scoped(program())
.pipe(fx.provide(served))
.pipe(fx.provide(uvicorn_server.layer_test()))
.pipe(fx.provide(layer_subtle))
)To wait for a sync, run log.synced: it succeeds once the server has acknowledged
everything the log wrote and the log has applied the server's changes up to the latest
entry the server held when asked. Effect has no such call; pyfect adds it (with the
EventLog.LatestSequence RPC it asks the server through), so a test or a script can sync
and then check, without sleeping.
After log.destroy, log.synced fails with EventLogDestroyedError: the journal is empty,
and the remotes do not send their entries again (Effect does not restart them either), so
the log can never catch up. synced knows of a destroy made through the log; a journal
destroyed directly through EventJournal is not noticed.
Payloads, entries and the write and change frames are SchemaBinary frames with Effect's bytes.
Remote compaction#
An encrypted server that sets HelloResponse.compaction can keep a compacted snapshot of
a store's history, so a new device replays the snapshot and a recent tail instead of every
write. Only clients can decrypt the entries, so clients build the snapshots. A compactor
opts in with remote=True on group_compaction or group_batch_compaction
(event_row.compaction always does). That promises what the library can't check: the
entries it keeps depend only on the set of input entries, never on services, the clock or
their order, and compaction composes, so drop(drop(a) + b) keeps what drop(a + b) keeps
even when b holds older entries. A remote run only drops entries: a call that writes a
new entry keeps its whole input, and entries that don't decode, including ones with fields
the schema doesn't know, are kept. Before uploading, the run passes the kept entries through
the compactors again and uploads nothing when that drops more or a compactor dies.
The log polls the compaction status when it connects and about hourly. When the server
wants compaction and the journal has caught up, it starts a background run after a random
delay of one to five minutes: it reads the input (from the journal where it can, fetching
the rest), runs the opted-in compactors and uploads what they keep in segments of about
512 KB. The server lets one run per store hold a stage, so devices that start together
download the input once. log.compact() (or log.compact(remote_id=...)) runs one now and
returns a CompactOutcome per remote. A device with a large store can opt out with
event_log_remote.layer_encrypted_with(compaction=False) and leave compaction to a device
with more memory, since a run holds every kept entry in memory.
On the server, both storages compact: event_log_server_encrypted.layer_storage_memory_with(...)
and sql_event_log_server_encrypted.layer_storage(compaction=...). A store wants compaction
once at least threshold_floor (1000) entries, and at least as many as the active snapshot
keeps, were stored more than lag (24 hours) ago. After runs that kept everything, it waits
for that many new entries, doubling the wait after each one. A stage's claim lasts stage_ttl (10 minutes) from its last
uploaded part, and a replaced snapshot stays restorable for grace (30 days), which must be
longer than both. storage.compaction.rollback(public_key, store_id, activation_id) withdraws
the active snapshot and appends its input again at the head, so readers that already passed
the snapshot get every entry. A sweep every sweep_interval (1 hour) closes expired stages
and prunes what nothing restorable needs; the SQL sweep resumes a prune that died midway.
Reference: pyfect.eventlog,
event_log,
event_row,
event_log_remote.