Programs reach the operating system through services: FileSystem and Path for files,
ChildProcessSpawner for processes, HttpClient for HTTP. Your code asks for the service and
the layer you provide decides what answers, so a test can swap the network or the disk for a
stand-in. Every file and process operation fails with a typed PlatformError whose reason
says what went wrong (NotFound, PermissionDenied, AlreadyExists, ...).
Files#
fs.read_file_string and fs.write_file_string read and write a whole file; fs.stream
reads one as a stream of byte chunks, so memory stays bounded however large it is, and
fs.sink writes one from a stream. make_temp_directory_scoped gives a directory that is
removed, with everything in it, when its scope closes. fs.glob finds the paths under a
root that match a pattern (** crosses directories, {a,b} and [a-z] choose, dot=True
includes names starting with .). exclude= takes patterns of paths to leave out; a
directory that matches one is not read at all, so excluding .venv or node_modules saves
walking them. pyfect.glob matches the same patterns against a path string.
import os
import pyfect as fx
from pyfect import platform
from pyfect.file_system import FileSystem
from pyfect.path import Path
@fx.fn
def program():
fs = yield from FileSystem()
path = yield from Path()
# Removed, with everything in it, when the scope closes.
root = yield from fs.make_temp_directory_scoped(prefix="notes-")
notes = path.join(root, "notes.txt")
yield from fs.write_file_string(notes, "milk\neggs\nbread\n")
print(repr((yield from fs.read_file_string(notes))))
# A large file is better read as a stream of byte chunks.
lines = yield from fs.stream(notes, chunk_size=4).decode_text().split_lines().run_collect()
print(lines)
print("exists:", (yield from fs.exists(notes)))
# Paths under a root matching a glob, sorted by name, depth first.
yield from fs.make_directory(path.join(root, "drafts"))
yield from fs.write_file_string(path.join(root, "drafts", "todo.txt"), "")
print((yield from fs.glob("**/*.txt", root=root)))
# An excluded directory is not read at all.
print((yield from fs.glob("**/*.txt", root=root, exclude=("drafts",))))
return root
root = fx.run_main(fx.scoped(program()).pipe(fx.provide(platform.layer)))
assert not os.path.exists(root)Path joins, resolves and splits paths with the rules of Node's path module:
path.posix and path.win32 give what Node's path.posix and path.win32 give, on any host
(checked against every case in Node's own path tests), so path.layer_win32 runs a program
under Windows path rules on macOS or Linux. resolve on Windows rules uses the drive's own
working directory (the =C: variable), as Node does.
These programs run with fx.run_main rather than run_blocking, which only takes effects that
cannot fail: run_main logs a failure and exits the process with a nonzero status.
Watching for changes#
fs.watch(path) is a stream of Create, Update and Remove events for a directory's
entries, or for one file. An event's path is relative to the watched directory (notes.txt,
or drafts/todo.txt in a recursive watch), and a watched file's events carry its name; recursive=True reports
changes in subdirectories too, and on_ready runs once the watcher is installed. The events
are Node's fs.watch's as Effect reads them: an entry that appeared or went away is Create
if it exists afterwards and Remove if not, and a changed one is Update. So a rename is a
Remove of the old name followed by a Create of the new one. With the watch
extra (uv add "pyfect-dev[watch]", on watchfiles) the operating system reports changes as
they happen (inotify, FSEvents, ReadDirectoryChangesW) through WatchfilesBackend, on a thread
of its own that stops when the stream does. Without it, watch compares listings every 50
milliseconds (LiveFileSystem(watch_interval=...)), which reports a change up to that late
and misses a file created and removed between two listings; each listing reads the whole tree,
so on a large one (a .venv, a .git) polling keeps a core busy and the extra is the fix. A WatchBackend provided where the
stream runs, or where the layer is built, takes the path first.
User directories#
UserDirectories gives the directories a program keeps its files in for the current user:
config, data, cache and state, each for an application name, and home. Provide
user_directories.layer; platform.layer doesn't include it, as Effect's NodeServices
doesn't. An absolute XDG_CONFIG_HOME, XDG_DATA_HOME, XDG_CACHE_HOME or XDG_STATE_HOME
wins on every platform. Otherwise macOS uses ~/Library/Application Support (and
~/Library/Caches for the cache), Windows uses APPDATA and LOCALAPPDATA (or AppData
under USERPROFILE, with the cache in a Cache folder), and other systems use the XDG
defaults. The variables are read from the ConfigProvider on every call, so a test can fix
them, and layer_platform(...) picks the platform's rules. Without a home directory a
lookup fails with a NotFound PlatformError.
import pyfect as fx
from pyfect import config_provider
from pyfect.persistence import user_directories
from pyfect.persistence.user_directories import UserDirectories
@fx.fn
def directories():
dirs = yield from UserDirectories()
return (yield from dirs.config("myapp")), (yield from dirs.cache("myapp"))
# The variables come from the ConfigProvider; here a fixed environment.
def on(platform: str, env: dict[str, str]) -> tuple[str, str]:
program = directories().pipe(fx.provide(user_directories.layer_platform(platform)))
environment = config_provider.layer(config_provider.from_env(env=env))
return fx.run_blocking(program.pipe(fx.provide(environment)).or_die())
print(on("linux", {"HOME": "/home/ada"}))
print(on("linux", {"HOME": "/home/ada", "XDG_CONFIG_HOME": "/etc/ada"}))
print(on("darwin", {"HOME": "/Users/ada"}))
print(on("win32", {"APPDATA": "C:\\Users\\ada\\AppData\\Roaming", "USERPROFILE": "C:\\Users\\ada"}))
# The host's own directories: user_directories.layer (platform.layer doesn't include it).
config_dir, _ = fx.run_blocking(directories().pipe(fx.provide(user_directories.layer)).or_die())
assert config_dir.endswith("myapp")
missing = directories().pipe(fx.provide(user_directories.layer_platform("linux")))
error = fx.run_blocking(
missing.flip()
.or_die()
.pipe(fx.provide(config_provider.layer(config_provider.from_env(env={}))))
)
print(error.reason.tag, "-", error.reason.description)Child processes#
process.make(command, args) describes a process. Running it in a scope starts the process
and gives a handle whose stdout and stderr are byte streams and whose stdin is a sink.
As in Node, stdin closes as soon as the process exits, before exit_code reports the exit,
and a write still in progress then fails, as does any write after it. When the scope closes, a process still running is killed. The spawner's string, lines
and stream_lines start a command, read its output and clean up in one call.
import sys
import pyfect as fx
from pyfect import platform, process
from pyfect.process import ChildProcessSpawner
COUNT = "import time\nfor i in range(3):\n print('tick', i, flush=True)\n time.sleep(0.01)"
@fx.fn
def program():
spawner = yield from ChildProcessSpawner()
# Run a command and collect everything it printed.
greeting = yield from spawner.string(process.make(sys.executable, ["-c", "print('hi')"]))
print(repr(greeting))
# Read each line as the child writes it.
ticks = process.make(sys.executable, ["-c", COUNT])
yield from spawner.stream_lines(ticks).run_for_each(lambda line: fx.console.log("got", line))
# Or hold the handle: its stdout and stderr are byte streams.
handle = yield from process.make(sys.executable, ["-c", "import sys; sys.exit(3)"])
print("exit code:", (yield from handle.exit_code))
fx.run_main(fx.scoped(program()).pipe(fx.provide(platform.layer)))pipe_to feeds one command's output into the next, as | does in a shell.
A command can also be a template, written as in a shell: process.make(["git commit -m ", ""], message) gives its text parts and the values between them, as a tagged template does (on
Python 3.14, process.make(t"git commit -m {message}")). Quotes group words and each value is
one literal argument, so a value never splits or expands. Text that needs a shell (|,
$HOME, *, ~) raises CommandLineError where the command is made. With shell=True the
command is the whole script, and arguments next to it are refused with BadArgument:
command_line.join and command_line.quote build a script with values quoted for POSIX
shells, PowerShell or Windows programs. Arguments and env values can be Redacted: the
process gets their values, while errors show commands through command_line.display, which
masks them.
import pyfect as fx
from pyfect import platform, process, redacted
from pyfect.process import ChildProcessSpawner, CommandLineError, command_line
@fx.fn
def program():
spawner = yield from ChildProcessSpawner()
# A template is written as in a shell: quotes group words, and each value
# is one literal argument, whatever it contains.
message = "fix: a | b; $HOME"
echo = process.make(["printf '%s\\n' ", ' "two words"'], message)
print((yield from spawner.string(echo)), end="")
# Text that needs a shell is refused where the command is made.
try:
process.make(["ls *.txt | wc -l"])
except CommandLineError as error:
print(error.reason, "-", error.message)
# A script for a shell: build it with command_line.join, then run it with shell=True.
script = command_line.join(["printf", "%s|", "a b", "it's"]).get_or_raise()
print(script)
print((yield from spawner.string(process.make(script, shell=True))))
# Redacted arguments reach the process but never logs or errors.
token = redacted.make("hunter2")
print(command_line.display(["curl", "-H", token, "https://example.com/a b"]))
fx.run_main(fx.scoped(program()).pipe(fx.provide(platform.layer)))StdoutConfig and StderrConfig wrap an output option, and StdinConfig sets how standard
input ends and the encoding text written to the handle's stdin is encoded with. A
DuplexFd is an extra descriptor both sides read and write, a socket pair; ending its input
shuts down only the writing half, so the child can still answer on it.
HTTP requests#
An HttpClient is a value: methods like filter_status_ok, retry_transient and transform
each give back a new client. retry_transient retries timeouts, transport errors and the
statuses worth retrying (408, 429, 500, 502, 503 and 504), and waits at least as long as a
response's Retry-After asks. timeout fails a request that gets no response in time with a
TransportError (so retry_transient retries it); applied before retry_transient each
attempt gets its own budget, applied after it the budget covers every attempt.
client.is_transient_error and client.is_transient_response classify failures the same way
for a policy of your own. response.schema_body_json decodes the body with a
schema, so a payload of the wrong shape fails with a SchemaError.
This program runs against client.layer_test, which answers each request with a function.
In production, provide httpx_client.layer instead (install the pyfect-dev[httpx] extra).
from typing import TypedDict
import pyfect as fx
from pyfect import schedule
from pyfect import schema as S
from pyfect.http import client, request, response
from pyfect.http.client import HttpClient
class Todo(TypedDict):
id: int
title: str
completed: bool
TodoSchema = S.Struct(Todo)
@fx.fn
def get_todo(todo_id: int):
http = (
(yield from HttpClient())
.filter_status_ok() # a 4xx or 5xx status becomes an HttpClientError
.timeout("2 seconds") # before retry_transient: a fresh budget per attempt
.retry_transient(schedule=schedule.exponential("10 millis"), times=3)
)
res = yield from http.get(f"https://api.example.com/todos/{todo_id}")
return (yield from response.schema_body_json(TodoSchema)(res))
# A stand-in server: busy on the first request, then answers.
attempts: list[str] = []
def handle(req: request.HttpClientRequest):
attempts.append(req.url)
if len(attempts) == 1:
return fx.succeed(response.make(req, status=503))
return fx.succeed(response.make(req, '{"id": 1, "title": "Write docs", "completed": false}'))
todo = fx.run_main(get_todo(1).pipe(fx.provide(client.layer_test(handle))))
print(todo)
print(len(attempts), "attempts")Every request runs in a client span and sends trace headers (see observability); interrupting a request aborts it.
A body is sent as it is pulled when it is a stream (body.stream), and so is a form file
whose content is a function opening a stream (body.File(lambda: chunks, "big.bin")): the
file is read only as fast as the connection takes it, and an interrupted request stops it.
with_cookies_ref(ref) keeps the cookies responses set in a Ref[tuple[Cookie, ...]] and
sends them back the way a browser does (RFC 6265): only to the host that set them, or the
hosts their Domain covers, on their Path, over https when they're Secure, and not after
they expire. cookies.merge_into_array and cookies.to_request_cookie_header are the same
rules on a plain tuple, for seeding a jar or reading one.
import pyfect as fx
from pyfect.http import client, request, response
from pyfect.http.client import HttpClient
from pyfect.http.cookies import Cookie
from pyfect.ref import Ref
@fx.fn
def visit():
jar = yield from Ref[tuple[Cookie, ...]].make(())
http = (yield from HttpClient()).with_cookies_ref(jar)
yield from http.post("https://api.example.com/login")
yield from http.get("https://api.example.com/me")
yield from http.get("https://other.example/") # another host: no cookie
return (yield from jar.get())
# A stand-in server: /login sets a session cookie; each request's Cookie header is recorded.
sent: list[str | None] = []
def handle(req: request.HttpClientRequest):
sent.append(req.headers.get("cookie"))
if req.url.endswith("/login"):
return fx.succeed(response.make(req, headers={"set-cookie": "session=abc; HttpOnly"}))
return fx.succeed(response.make(req))
stored = fx.run_main(visit().pipe(fx.provide(client.layer_test(handle))))
print(sent)
assert stored[0].options is not None and stored[0].options.host_only # no Domain: this host onlyURLs#
fx.Url is JavaScript's URL as an immutable Python value: fx.Url(text, base) parses
and normalizes as the WHATWG URL Standard says (the same results as Node on the
web-platform-tests suite, international domain names included), and raises ValueError
for text that isn't a URL; fx.Url.parse gives None instead and fx.Url.can_parse
asks. It has URL's getters (href, origin, host, pathname, search, hash,
search_params, ...), and two URLs are equal when their href is. str(url) is the
href. pyfect.http.url has Effect's Url functions: set_port, set_search,
set_url_params and the other setters give a new Url, as URL's setters would change
the old one, and mutate hands a function a mutable copy. config.url, the URL
schemas, request URLs and Path.to_file_url / from_file_url all use Url.
import pyfect as fx
from pyfect.http import url
# Parsed and normalized as JavaScript's `new URL(text, base)` does.
api = fx.Url("../v2/users/?page=1#top", "HTTPS://API.Example.com:443/v1/")
print(api.href)
print(api.origin, api.pathname, api.search, api.hash)
print(fx.Url("https://例え.jp/a b").href)
# Immutable: each setter gives a new Url, as Effect's `Url.setX` does.
moved = url.set_url_params(url.set_port(api, 8443), {"page": 2, "q": "a b"})
print(moved)
print(dict(moved.search_params))
assert fx.Url.parse("http://[::1") is None
assert fx.Url("https://a.test/x") == fx.Url("HTTPS://A.TEST/./x")HTTP client errors#
Every failure of a request is an HttpClientError whose reason says what went wrong:
TransportError (the request couldn't be sent or its response received), EncodeError,
InvalidUrlError, StatusCodeError, DecodeError or EmptyBodyError.
fx.catch_reason(HttpClientError, StatusCodeError, f) recovers from one of them. The
reason's cause is the underlying exception, such as a refused connection or a client used
after the scope that owned it closed. error.message is Effect's text and leaves the cause
out; str(error) adds it when the reason has no description of its own.
import pyfect as fx
from pyfect.http import client, request
from pyfect.http.client import HttpClient
from pyfect.http.error import HttpClientError, TransportError
# A stand-in network that refuses every connection, as a server that is down.
def refuse(req: request.HttpClientRequest):
cause = ConnectionRefusedError(61, "Connection refused")
return fx.fail(HttpClientError(TransportError(req, cause)))
@fx.fn
def fetch():
http = (yield from HttpClient()).filter_status_ok()
return (yield from http.get("https://api.example.com/todos/1"))
@fx.fn
def program():
error = yield from fetch().flip()
# The reason says what failed; its cause, why. str() includes the cause when the
# reason has no description of its own.
print(type(error.reason).__name__)
print("message:", error.message)
print("cause: ", repr(error.reason.cause))
print("str: ", error)
fx.run_main(program().pipe(fx.provide(client.layer_test(refuse))))A mocked client in tests#
layer_test makes the stand-in server a one-line fixture. Under testing.effect the clock
is virtual, so a test of a five-second timeout finishes at once.
from typing import TypedDict
import pyfect as fx
from pyfect import schema as S
from pyfect import testing
from pyfect.http import client, request, response
from pyfect.http.client import HttpClient
class Title(TypedDict):
title: str
TitleSchema = S.Struct(Title)
@fx.fn
def get_title(todo_id: int):
http = (yield from HttpClient()).filter_status_ok()
res = yield from http.get(f"https://api.example.com/todos/{todo_id}").timeout("5 seconds")
return (yield from response.schema_body_json(TitleSchema)(res))["title"]
def answer(body: str):
"""A client that answers every request with `body`."""
return client.layer_test(lambda req: fx.succeed(response.make(req, body)))
def hang(_req: request.HttpClientRequest):
return fx.never
@testing.effect
def test_decodes_the_title():
title = yield from get_title(1).pipe(fx.provide(answer('{"title": "Write docs"}')))
assert title == "Write docs"
@testing.effect
def test_rejects_a_bad_body():
error = yield from get_title(1).flip().pipe(fx.provide(answer('{"title": 1}')))
assert isinstance(error, S.SchemaError)
@testing.effect
def test_times_out():
fiber = yield from get_title(1).flip().pipe(fx.provide(client.layer_test(hang))).fork()
yield from testing.adjust("5 seconds") # virtual time: no real waiting
assert isinstance((yield from fiber.join()), fx.TimeoutException)Reference: pyfect.file_system,
pyfect.glob,
pyfect.process, pyfect.http,
pyfect.platform.