GitHub

Platform

Files, child processes and HTTP requests as services you can swap in tests.

Programs reach the operating system through services: FileSystem and Path for files, ChildProcessSpawner for processes, HttpClient for HTTP. Your code asks for the service and the layer you provide decides what answers, so a test can swap the network or the disk for a stand-in. Every file and process operation fails with a typed PlatformError whose reason says what went wrong (NotFound, PermissionDenied, AlreadyExists, ...).

FileSystem, PathChildProcessSpawnerStdio, TerminalHttpClientplatform.layeryour programhttpx_client.layer, or layer_testin tests
FileSystem, PathChildProcessSpawnerStdio, TerminalHttpClientplatform.layeryour programhttpx_client.layer, or layer_testin tests

Files

fs.read_file_string and fs.write_file_string read and write a whole file; fs.stream reads one as a stream of byte chunks, so memory stays bounded however large it is, and fs.sink writes one from a stream. make_temp_directory_scoped gives a directory that is removed, with everything in it, when its scope closes. fs.glob finds the paths under a root that match a pattern (** crosses directories, {a,b} and [a-z] choose, dot=True includes names starting with .). exclude= takes patterns of paths to leave out; a directory that matches one is not read at all, so excluding .venv or node_modules saves walking them. pyfect.glob matches the same patterns against a path string.

files.py
import os

import pyfect as fx
from pyfect import platform
from pyfect.file_system import FileSystem
from pyfect.path import Path


@fx.fn
def program():
    fs = yield from FileSystem()
    path = yield from Path()
    # Removed, with everything in it, when the scope closes.
    root = yield from fs.make_temp_directory_scoped(prefix="notes-")
    notes = path.join(root, "notes.txt")

    yield from fs.write_file_string(notes, "milk\neggs\nbread\n")
    print(repr((yield from fs.read_file_string(notes))))

    # A large file is better read as a stream of byte chunks.
    lines = yield from fs.stream(notes, chunk_size=4).decode_text().split_lines().run_collect()
    print(lines)
    print("exists:", (yield from fs.exists(notes)))

    # Paths under a root matching a glob, sorted by name, depth first.
    yield from fs.make_directory(path.join(root, "drafts"))
    yield from fs.write_file_string(path.join(root, "drafts", "todo.txt"), "")
    print((yield from fs.glob("**/*.txt", root=root)))
    # An excluded directory is not read at all.
    print((yield from fs.glob("**/*.txt", root=root, exclude=("drafts",))))
    return root


root = fx.run_main(fx.scoped(program()).pipe(fx.provide(platform.layer)))
assert not os.path.exists(root)

Path joins, resolves and splits paths with the rules of Node's path module: path.posix and path.win32 give what Node's path.posix and path.win32 give, on any host (checked against every case in Node's own path tests), so path.layer_win32 runs a program under Windows path rules on macOS or Linux. resolve on Windows rules uses the drive's own working directory (the =C: variable), as Node does.

These programs run with fx.run_main rather than run_blocking, which only takes effects that cannot fail: run_main logs a failure and exits the process with a nonzero status.

Watching for changes

fs.watch(path) is a stream of Create, Update and Remove events for a directory's entries, or for one file. An event's path is relative to the watched directory (notes.txt, or drafts/todo.txt in a recursive watch), and a watched file's events carry its name; recursive=True reports changes in subdirectories too, and on_ready runs once the watcher is installed. The events are Node's fs.watch's as Effect reads them: an entry that appeared or went away is Create if it exists afterwards and Remove if not, and a changed one is Update. So a rename is a Remove of the old name followed by a Create of the new one. With the watch extra (uv add "pyfect-dev[watch]", on watchfiles) the operating system reports changes as they happen (inotify, FSEvents, ReadDirectoryChangesW) through WatchfilesBackend, on a thread of its own that stops when the stream does. Without it, watch compares listings every 50 milliseconds (LiveFileSystem(watch_interval=...)), which reports a change up to that late and misses a file created and removed between two listings; each listing reads the whole tree, so on a large one (a .venv, a .git) polling keeps a core busy and the extra is the fix. A WatchBackend provided where the stream runs, or where the layer is built, takes the path first.

User directories

UserDirectories gives the directories a program keeps its files in for the current user: config, data, cache and state, each for an application name, and home. Provide user_directories.layer; platform.layer doesn't include it, as Effect's NodeServices doesn't. An absolute XDG_CONFIG_HOME, XDG_DATA_HOME, XDG_CACHE_HOME or XDG_STATE_HOME wins on every platform. Otherwise macOS uses ~/Library/Application Support (and ~/Library/Caches for the cache), Windows uses APPDATA and LOCALAPPDATA (or AppData under USERPROFILE, with the cache in a Cache folder), and other systems use the XDG defaults. The variables are read from the ConfigProvider on every call, so a test can fix them, and layer_platform(...) picks the platform's rules. Without a home directory a lookup fails with a NotFound PlatformError.

user_directories.py
import pyfect as fx
from pyfect import config_provider
from pyfect.persistence import user_directories
from pyfect.persistence.user_directories import UserDirectories


@fx.fn
def directories():
    dirs = yield from UserDirectories()
    return (yield from dirs.config("myapp")), (yield from dirs.cache("myapp"))


# The variables come from the ConfigProvider; here a fixed environment.
def on(platform: str, env: dict[str, str]) -> tuple[str, str]:
    program = directories().pipe(fx.provide(user_directories.layer_platform(platform)))
    environment = config_provider.layer(config_provider.from_env(env=env))
    return fx.run_blocking(program.pipe(fx.provide(environment)).or_die())


print(on("linux", {"HOME": "/home/ada"}))
print(on("linux", {"HOME": "/home/ada", "XDG_CONFIG_HOME": "/etc/ada"}))
print(on("darwin", {"HOME": "/Users/ada"}))
print(on("win32", {"APPDATA": "C:\\Users\\ada\\AppData\\Roaming", "USERPROFILE": "C:\\Users\\ada"}))

# The host's own directories: user_directories.layer (platform.layer doesn't include it).
config_dir, _ = fx.run_blocking(directories().pipe(fx.provide(user_directories.layer)).or_die())
assert config_dir.endswith("myapp")

missing = directories().pipe(fx.provide(user_directories.layer_platform("linux")))
error = fx.run_blocking(
    missing.flip()
    .or_die()
    .pipe(fx.provide(config_provider.layer(config_provider.from_env(env={}))))
)
print(error.reason.tag, "-", error.reason.description)

Child processes

process.make(command, args) describes a process. Running it in a scope starts the process and gives a handle whose stdout and stderr are byte streams and whose stdin is a sink. As in Node, stdin closes as soon as the process exits, before exit_code reports the exit, and a write still in progress then fails, as does any write after it. When the scope closes, a process still running is killed. The spawner's string, lines and stream_lines start a command, read its output and clean up in one call.

child.py
import sys

import pyfect as fx
from pyfect import platform, process
from pyfect.process import ChildProcessSpawner

COUNT = "import time\nfor i in range(3):\n    print('tick', i, flush=True)\n    time.sleep(0.01)"


@fx.fn
def program():
    spawner = yield from ChildProcessSpawner()

    # Run a command and collect everything it printed.
    greeting = yield from spawner.string(process.make(sys.executable, ["-c", "print('hi')"]))
    print(repr(greeting))

    # Read each line as the child writes it.
    ticks = process.make(sys.executable, ["-c", COUNT])
    yield from spawner.stream_lines(ticks).run_for_each(lambda line: fx.console.log("got", line))

    # Or hold the handle: its stdout and stderr are byte streams.
    handle = yield from process.make(sys.executable, ["-c", "import sys; sys.exit(3)"])
    print("exit code:", (yield from handle.exit_code))


fx.run_main(fx.scoped(program()).pipe(fx.provide(platform.layer)))

pipe_to feeds one command's output into the next, as | does in a shell.

A command can also be a template, written as in a shell: process.make(["git commit -m ", ""], message) gives its text parts and the values between them, as a tagged template does (on Python 3.14, process.make(t"git commit -m {message}")). Quotes group words and each value is one literal argument, so a value never splits or expands. Text that needs a shell (|, $HOME, *, ~) raises CommandLineError where the command is made. With shell=True the command is the whole script, and arguments next to it are refused with BadArgument: command_line.join and command_line.quote build a script with values quoted for POSIX shells, PowerShell or Windows programs. Arguments and env values can be Redacted: the process gets their values, while errors show commands through command_line.display, which masks them.

command_line.py
import pyfect as fx
from pyfect import platform, process, redacted
from pyfect.process import ChildProcessSpawner, CommandLineError, command_line


@fx.fn
def program():
    spawner = yield from ChildProcessSpawner()

    # A template is written as in a shell: quotes group words, and each value
    # is one literal argument, whatever it contains.
    message = "fix: a | b; $HOME"
    echo = process.make(["printf '%s\\n' ", ' "two  words"'], message)
    print((yield from spawner.string(echo)), end="")

    # Text that needs a shell is refused where the command is made.
    try:
        process.make(["ls *.txt | wc -l"])
    except CommandLineError as error:
        print(error.reason, "-", error.message)

    # A script for a shell: build it with command_line.join, then run it with shell=True.
    script = command_line.join(["printf", "%s|", "a b", "it's"]).get_or_raise()
    print(script)
    print((yield from spawner.string(process.make(script, shell=True))))

    # Redacted arguments reach the process but never logs or errors.
    token = redacted.make("hunter2")
    print(command_line.display(["curl", "-H", token, "https://example.com/a b"]))


fx.run_main(fx.scoped(program()).pipe(fx.provide(platform.layer)))

StdoutConfig and StderrConfig wrap an output option, and StdinConfig sets how standard input ends and the encoding text written to the handle's stdin is encoded with. A DuplexFd is an extra descriptor both sides read and write, a socket pair; ending its input shuts down only the writing half, so the child can still answer on it.

HTTP requests

An HttpClient is a value: methods like filter_status_ok, retry_transient and transform each give back a new client. retry_transient retries timeouts, transport errors and the statuses worth retrying (408, 429, 500, 502, 503 and 504), and waits at least as long as a response's Retry-After asks. timeout fails a request that gets no response in time with a TransportError (so retry_transient retries it); applied before retry_transient each attempt gets its own budget, applied after it the budget covers every attempt. client.is_transient_error and client.is_transient_response classify failures the same way for a policy of your own. response.schema_body_json decodes the body with a schema, so a payload of the wrong shape fails with a SchemaError.

This program runs against client.layer_test, which answers each request with a function. In production, provide httpx_client.layer instead (install the pyfect-dev[httpx] extra).

todos.py
from typing import TypedDict

import pyfect as fx
from pyfect import schedule
from pyfect import schema as S
from pyfect.http import client, request, response
from pyfect.http.client import HttpClient


class Todo(TypedDict):
    id: int
    title: str
    completed: bool


TodoSchema = S.Struct(Todo)


@fx.fn
def get_todo(todo_id: int):
    http = (
        (yield from HttpClient())
        .filter_status_ok()  # a 4xx or 5xx status becomes an HttpClientError
        .timeout("2 seconds")  # before retry_transient: a fresh budget per attempt
        .retry_transient(schedule=schedule.exponential("10 millis"), times=3)
    )
    res = yield from http.get(f"https://api.example.com/todos/{todo_id}")
    return (yield from response.schema_body_json(TodoSchema)(res))


# A stand-in server: busy on the first request, then answers.
attempts: list[str] = []


def handle(req: request.HttpClientRequest):
    attempts.append(req.url)
    if len(attempts) == 1:
        return fx.succeed(response.make(req, status=503))
    return fx.succeed(response.make(req, '{"id": 1, "title": "Write docs", "completed": false}'))


todo = fx.run_main(get_todo(1).pipe(fx.provide(client.layer_test(handle))))
print(todo)
print(len(attempts), "attempts")

Every request runs in a client span and sends trace headers (see observability); interrupting a request aborts it.

A body is sent as it is pulled when it is a stream (body.stream), and so is a form file whose content is a function opening a stream (body.File(lambda: chunks, "big.bin")): the file is read only as fast as the connection takes it, and an interrupted request stops it.

with_cookies_ref(ref) keeps the cookies responses set in a Ref[tuple[Cookie, ...]] and sends them back the way a browser does (RFC 6265): only to the host that set them, or the hosts their Domain covers, on their Path, over https when they're Secure, and not after they expire. cookies.merge_into_array and cookies.to_request_cookie_header are the same rules on a plain tuple, for seeding a jar or reading one.

cookies.py
import pyfect as fx
from pyfect.http import client, request, response
from pyfect.http.client import HttpClient
from pyfect.http.cookies import Cookie
from pyfect.ref import Ref


@fx.fn
def visit():
    jar = yield from Ref[tuple[Cookie, ...]].make(())
    http = (yield from HttpClient()).with_cookies_ref(jar)
    yield from http.post("https://api.example.com/login")
    yield from http.get("https://api.example.com/me")
    yield from http.get("https://other.example/")  # another host: no cookie
    return (yield from jar.get())


# A stand-in server: /login sets a session cookie; each request's Cookie header is recorded.
sent: list[str | None] = []


def handle(req: request.HttpClientRequest):
    sent.append(req.headers.get("cookie"))
    if req.url.endswith("/login"):
        return fx.succeed(response.make(req, headers={"set-cookie": "session=abc; HttpOnly"}))
    return fx.succeed(response.make(req))


stored = fx.run_main(visit().pipe(fx.provide(client.layer_test(handle))))
print(sent)
assert stored[0].options is not None and stored[0].options.host_only  # no Domain: this host only

URLs

fx.Url is JavaScript's URL as an immutable Python value: fx.Url(text, base) parses and normalizes as the WHATWG URL Standard says (the same results as Node on the web-platform-tests suite, international domain names included), and raises ValueError for text that isn't a URL; fx.Url.parse gives None instead and fx.Url.can_parse asks. It has URL's getters (href, origin, host, pathname, search, hash, search_params, ...), and two URLs are equal when their href is. str(url) is the href. pyfect.http.url has Effect's Url functions: set_port, set_search, set_url_params and the other setters give a new Url, as URL's setters would change the old one, and mutate hands a function a mutable copy. config.url, the URL schemas, request URLs and Path.to_file_url / from_file_url all use Url.

urls.py
import pyfect as fx
from pyfect.http import url

# Parsed and normalized as JavaScript's `new URL(text, base)` does.
api = fx.Url("../v2/users/?page=1#top", "HTTPS://API.Example.com:443/v1/")
print(api.href)
print(api.origin, api.pathname, api.search, api.hash)
print(fx.Url("https://例え.jp/a b").href)

# Immutable: each setter gives a new Url, as Effect's `Url.setX` does.
moved = url.set_url_params(url.set_port(api, 8443), {"page": 2, "q": "a b"})
print(moved)
print(dict(moved.search_params))

assert fx.Url.parse("http://[::1") is None
assert fx.Url("https://a.test/x") == fx.Url("HTTPS://A.TEST/./x")

HTTP client errors

Every failure of a request is an HttpClientError whose reason says what went wrong: TransportError (the request couldn't be sent or its response received), EncodeError, InvalidUrlError, StatusCodeError, DecodeError or EmptyBodyError. fx.catch_reason(HttpClientError, StatusCodeError, f) recovers from one of them. The reason's cause is the underlying exception, such as a refused connection or a client used after the scope that owned it closed. error.message is Effect's text and leaves the cause out; str(error) adds it when the reason has no description of its own.

http_errors.py
import pyfect as fx
from pyfect.http import client, request
from pyfect.http.client import HttpClient
from pyfect.http.error import HttpClientError, TransportError


# A stand-in network that refuses every connection, as a server that is down.
def refuse(req: request.HttpClientRequest):
    cause = ConnectionRefusedError(61, "Connection refused")
    return fx.fail(HttpClientError(TransportError(req, cause)))


@fx.fn
def fetch():
    http = (yield from HttpClient()).filter_status_ok()
    return (yield from http.get("https://api.example.com/todos/1"))


@fx.fn
def program():
    error = yield from fetch().flip()
    # The reason says what failed; its cause, why. str() includes the cause when the
    # reason has no description of its own.
    print(type(error.reason).__name__)
    print("message:", error.message)
    print("cause:  ", repr(error.reason.cause))
    print("str:    ", error)


fx.run_main(program().pipe(fx.provide(client.layer_test(refuse))))

A mocked client in tests

layer_test makes the stand-in server a one-line fixture. Under testing.effect the clock is virtual, so a test of a five-second timeout finishes at once.

test_todos.py
from typing import TypedDict

import pyfect as fx
from pyfect import schema as S
from pyfect import testing
from pyfect.http import client, request, response
from pyfect.http.client import HttpClient


class Title(TypedDict):
    title: str


TitleSchema = S.Struct(Title)


@fx.fn
def get_title(todo_id: int):
    http = (yield from HttpClient()).filter_status_ok()
    res = yield from http.get(f"https://api.example.com/todos/{todo_id}").timeout("5 seconds")
    return (yield from response.schema_body_json(TitleSchema)(res))["title"]


def answer(body: str):
    """A client that answers every request with `body`."""
    return client.layer_test(lambda req: fx.succeed(response.make(req, body)))


def hang(_req: request.HttpClientRequest):
    return fx.never


@testing.effect
def test_decodes_the_title():
    title = yield from get_title(1).pipe(fx.provide(answer('{"title": "Write docs"}')))
    assert title == "Write docs"


@testing.effect
def test_rejects_a_bad_body():
    error = yield from get_title(1).flip().pipe(fx.provide(answer('{"title": 1}')))
    assert isinstance(error, S.SchemaError)


@testing.effect
def test_times_out():
    fiber = yield from get_title(1).flip().pipe(fx.provide(client.layer_test(hang))).fork()
    yield from testing.adjust("5 seconds")  # virtual time: no real waiting
    assert isinstance((yield from fiber.join()), fx.TimeoutException)

Reference: pyfect.file_system, pyfect.glob, pyfect.process, pyfect.http, pyfect.platform.