GitHub

tokenizer

API reference for tokenizer.

Counting and truncating a prompt's tokens (Effect's Tokenizer).

Re-exports

Types

TypeDefinitionDescription
Tokenizetype Tokenize = Callable[[P.Prompt], Effect[Sequence[int], AiError]]

Classes

Functions

make

make(*, tokenize) -> Service
ParameterTypeDescription
tokenizeTokenize

Returns Service

def make(*, tokenize: Tokenize) -> Service:
    return Service(tokenize)