# Quickstart This page runs a pretrained PyTorch MNIST classifier on encrypted images on a Belfort GPU. You play both roles: the model provider, who compiles and deploys the model, and the end client, who encrypts images and decrypts the answers.  ## What you need - macOS 14 or newer on Apple Silicon, or Linux (x86-64 or ARM) on a recent distribution - Python 3.12 or newer - A C++20 compiler ## 1. Install Create a virtual environment with Python 3.12 or newer (e.g., see [uv's Python guide](https://docs.astral.sh/uv/guides/install-python/)): ```bash uv venv --python 3.12 source .venv/bin/activate ``` Install the Belfort packages, along with the Hugging Face and image libraries the example uses to download the model and the test images: ```bash uv pip install --torch-backend=cpu belfort-ml belfort-ml-client huggingface_hub safetensors pyarrow pillow ``` Without uv, create the environment with `python3.12 -m venv .venv`, install with pip from PyTorch's CPU index, and run the scripts below with `python` in place of `uv run`: ```bash pip install --extra-index-url https://download.pytorch.org/whl/cpu belfort-ml belfort-ml-client huggingface_hub safetensors pyarrow pillow ``` ## 2. Get an API key Sign in to the [Belfort console](https://eml.belfortlabs.com/api-keys), open **API keys**, and create one. The key (`blf_live_...`) is shown once. Set it in the shell you will run `provider.py` from: ```bash title="Provider terminal" export BELFORT_API_KEY=blf_live_... ``` ## 3. The model and the data > **Compilation limits** > > Take into consideration the following limits when > writing your own model: > > - One encrypted input per model > - No grouped convolutions, except depthwise (`groups == in_channels == out_channels`) > - No MaxPool: use AvgPool > - No softmax: apply it to the logits after decryption > - Shallow networks: every layer adds compile time and bootstraps > > We're continuously working on addressing these limitations! > > [More information on what compiles](/docs/concepts/compilation-pipeline.md#what-compiles). `mnist.py` downloads a pretrained MNIST classifier and the MNIST test images from Hugging Face; no account is needed. Save it in the directory that holds `.venv`: ```python title="mnist.py" """The model and the test images, both downloaded from Hugging Face.""" import io import numpy as np import pyarrow.parquet as pq import torch from huggingface_hub import hf_hub_download from PIL import Image from safetensors.torch import load_file from torch import nn class MLP(nn.Module): """dacorvo/mnist-mlp: a 28x28 image, flattened, to ten digit scores.""" def __init__(self) -> None: super().__init__() self.input_layer = nn.Linear(784, 256) self.mid_layer = nn.Linear(256, 256) self.output_layer = nn.Linear(256, 10) def forward(self, x: torch.Tensor) -> torch.Tensor: x = torch.relu(self.input_layer(x)) x = torch.relu(self.mid_layer(x)) return self.output_layer(x) def load_model() -> MLP: weights = hf_hub_download( "dacorvo/mnist-mlp", "model.safetensors", revision="7ed1e96d182395fd7380ea8495b9eb2364ce9dc1", ) net = MLP() net.load_state_dict(load_file(weights)) return net.eval() def test_images() -> tuple[torch.Tensor, list[int]]: """The 10,000 MNIST test images, normalized and flattened as the model expects.""" path = hf_hub_download( "ylecun/mnist", "mnist/test-00000-of-00001.parquet", repo_type="dataset", revision="77f3279092a1c1579b2250db8eafed0ad422088c", ) table = pq.read_table(path) pngs = table["image"].to_pylist() pixels = np.stack([np.array(Image.open(io.BytesIO(png["bytes"]))) for png in pngs]) pixels = pixels.astype(np.float32) / 255 images = torch.from_numpy((pixels - 0.1307) / 0.3081).flatten(1) return images, table["label"].to_pylist() ``` ## 4. Compile Save `provider.py` next to it: ```python title="provider.py" """The model provider: compile the model, then serve it to one end client. uv run provider.py compile # compile the model on Belfort uv run provider.py serve # mint a client token and start a GPU deploy uv run provider.py serve --dev # testing only: the gateway chooses the seed Both are safe to run again: a model already compiled from the same MLIR is reused, not compiled again. """ import hashlib import sys from datetime import timedelta from pathlib import Path import belfort_ml from mnist import load_model, test_images NAME = "quickstart_mnist" # client.py on this machine reads the token from here; it exists while serve # runs. TOKEN_FILE = Path(".client-token") def compiled_model(provider: belfort_ml.Provider) -> belfort_ml.Model: images, _ = test_images() # The last 512 test images; client.py sends the first ones. calibration = list(images[-512:].split(1)) artifacts = belfort_ml.compile( load_model(), calibration[0], NAME, "./build", calibration_inputs=calibration ) return provider.models.get( name=NAME, create_from=artifacts, timeout=1800, use_composite_relu=False ) def hand_over(token: str, seed: str | None = None) -> None: """Write the token, and a developer-mode seed where client.py looks for it.""" TOKEN_FILE.touch(mode=0o600) TOKEN_FILE.write_text(token) export = f" export BELFORT_CLIENT_TOKEN={token}\n" if seed is not None: digest = hashlib.sha256(token.encode()).hexdigest()[:12] seed_file = Path(f".secret-seed-{digest}") seed_file.touch(mode=0o600) seed_file.write_text(seed) export += f" export BELFORT_CLIENT_SECRET_SEED={seed}\n" print( f"\nStart client.py now. On this machine it reads the token from" f" {TOKEN_FILE}.\nOn another machine, run this before client.py:\n{export}", flush=True, ) def serve(model: belfort_ml.Model, dev: bool = False) -> None: TOKEN_FILE.unlink(missing_ok=True) # a killed earlier run may have left one try: token = None if dev else model.create_client(label="quickstart") if token is not None: hand_over(token.token) # The GPU bills from here, so start it only once the client's keys # are in. model.wait_for_keys(token) print("Keys uploaded. Starting the deploy; Ctrl-C stops it.", flush=True) with model.deploy( key_set=token, max_lifetime=timedelta(minutes=30), timeout=1800, dev_mode=dev, ) as deploy: if dev: # The gateway supplies the developer seed and a client token. assert deploy.client_token is not None hand_over(deploy.client_token, deploy.secret_seed) input("Deploy ready. Press Enter to stop it...") finally: TOKEN_FILE.unlink(missing_ok=True) if __name__ == "__main__": command = sys.argv[1] if len(sys.argv) > 1 else "" dev = sys.argv[2:] == ["--dev"] if command not in ("compile", "serve") or (sys.argv[2:] and not dev): sys.exit(__doc__) with belfort_ml.Provider() as provider: # reads BELFORT_API_KEY model = compiled_model(provider) if command == "compile": print(f"{NAME} is compiled ({model.id})") else: serve(model, dev) ``` Compile the model: ```bash title="Provider terminal" uv run provider.py compile ``` `belfort_ml.compile` traces the model on your machine and runs the example images through it to calibrate the value ranges. The calibration images are test images, so they do not include the ones the end client sends. It writes `build/quickstart_mnist_torch.mlir`. `provider.models.get(name=NAME, create_from=artifacts)` returns the named mode, or uploads that file to the Belfort gateway and waits while Belfort generates the GPU code and the end client's code. Belfort receives the MLIR file, not your Python code. Running the command again reuses the compiled model as long as the MLIR is unchanged. ## 5. Serve ```bash title="Provider terminal" uv run provider.py serve ``` This compiles the model first if needed, mints a client token, writes it to `.client-token` in this directory, and prints it as `export BELFORT_CLIENT_TOKEN=...`. In production, you send the token to the end client yourself. The script then waits with the GPU deploy, until the end client has uploaded its evaluation keys. Only then does it start a GPU deploy that serves that token's keys, and it waits for Enter. The file is removed when the script exits. The deploy runs at most 30 minutes, and you are charged for the time it ran (rounded up to 10 minutes). Stop the model by pressing enter when you finished your inferences. ## 6. Run the end client Save `client.py` in the same directory: ```python title="client.py" """The end client: encrypt MNIST test images, have the deploy classify them, and decrypt the scores. uv run client.py [COUNT] The secret key derives from a seed this script keeps in a local file per token, so running it again on the same token reuses the key pair. """ import hashlib import os import sys from pathlib import Path from belfort_ml_client import Client from mnist import test_images token_file = Path(".client-token") if token_file.exists(): token = token_file.read_text().strip() elif not (token := os.environ.get("BELFORT_CLIENT_TOKEN")): sys.exit("Run `uv run provider.py serve` here first, or set BELFORT_CLIENT_TOKEN.") seed_file = Path(f".secret-seed-{hashlib.sha256(token.encode()).hexdigest()[:12]}") seed = bytes.fromhex(seed_file.read_text()) if seed_file.exists() else None images, labels = test_images() count = int(sys.argv[1]) if len(sys.argv) > 1 else 10 with Client(token, secret_seed=seed) as client: client.prepare() if seed is None: seed_file.touch(mode=0o600) seed_file.write_text(client.secret_seed.hex()) correct = 0 for image, label in zip(images[:count], labels[:count]): answer = client.infer([image.tolist()]) scores = answer["values"] predicted = scores.index(max(scores)) correct += predicted == label print(f"digit {label}: predicted {predicted} in {answer['latency_ms']} ms") print(f"{correct} of {count} correct") ``` Run the client in a second terminal, in the same directory: ```bash title="Client terminal" uv run client.py ``` `client.py` reads the token from `.client-token`, even if `BELFORT_CLIENT_TOKEN` is set. Without the file, it takes `BELFORT_CLIENT_TOKEN`. > **Running the client on another machine** > > The end client normally runs on a different machine than the provider, and > needs no API key there. On that machine: > > 1. Do [step 1](#1-install) to install the packages. > 2. Save `mnist.py` from [step 3](#3-the-model-and-the-data) and `client.py` > below in the directory that holds `.venv`. > 3. Copy the `export BELFORT_CLIENT_TOKEN=...` line the provider printed, > run it in that machine's terminal, then run `uv run client.py`. > > Keep the provider running until the client finishes: it starts the deploy > once the client's keys are uploaded, then waits at **Press Enter**. The client builds the encryption code generated for your model, generates a key pair, and uploads the evaluation keys. This takes about 1.5 minutes on the first run, and happens before the deploy starts; the client then waits for the deploy. Then it encrypts each test image, sends it, decrypts the ten scores, and prints the predicted digit with `latency_ms`, the time the gateway's call to the GPU took: ```text title="Output" digit 7: predicted 7 in 287 ms digit 2: predicted 2 in 189 ms ... ``` `uv run client.py 20` classifies 20 images instead of 10 and ends with a line such as `18 of 20 correct`. Running it again on the same token reuses the key pair and takes about 20 seconds. The secret key exists only on the end client's machine, so neither you nor Belfort sees the images or the scores. Press Enter in the first terminal to stop the deploy. ## Testing with developer mode ```bash title="Provider terminal" uv run provider.py serve --dev ``` `--dev` skips the key upload: the gateway chooses the secret seed and the GPU generates the evaluation keys itself. Once the deploy is ready, the script writes the token to `.client-token` and the seed to the `.secret-seed-...` file `client.py` reads, and prints both as `export` lines for another machine. Then run `uv run client.py` as above; it only builds the client code and starts inferring. > **Testing only** > > In developer mode Belfort knows the secret seed, so it can decrypt every > input and output of the deploy. Never send real data to a developer-mode > deploy. See [Developer mode](/docs/concepts/deploys-and-credits.md#developer-mode). ## Next - [Compilation pipeline](/docs/concepts/compilation-pipeline.md): which models compile, and what each operation costs - [The Belfort ML client](/docs/sdk/belfort-ml-client.md): the end client from Python, and the input format for models with several arguments - [Deploys and credits](/docs/concepts/deploys-and-credits.md): how deploys are billed - [Examples](/docs/getting-started/examples.md): more example models --- # Introduction Belfort runs your PyTorch model on data that stays encrypted. Your end client encrypts an input on their own machine, a Belfort GPU runs the model on the ciphertext, and the end client decrypts the result. The secret key never leaves the end client, so neither you nor Belfort can read the input or the result. The encryption is fully homomorphic encryption (FHE) in the CKKS scheme, which computes additions and multiplications on encrypted real numbers. Any other operation, such as a ReLU, is replaced by a polynomial approximation, which costs time and some accuracy. ## Who it is for Model providers whose end clients cannot hand over their raw data, such as health records, financial transactions or sensor readings, but want the provider's predictions on it. You never hold the end client's data, and the model runs on Belfort's GPUs. ## Which models compile A trained `nn.Module` with one encrypted input of a fixed shape, built from linear layers, convolutions, average pooling, batch normalization and elementwise activations. Depthwise convolutions compile. Max pooling, other grouped convolutions and softmax are not supported yet. [What compiles](/docs/concepts/compilation-pipeline.md#what-compiles) lists the supported and unsupported operations and what each costs. ## The path from a model to an encrypted inference 1. Compile locally: `belfort_ml.compile` traces your `nn.Module` and writes it out as MLIR, without a network call. 2. Upload and compile for FHE: `provider.models.get(name=..., create_from=artifacts)` uploads the MLIR, or returns the model already compiled from it. Belfort's compiler chooses encryption parameters and generates GPU code for the server and C++ for the Belfort ML client (key generation, encryption, decryption). 3. Mint a client token: `model.create_client` returns a token that lets one end client use this model through the Belfort gateway. Belfort does not deliver it: you pass it to the end client yourself, the same way you already exchange data with them. 4. Deploy: `model.deploy` starts a GPU instance with your model. 5. Infer: your end client runs the Belfort ML client on the token, as the `belfort-ml-client infer` command or `belfort_ml_client.Client` in Python. It downloads the client-side C++ for your model, builds it, generates a key pair, and uploads the public evaluation keys, which the gateway loads onto your deploy. It then encrypts, sends the ciphertext to the deploy, and decrypts the answer. The secret key stays in that process, and neither the input nor the answer passes through your code. The [Quickstart](/docs/getting-started/quickstart.md) walks through all five steps with a pretrained MNIST classifier, running the Belfort ML client yourself in place of an end client. A coding agent such as Claude Code can follow these steps once you install the [Belfort skill](/docs/getting-started/coding-agents.md). ## Two packages | Package | Who runs it | What it does | |---|---|---| | `belfort-ml` | You, the model provider | Compiles models, and drives the Belfort gateway: models, client tokens, deploys, credits | | `belfort-ml-client` | Your end client | Holds the secret key: on a client token from you, builds the client code, generates keys, encrypts, decrypts | [Model providers and end clients](/docs/concepts/providers-and-end-clients.md) explains what crosses between them. --- # Installation `belfort-ml` and `belfort-ml-client` import as `belfort_ml` and `belfort_ml_client`. Both depend on `belfort-ml-bucket`, the models bucket layout they share; neither depends on the other. ## belfort-ml In a virtual environment with Python 3.12 or newer, e.g. created with `uv venv --python 3.12`: ```bash uv pip install --torch-backend=cpu belfort-ml ``` With pip, take `torch` from PyTorch's CPU index; on Linux, the default index would pull the CUDA build: ```bash pip install --extra-index-url https://download.pytorch.org/whl/cpu belfort-ml ``` It pulls in `torch`, `belfort-torch-mlir`, `httpx` and `belfort-ml-bucket`. Compiling runs on the CPU, on macOS on Apple Silicon or Linux (x86-64 or ARM). ## An API key Sign in to the Belfort console at [eml.belfortlabs.com](https://eml.belfortlabs.com), open **API keys**, and create one. The key (`blf_live_...`) is shown once. An account holds up to five active keys. Keep the key out of source. The SDK reads it from the environment: ```bash export BELFORT_API_KEY=blf_live_... ``` Check the key with `belfort-ml`: ```python title="check_key.py" import belfort_ml with belfort_ml.Provider() as provider: account = provider.account.get() print(account["mp_id"], account["credits"]["available"]) ``` A new account starts with a few credits; [Deploys and credits](/docs/concepts/deploys-and-credits.md) has the details. ## The Belfort ML client ```bash uv pip install belfort-ml-client # or: pip install belfort-ml-client ``` To run the client yourself while testing, as the [Quickstart](/docs/getting-started/quickstart.md) does, install it next to `belfort-ml`. The Belfort ML client compiles the client-side C++ generated for your model and links it against the Cyclops client library. It needs no GPU; the machine needs: - macOS 14 or newer on Apple Silicon, or Linux (x86-64 or ARM) on a recent distribution. On ARM, encryption and decryption are slower than on x86-64. - A C++20 compiler On these platforms `belfort-ml-client` installs a platform wheel that contains the Cyclops client library and its headers, with libtommath linked in. Neither a Cyclops build nor libtommath needs to be installed. [The Belfort ML client](/docs/sdk/belfort-ml-client.md) covers running it. --- # Examples The examples are in https://github.com/belfortlabs/eml-examples. Each `vX.Y.Z` tag of the repository pins them to that Belfort release. The paths below are relative to its root: ```bash git clone https://github.com/belfortlabs/eml-examples cd eml-examples ``` `quickstart/` holds the [Quickstart](/docs/getting-started/quickstart.md)'s files as a standalone uv project; its README has the steps. The directories below hold trained models with their definitions and 2000 samples of test data each. | Example | What it is | Input shape | Activations | |---|---|---|---| | `mlp` | A three-layer MLP classifying MNIST digits | `(1, 1, 28, 28)` | squares | | `lola` | The LoLA CNN on MNIST: one convolution block, then two linear layers | `(1, 1, 28, 28)` | squares | Each has `.py` (the `nn.Module`), the trained model (`.pth`), and `test_data.npz` with the inputs under `X` and the labels under `y`. ## Compiling one The `.pth` files hold whole pickled modules, so the example's own module has to be importable when you load one. From the examples' root: ```python title="compile_mlp.py" import sys import numpy as np import torch import belfort_ml sys.path.insert(0, "mlp") net = torch.load("mlp/mlp.pth", weights_only=False).eval() samples = np.load("mlp/test_data.npz")["X"][:512] calibration = [torch.from_numpy(x).float().unsqueeze(0) for x in samples] artifacts = belfort_ml.compile( net, calibration[0], "mlp", "./build", calibration_inputs=calibration ) ``` From there, upload it with `provider.models.get(name="mlp", create_from=artifacts)` as `provider.py` does in step 4 of the [Quickstart](/docs/getting-started/quickstart.md). The end client passes each input as a flat list: `client.infer([samples[0].flatten().tolist()])` for the first test image. --- # Coding agents The Belfort skill teaches a coding agent to use Belfort's encrypted Machine Learning platform: compiling a model, calibrating it, uploading it, minting client tokens, deploying, running the Belfort ML client, and what to do about the common errors. It is one file, `SKILL.md`, in the [Agent Skills](https://agentskills.io) format; the agent loads it when a task involves Belfort and fetches these docs for anything the skill does not cover. ## Claude Code To have it in every project: ```bash mkdir -p ~/.claude/skills/belfort curl -fsSL https://eml.belfortlabs.com/skills/belfort/SKILL.md -o ~/.claude/skills/belfort/SKILL.md ``` To share it with everyone working in one repository, put it there instead and commit it: ```bash mkdir -p .claude/skills/belfort curl -fsSL https://eml.belfortlabs.com/skills/belfort/SKILL.md -o .claude/skills/belfort/SKILL.md ``` New Claude Code sessions pick it up. Ask for something like "compile my model in `model.py` for Belfort and deploy it", and Claude uses the skill. ## The Claude apps Download [belfort.zip](https://eml.belfortlabs.com/skills/belfort.zip) and upload it where Claude's settings list your skills. ## Other agents Agents that support Agent Skills take the same `belfort` folder in their own skills directory; the zip above holds it. Agents without skills can read these docs directly: every page is also Markdown at its URL plus `.md`, [/llms.txt](https://eml.belfortlabs.com/llms.txt) lists the pages, and [/llms-full.txt](https://eml.belfortlabs.com/llms-full.txt) holds all of them in one file. ## Updating The skill follows these docs. Run the install command again to update it. --- # Troubleshooting ## Compiling **`CompileError` from `belfort_ml.compile`.** The model could not be traced or lowered, usually because of an operation the compiler does not support. The message names it and the line of your model that uses it. Replace the operation, or check [What compiles](/docs/concepts/compilation-pipeline.md#what-compiles). **`CompileError` from `provider.models.create`.** Belfort's compiler rejected the model. The gateway fails a compile after 60 minutes with `compile_timeout`. A compile whose weights total over 80 GiB, or that writes any other file over 1 GiB, fails with `artifacts_too_large`. The message starts with the model's id; quote it when you contact Belfort. **`TimeoutError` from `provider.models.create`.** The compile took longer than `timeout`. It may still finish, and `provider.models.list(status="compiling")` finds the model. To keep the model in hand for long compiles, create it with `wait=False` and call `model.wait_until_ready(timeout=...)`. **`ConflictError` with code `name_taken` from `provider.models.create`.** Your account already has a model with this name. Names are unique per account and case-sensitive. Delete the old model with `model.delete()`, or choose another name. To reuse the model when it was built from the same MLIR, call `provider.models.get(name=..., create_from=artifacts)` instead of `create`; it raises `name_taken` only when the MLIR differs. **The Belfort ML client fails to build, naming a C library function.** The model's name is also the name of a generated C++ function, so a name such as `exp` or `log` collides with the C library. Choose another name. ## Making your model more FHE friendly When a model does not compile, or compiles too slowly, these changes often help: - **Pack less data into each ciphertext.** A ciphertext holds at most 32,768 values. When a layer does not fit, reduce its parameters, for example increase the stride of a convolution, or downsample the input. The same model can fail at 64×64 and compile at 32×32. - **Replace an unsupported layer.** For example, replace `nn.MaxPool2d` with `nn.AvgPool2d`. [What compiles](/docs/concepts/compilation-pipeline.md#what-compiles) lists the layers that do not compile and what to use instead. - **Use fewer layers.** A shallower model compiles faster, produces smaller artifacts, and needs fewer bootstraps when it runs. ## Running the Belfort ML client **`belfort-ml-client infer` fails with `No cyclops build available`.** The Belfort ML client needs the Cyclops client library to build your model's code. The platform wheels of `belfort-ml-client` contain it. **`the downloaded artifacts hash to ...`.** The files the client downloaded do not match what the compiler produced. The client fails before it generates keys, so it can run again on the same token. **`could not reach the gateway that this token names (...)`.** The client reads the gateway's address from the token. Check that the address is the gateway you expect, and that this machine can reach it. A token minted on a local gateway names `http://localhost:8000/api/v1`; mint a new token on the gateway the end client can reach. **`the gateway answered 401 (invalid_credential: ...)`.** The client token has expired, or its key set was deleted. Mint a new one with `model.create_client`. **`the gateway answered 409 (keys_already_named: ...)`.** The token was already used for a run. A token serves one key pair, and the secret key ended with that run's process; mint a new token, or rerun with that run's secret seed (`BELFORT_CLIENT_SECRET_SEED`, or `Client(token, secret_seed=...)`). **`no deploy held this client's keys within ...s (last state 'no_deploy')`.** No ready deploy served the client's key set while the client waited. Start one with `model.deploy(key_set=minted, ...)`, with the token you handed over. A deploy started for another token serves only that token's key set. The client waits an hour by default (`--timeout`). **`the gateway answered 413 (key_too_large: ...)` or `(key_set_too_large: ...)`.** One evaluation key is over 1 GiB (1,073,741,824 bytes), or the keys total over 64 GiB. Contact Belfort; no current model produces keys that large. **`inference_outcome_unknown`.** The deploy stopped before the answer to an inference arrived, the gateway answered it with a 5xx, or it refused a retry after the first answer was lost, so the inference may have run. The client does not send it a second time on its own. Send it again if running it twice is acceptable. **`the gateway reports failed: ...`.** Loading the end client's keys onto the deploy failed; the message is the deploy's own. Quote it when you contact Belfort. **`the gateway reports failed: deployment ... failed to start (...)`.** The deploy you started for the end client's token failed, with the code in parentheses. With `wait=True`, the default, `model.deploy(...)` raised the full error on your side. Start a new deploy with the same token, then run the end client again. ## Loading keys yourself These apply when you load a key set onto a deploy with `deploy.load_keys`. **`ConflictError` with code `key_set_mismatch`.** A deploy serves only the key set it was started for. Stop it and start a deploy for the other key set. **`ConflictError` with code `keys_already_loaded`.** The deploy holds or loads its key set already. `deploy.refresh().loaded_keys` names the loaded keys. ## Deploying and inferring **`InsufficientCredits` from `model.deploy`.** The credits for the whole `max_lifetime` are reserved up front. Shorten it, or use **Request more credits** on the console's Credits page. `provider.credits.get()` shows the balance. **`DeployLimitReached` from `model.deploy`.** An account runs one deploy at a time, and another is starting or ready. Stop it with `deploy.stop()`, or find it with `provider.deploys.list()`. **Encrypted answers far from the plaintext ones.** An approximated activation received values outside its calibrated range. Calibrate with real samples that cover the inputs your end clients send. --- # Model providers and end clients Three parties take part in an encrypted inference. | Party | Holds | Runs | |---|---|---| | Model provider (you) | The model, a Belfort API key | `belfort-ml` | | End client | The input data, the secret key | `belfort-ml-client` | | Belfort | The compiled model, evaluation keys, ciphertexts | The gateway, the compiler, GPU deploys | The end client has no Belfort account and never receives an API key. It reaches the gateway with a client token you mint, and Belfort's storage on presigned URLs the gateway issues for that token, to download its client code and upload its evaluation keys. ## What travels between you and the end client One client token. It authenticates the end client to the Belfort gateway for one model. Belfort does not deliver it: you send it to the end client by whatever means you already use with them. `model.create_client(label="acme-clinic")` opens a key set for that end client on the model and returns a `ClientToken`; the label is your name for that end client. Send its `token`, which also names the gateway the end client calls. The token reaches the gateway's client routes and no others, and only for that one key set. With it, the Belfort ML client: 1. Fetches the compile's client code: temporary download links with the digest the gateway verified. It downloads the files, checks them against the digest, and builds them. 2. Generates a key pair, names its evaluation keys to the gateway, and uploads them to the temporary upload links it gets back. 3. Waits until the gateway has loaded those keys onto the deploy you started for its key set, then sends ciphertexts for inference. A token opens one key set, which one end client uses; a later run needs the same secret seed or a new token. Another end client needs a new token, and a new deploy: a deploy serves the one key set you start it for. No model weights reach the end client. Its token downloads only the generated client code, never the weights or the server code. `GET /models/{m_id}/artifacts` returns every file of the compile, the weights included, so do not forward its links to an end client. ## Who sees what The secret key, plaintext inputs and decrypted outputs stay on the end client's machine. Belfort holds the compiled model, the evaluation keys and the ciphertexts, none of which reveal the data. See [Keys and encryption](/docs/concepts/keys-and-encryption.md) for the key's lifetime. --- # Compilation pipeline Compilation happens in two stages. | Stage | Where | Input | Output | |---|---|---|---| | `belfort_ml.compile` | Your machine, no network | `nn.Module`, example input, calibration inputs | `_torch.mlir` | | Belfort compiler | Belfort, started by `provider.models.create` or `provider.models.get(name=, create_from=)` | The uploaded MLIR | GPU code for the deploy, C++ for the Belfort ML client, encryption parameters | The first stage traces your model with `torch.export`, lowers it to MLIR, and attaches two annotations: which inputs are encrypted, and the value range of every operation that must be approximated. The MLIR contains the weights, so it is the only file uploaded. The second stage chooses the CKKS parameters, inserts bootstrapping where the circuit runs out of levels, and generates the GPU and client code. ## What compiles CKKS computes additions and multiplications on encrypted real numbers. How a neural network compiles to a sequence of additions and multiplications depends on the operations it contains. | Kind | Examples | Cost | |---|---|---| | Linear | `nn.Linear`, `nn.Conv1d`/`Conv2d`, depthwise convolutions, `AvgPool`, `BatchNorm` (in eval mode), `flatten`, reshapes, add, sub | Exact | | Polynomial | `x * x`, products of tensors | Exact, but each multiplication uses up one level | | Approximated | `relu`, `gelu`, `silu`, `sigmoid`, `tanh`, `exp`, `log`, `sqrt`, `reciprocal`, `maximum`, `minimum`, `clamp`, and similar elementwise functions | Replaced by a polynomial that is accurate over a calibrated range; uses several levels | The following layers do not compile as they are currently not supported: - `nn.MaxPool1d` and `nn.MaxPool2d`. Use `AvgPool` instead. - Grouped convolutions (`groups > 1`), except a depthwise `nn.Conv1d` or `nn.Conv2d` with `groups == in_channels == out_channels`, which compiles. Use `groups=1`. - `softmax` and `log_softmax`. A classifier can return the logits and apply softmax after decryption. - `ConvTranspose` and `Upsample`. - Indexing with a tensor, such as `x[:, idx]` or `torch.gather`. Slices with fixed bounds, such as `x[:, :8]`, compile. - A sum or mean over several dimensions at once, such as `x.mean(dim=(2, 3))` for global pooling. Use `nn.AdaptiveAvgPool2d(1)`. A sum can also reduce one dimension at a time, `x.sum(3).sum(2)`. A check looks at operations on the encrypted input and on the values computed from it. An operation on the weights alone, such as a max pool of a parameter, runs before encryption and compiles. To write the MLIR anyway, for example to try a compiler that supports more operations, pass `allow_unsupported_ops=True` to `belfort_ml.compile`. It logs the same list as a warning. The Belfort compiler still rejects the model. Each multiplication consumes one level of the ciphertext. Once a ciphertext has used its levels, the compiler inserts a bootstrap, which refreshes it and is the most expensive GPU operation. A model shallow enough to avoid bootstrapping runs much faster: - Prefer polynomial activations (`x * x`) where your accuracy allows it. The `mlp` and `lola` examples use squares; a model that keeps ReLU pays for its approximation. - An approximated operation is accurate only inside its calibrated range; values outside it produce wrong results without an error. - Every approximated operation except ReLU, `reciprocal` and division becomes a polynomial of degree 5 over its calibrated range. The degree cannot be changed yet. A function that a degree-5 polynomial fits poorly over that range, such as a steep `sigmoid` or `tanh` over a wide range, loses accuracy; compare the encrypted results with the plaintext model. Tighter calibration ranges, from representative calibration inputs, improve the fit. - ReLU uses a composite approximation by default: accurate, but deep enough to force bootstrapping. `use_composite_relu=False` trades accuracy for a shallower model and smaller keys; see [ReLU approximation](/docs/sdk/provider.md#relu-approximation). A ciphertext holds at most 32,768 values: the largest ring Belfort uses has degree 2^16, and a CKKS ciphertext holds half that many. Every linear layer has to fit in one ciphertext: the longer side of an `nn.Linear` weight matrix, and for a convolution, channels times spatial size of both its input and its output. A convolution with many channels on a large input can exceed it. In this case, downsampling the input can sometimes help. Neural network that are too deep can be problematic. If you encounter issues, you can try to reduce the input size and/or remove layers. ## The example input fixes the shape The graph is specialized to the shape of the example input, including the batch dimension. The deployed model accepts exactly that shape. Use batch size 1; one inference carries one input. The whole input travels as one ciphertext, however many numbers it holds. ## Calibration `calibration_inputs` is a list of tensors with the example input's shape. The compile step runs each of them through the model and records the full range of values reaching every approximated operation. It then doubles each range about its centre, except for operations whose domain is not all real numbers (`sqrt`, `log`, `reciprocal`, `div` and similar), which keep the observed range. Only these ranges go into the MLIR. Use real samples from your training or validation data, up to 512 of them. A small set can miss the inputs your end clients send, and then the encrypted result is wrong. Random inputs give ranges that do not match what your end clients will send. With no calibration inputs the MLIR carries no ranges, and the compiler assumes a fixed range, [-1, 1] for most operations. ## Encrypted and plaintext inputs By default every argument of `forward` is encrypted. If your model also takes inputs the server may see, mark only the secret ones with `belfort_ml.Secret`: ```python title="model.py" from typing import Annotated import torch from belfort_ml import Secret class Model(torch.nn.Module): def forward(self, x: Annotated[torch.Tensor, Secret], scale: torch.Tensor): ... ``` The Belfort ML client currently sends one ciphertext per inference, so a model can have at most one encrypted argument. ## Changing a model A model compiles once, from the MLIR uploaded with it. Model names are unique in your account. `provider.models.get(name=, create_from=artifacts)` returns the model of that name when it was built from the same MLIR, and raises `ConflictError` `name_taken` when the MLIR changed. To change the model, delete the old one with `model.delete()` first, or upload the new one under another name. --- # Keys and encryption ## The keys The Belfort ML client generates an FHE key pair when it starts the client binary for a model. | Key | Who holds it | What it allows | |---|---|---| | Secret key | Only the Belfort ML client's process | Encrypting and decrypting | | Evaluation keys | Belfort, loaded onto a deploy | Rotations and multiplications on ciphertexts; cannot decrypt | The set of evaluation keys is large, from hundreds of megabytes to several gigabytes depending on the model, and is stored as one file per key: `.evk` files and one `.s2d` file. The compiler decides which keys a model needs. ## Key sets A key set is one end client's evaluation keys, registered against one model and one compile. `model.create_client` opens one and mints the token the end client uses to reach it: ```python minted = model.create_client(label="acme-clinic") ``` The key set starts in `awaiting_keys`. On that token, the Belfort ML client names the evaluation keys it generated, and the gateway checks the names and returns one presigned upload URL per key. The URLs need no Belfort credential, so the client uploads to them directly; they expire after an hour. Keys are named once per token. Until the upload is verified, naming the same keys with the same sizes again returns fresh links for them, so a lost answer does not end the run. The Belfort ML client does not ask for new links, so with it a key set that takes longer than an hour to upload fails, and the end client needs a new token and a fresh key pair. The client then has the gateway verify that every named key arrived, which moves the set to `ready`, and the gateway loads it onto the deploy you started for it with `model.deploy(key_set=minted, ...)`. A key set is the evaluation keys of one private key. A deploy serves the one key set you name when you start it, and no other. Today the secret key stays in the Belfort ML client process that generated it, so one end client uses a deploy. `label` is your reference for the end client. It is the only information about the end client that Belfort stores. ## Key sets belong to the model's compile Keys are generated under the encryption parameters of the model's compile, and the Belfort ML client fetches that compile's code. ## The secret key's lifetime The secret key lives in the memory of the client binary the Belfort ML client starts. It derives from a 32-byte secret seed (`client.secret_seed`, or `BELFORT_CLIENT_SECRET_SEED`). A later process on the same token that passes the seed back restores the same secret key and skips the key upload; without it, the next run needs a new token. Whoever holds the seed can decrypt the end client's data. The one exception is a [developer-mode](/docs/concepts/deploys-and-credits.md#developer-mode) deploy, for testing: the gateway chooses the seed and the GPU generates the evaluation keys, so Belfort can decrypt. Belfort ML clients never share a secret key. Each one runs on its own token, with its own key pair and key set, and the gateway treats it as a separate end client. A deploy holds one key set at a time, so clients of the same model use a deploy one after the other, or one deploy each when they run at the same time. ## Ciphertexts One inference sends one ciphertext holding the whole input, and returns one ciphertext holding the whole output. Its size depends on the encryption parameters the compiler chose. CKKS is approximate: decrypted results differ from the plaintext model's by a small error. Approximated activations add their own error on top, which grows quickly for inputs outside the calibrated range. ## Client tokens The end client authenticates to the gateway with a token you mint: ```python minted = model.create_client(label="acme-clinic", ttl_seconds=86400) minted.token # send to the end client; it names the gateway too minted.expires_at ``` A client token reaches the gateway's client routes for one key set: the client code, the key upload, the key set's status, inference on the deploy that holds the keys, and ending the client. It reaches none of your account's other routes. It expires after `ttl_seconds`: 86400 (24 hours) by default, 60 to 604800 (7 days). It has to cover the end client's whole run: the download, the build, the key upload, the wait for a deploy and the inferences. It ends earlier when the end client releases its key set, when you delete the key set with `model.delete_client(c_id)` (stop the deploys that serve it first), or when you delete the model. ### When to mint a new token A token belongs to one key set, not to a deploy. Every deploy you start with `key_set=minted` (or its `c_id`) serves that key set, so one token works across any number of deploys while it is valid and the end client still holds its secret key: a `Client` that keeps running, or a later process that passes the same secret seed. Mint a new token: - for each new end client, - when the token expires, - when the end client lost its secret seed, or ran `belfort-ml-client infer` without `BELFORT_CLIENT_SECRET_SEED`, which releases the key set at exit. --- # Deploys and credits ## Deploys A deploy is one GPU instance serving one compiled model. Each deploy starts a fresh instance, which takes a few minutes. ```python from datetime import timedelta minted = model.create_client(label="acme-clinic") # hand minted.token to the client with model.deploy(key_set=minted, max_lifetime=timedelta(hours=1)) as deploy: ... # keep the deploy up while the client runs ``` | Argument | Default | Range | |---|---|---| | `key_set` | required, unless `dev_mode=True` | The key set the deploy serves: a `ClientToken`, a registered `ClientKeySet`, or its `c_id` | | `dev_mode` | `False` | `True` starts a [developer-mode](#developer-mode) deploy; pass no `key_set` with it | | `max_lifetime` | required | 30 minutes to 5 hours | | `idle_timeout` | 900 s | 300 s to 3600 s; the gateway refuses a value outside that range | | `instance_type` | `rtx-pro-6000` | `rtx-pro-6000` only (1x NVIDIA RTX PRO 6000 Blackwell) | | `timeout` | 600 s | How long the SDK waits for the instance to become ready | A deploy stops at the first of: - leaving the `with` block, or `deploy.stop()` - `max_lifetime` after it started - `provider.deploys.stop_all()` The `with` form stops the instance even when your code raises. If the instance does not start within `timeout`, `model.deploy()` stops it and raises `belfort_ml.TimeoutError`. A Ctrl-C while the instance starts stops it too; if that stop fails, it prints the deploy id and a `python -c` command that stops it, since the instance keeps billing. A Python process that is killed does not stop its deploy; it runs until `max_lifetime` unless you call `provider.deploys.stop_all()`. A deploy serves one key set for its whole life: the one you name in `key_set`. A key set holds the evaluation keys of one private key, so the deploy holds one set of weights and one set of evaluation keys, and never takes another key set. The gateway loads the key set's keys once the deploy is ready and the end client has uploaded them. If the keys are in when the deploy turns ready, `model.deploy` starts that load itself and returns once the GPU holds them. A load that fails there only logs a warning, since the end client's next poll tries it again. The keys stay there until the Belfort ML client ends, you unload them, or the deploy stops. To keep a deploy running longer, `deploy.extend(timedelta(minutes=30))`. The total lifetime is capped at 5 hours from start. `deploy.terminates_at` says when a deploy will stop. `provider.deploys.list()` and the console show what is running. An account runs one deploy at a time. Since a deploy serves one key set, that is one end client at a time. While a deploy is starting or ready, `model.deploy()` raises `belfort_ml.DeployLimitReached` (a `ConflictError`, with `.limit` and `.running`); stop that deploy first. A deploy that is stopping does not count. ## Developer mode Developer mode skips the generation and upload of the end client's evaluation keys, which can take minutes for a large model. Use it to try a model, not to serve one. ```python with model.deploy(dev_mode=True, max_lifetime=timedelta(minutes=30)) as deploy: deploy.client_token # for the end client, as BELFORT_CLIENT_TOKEN deploy.secret_seed # 64 hex characters, as BELFORT_CLIENT_SECRET_SEED ``` The gateway opens a key set and chooses its secret seed, and the GPU generates the evaluation keys from it. Once the deploy is ready, it reports `secret_seed` and a `client_token` for that key set. An end client started with both resumes the key set, as in [The secret key's lifetime](/docs/concepts/keys-and-encryption.md#the-secret-keys-lifetime), and starts inferring without uploading keys. > **Belfort can decrypt** > > The gateway stores the secret seed, so Belfort can decrypt everything a > developer-mode deploy serves. Never send real data to one. To reuse the same seed on a later deploy, pass that deploy's `c_id` as `key_set`, without `dev_mode`; the deploy runs in developer mode because its key set does. The key set and its tokens expire 24 hours after the first deploy, and the gateway deletes the seed when the key set expires or the end client releases it. Passing both `key_set` and `dev_mode=True`, or neither, raises `ValueError`. ## Running continuously A deploy lives at most 5 hours. Serving an end client around the clock means starting a new deploy for its key set when the last one ends: ```python title="serve.py" import time from datetime import timedelta while True: with model.deploy(key_set=minted, max_lifetime=timedelta(hours=5)) as deploy: while deploy.refresh().status == "ready": time.sleep(60) ``` What happens to the end client across deploys: - Its key set outlives the deploy. When the end client next polls its status, the gateway loads the key set onto the next ready deploy that serves it. - An `infer()` that finds no deploy holding its keys waits for the next deploy and sends the same ciphertext again, if the first attempt cannot have run. Otherwise it reports the outcome as unknown. - The token must outlive the deploys: mint it with a `ttl_seconds` long enough. When it expires, the gateway deletes the key set's keys. - A long-running `Client` needs no new token until its token expires, after 7 days at most. `belfort-ml-client infer` needs a new token for every run unless every run sets the same `BELFORT_CLIENT_SECRET_SEED`. The end client cannot run inferences while the next deploy starts, which takes a few minutes, and while its keys load. An account runs one deploy at a time, so the next deploy starts only once the current one is stopping. A 5-hour deploy reserves 30 credits, three times the grant of a new account, so continuous serving needs more credits than the grant. ## Credits Credits pay for deploy time. Compiling, uploading keys and running inferences are free. | | | |---|---| | 1 credit | 10 minutes of deploy time, rounded up | | New account | 10 credits (100 minutes) | | More credits | **Request more credits** on the console's Credits page | When a deploy starts, `ceil(max_lifetime / 10 min)` credits are reserved, so the shortest deploy, 30 minutes, reserves 3; if your balance is too low, `model.deploy` raises `InsufficientCredits` (with `.required` and `.available`) and nothing starts. When it stops, you are charged for the time it ran from becoming ready, rounded up to 10 minutes, and the rest is released; a deploy that fails to start costs nothing. `extend` reserves the difference. Example: `max_lifetime=timedelta(hours=1)` reserves 6 credits; stopped after 25 minutes, it costs 3. `provider.credits.get()["credit_minutes"]` gives the length of a credit. ```python provider.credits.get() # {"granted": 10, "used": 3, "reserved": 6, "available": 1, "credit_minutes": 10} provider.credits.transactions() # one page, newest first ``` --- # `Provider` ## Configuration ```python import belfort_ml provider = belfort_ml.Provider() # from the environment provider = belfort_ml.Provider(api_key="blf_live_...") # or explicit ``` | Argument | Environment | Default | |---|---|---| | `api_key` | `BELFORT_API_KEY` | required | | `base_url` | `BELFORT_BASE_URL` | `https://eml.belfortlabs.com/api/v1` | | `timeouts` | | `Timeouts()`: 30 s standard, 300 s per upload part, 420 s inference, 10 s poll | | `max_retries` | | 3 | Explicit arguments take precedence over the environment. A missing key raises `ConfigurationError` when the provider is constructed. Set `BELFORT_BASE_URL` only to point at a different gateway, for example `http://localhost:8000/api/v1` for a local one. For the gateway serving these docs: ```bash export BELFORT_BASE_URL=https://eml.belfortlabs.com/api/v1 ``` The provider holds a connection pool. Use it as a context manager or call `close()`: ```python with belfort_ml.Provider() as provider: ... ``` ## Objects `provider.account.get()` and `provider.credits.get()` return plain dicts. `provider.models` and `provider.deploys` return objects that carry their own methods, so ids rarely need to be passed around. | Object | Returned by | Key attributes | |---|---|---| | `Model` | `provider.models.create/get/list` | `id`, `status`, `params`, `active_compile_id`, `level_zero_encryption`, `use_composite_relu` | | `ClientKeySet` | `model.clients` | `id`, `label`, `status`, `size_bytes` | | `Deploy` | `model.deploy`, `provider.deploys.get/list` | `id`, `c_id`, `status`, `terminates_at`, `credits_reserved`; `secret_seed` and `client_token` once a developer-mode deploy is ready | | `LoadedKeys` | `deploy.load_keys` | `k_id` | | `ClientToken` | `model.create_client` | `token`, `c_id`, `m_id`, `label`, `expires_at` | | `InferenceResult` | `deploy.infer` | `output`, `request_id`, `latency_ms` | ## Methods | Method | What it does | |---|---| | `belfort_ml.compile(model, example_input, name, output_dir, calibration_inputs=, allow_unsupported_ops=False)` | Local compile to MLIR; returns `Artifacts`. `name` must be a C++ identifier that is not a keyword. Raises `CompileError` on an operation the Belfort compiler does not support; `allow_unsupported_ops=True` writes the MLIR and logs a warning instead. Warns with `MultiCiphertextWarning` when a layer does not fit one ciphertext | | `provider.account.get()` | Your account, including `mp_id` and credits | | `provider.credits.get()` | Credit balance | | `provider.credits.transactions(cursor=, limit=, kind=)` | One page of credit transactions; follow `next_cursor` | | `provider.models.create(artifacts, name=, wait=True, timeout=3600, level_zero_encryption=False, use_composite_relu=True)` | Register, upload, compile, wait. See [Level-zero encryption](#level-zero-encryption) and [ReLU approximation](#relu-approximation) | | `provider.models.get(m_id)` / `provider.models.get(name=)` / `provider.models.list()` | Look up models; `list` follows pages lazily | | `provider.models.get(name=, create_from=artifacts, timeout=3600, level_zero_encryption=False, use_composite_relu=True)` | Return the model of that name, creating it from `artifacts` when missing and waiting until it is ready. A model of that name with a different MLIR, `level_zero_encryption` or `use_composite_relu` raises `ConflictError` `name_taken` | | `model.refresh()` / `model.wait_until_ready()` | Re-read the compile status | | `model.delete()` | Delete a model with no in-use deploy, with its uploaded MLIR, its key sets and its stopped deploys | | `model.create_client(label, ttl_seconds=)` | Open a key set for one end client and mint the token it authenticates to the gateway with, valid 24 hours by default | | `model.wait_for_keys(key_set, timeout=3600)` | Wait until the end client has uploaded its evaluation keys and the gateway verified them. Raises `KeyMaterialError` on a `failed` or `expired` key set | | `model.clients(status=)` | The model's key sets, with `id` (the `c_id`), `label`, `status` and `size_bytes` | | `model.delete_client(c_id)` | Delete a key set that no running deploy serves, which ends its client token | | `model.deploy(key_set=, max_lifetime=, timeout=, dev_mode=False)` | Start a GPU deploy that serves `key_set` (a `ClientToken`, a `ClientKeySet` or a `c_id`) for its whole life, and wait for it. `dev_mode=True`, without `key_set`, is for testing: see [Developer mode](/docs/concepts/deploys-and-credits.md#developer-mode) | | `provider.deploys.get(d_id)` / `.list()` / `.stop_all()` | Look up or stop deploys | | `deploy.load_keys()` / `deploy.unload_keys(loaded)` | Put the deploy's key set on the GPU, or remove it | | `deploy.infer(loaded, ciphertext)` / `deploy.infer_batch(loaded, [...])` | Run an inference on a ciphertext you hold | | `deploy.extend(additional)` / `deploy.stop()` / `deploy.refresh()` | Manage the deploy | `deploy.infer` is for ciphertexts that are already in your hands, for example test data an end client sent you. Normally the end client calls the gateway itself with the token from `model.create_client`, so the ciphertext does not pass through you, and the gateway loads its keys onto the deploy without a `load_keys` call. `load_keys` is for loading a key set onto a deploy yourself. Every method that waits (`create`, `get(name=, create_from=)`, `deploy`, `load_keys`) polls the resource's status, starting every 2 seconds and backing off to 10, and raises `TimeoutError` when its `timeout` runs out. `wait=False` returns immediately where it is offered. While a deploy starts, `deploy` writes the deploy id and each launch step (looking for a GPU instance, booting it, loading the model, then loading the client's keys if the end client has uploaded them) to stderr; `progress=False` turns that off. ## Level-zero encryption `level_zero_encryption=True` compiles the model so that the end client encrypts each input at the bottom of the modulus chain, and the deploy bootstraps each input before the computation. This is a tradeoff: it will make the ciphertext at the client much smaller, hence reducing upload time and bandwidth usage. The computation will however take longer as an additional bootstrap operation will be performed at the beginning. It is not recommended be use it with models that do not require bootstrapping as it will greatly increase the evaluation keys' size. ## ReLU approximation Encrypted computation cannot evaluate ReLU exactly, so the compiler replaces each ReLU with a polynomial over the range that calibration recorded. With `use_composite_relu=True`, the default, each ReLU is a composite approximation: three chained polynomials, of degrees 15, 15 and 27. It is accurate, but it uses many levels per activation, so the model bootstraps more often. More levels and bootstraps make each inference slower and the evaluation keys larger. `use_composite_relu=False` uses a single degree-5 polynomial instead. A small model can then fit without bootstrapping, with much smaller keys. The approximation is coarser, so compare the encrypted results with the plaintext model before you use it. The [Quickstart](/docs/getting-started/quickstart.md) uses it. These two are the only choices: the degree of the approximation, for ReLU and for the other approximated operations, cannot be set yet. ## Uploads `provider.models.create` uploads the MLIR, and the Python file that defines the model when `artifacts.source_path` is set, to storage through presigned URLs, in parts, without sending your API key. A failed part is retried; an upload that still fails raises `UploadError`. ## Logging The SDK logs to the standard `belfort_ml` logger, which is silent unless you configure a handler. DEBUG lines contain the method, path, status and request id, and omit headers and bodies. ```python import logging logging.basicConfig(level=logging.INFO) ``` --- # Errors and retries ## Errors Every exception the SDK raises descends from `belfort_ml.BelfortMLError`. ```mermaid flowchart LR B[BelfortMLError] --> Cfg["ConfigurationError
no API key, or a base URL that is not http(s)"] B --> Cmp["CompileError
local or Belfort-side compile failure"] B --> Up["UploadError
an upload part failed after retries"] B --> Dn["DownloadError
reading compiled artifacts failed"] B --> Km["KeyMaterialError
a key directory holds a file that is not an evaluation key"] B --> To["TimeoutError
a wait ran past its timeout"] B --> Cn["ConnectionError
the gateway was unreachable after retries"] B --> Api["APIError
the gateway returned an error"] Api --> V["ValidationError
400, 413"] Api --> Au["AuthenticationError
401"] Api --> Ic["InsufficientCredits
402, with .required and .available"] Api --> Pe["PermissionError
403"] Api --> Nf["NotFoundError
404"] Api --> Co["ConflictError
409"] Co --> Dl["DeployLimitReached
409 deploy_limit_reached, with .limit and .running"] Api --> Rl["RateLimitError
429, with .retry_after"] Api --> Su["ServiceUnavailable
503"] ``` `APIError` carries `message`, `code`, `detail`, `status_code` and `request_id`. Include the `request_id` when you report a problem to Belfort. ```python try: deploy = model.deploy(key_set=minted, max_lifetime=timedelta(hours=2)) except belfort_ml.InsufficientCredits as e: print(f"need {e.required} credits, have {e.available}") except belfort_ml.APIError as e: print(e.code, e.message, e.request_id) ``` Common `ConflictError` codes: | Code | Cause | |---|---| | `name_taken` | Creating a model with a name your account already uses, or `models.get(name=, create_from=)` with a different MLIR than that model's | | `deploy_running` | Deleting a model while one of its deploys is starting or ready | | `key_set_not_ready` | Loading a key set that has not been verified | | `key_set_unusable` | Starting a deploy for a key set whose keys failed verification or expired | | `key_loaded` | Deleting a key set that a running deploy serves | | `key_set_mismatch` | Loading another key set than the one the deploy serves | | `keys_already_loaded` | Loading the deploy's key set while it holds or loads it | | `deploy_not_ready` | Using a deploy that is still starting or already stopping | | `deploy_limit_reached` | Starting a deploy while the account's one deploy is starting or ready; raised as `DeployLimitReached` | ## Retries The SDK retries connection errors, timeouts, and 429, 500, 502 and 504 responses up to `max_retries` times (3 by default), with exponential backoff and jitter capped at 30 seconds. A 429's `Retry-After` takes precedence over the computed backoff. Other 4xx responses are not retried. A read or write timeout can come after the gateway acted on the request, so after one only GET requests are sent again; a `POST /deploys` that timed out raises `ConnectionError` instead of starting a second deploy. Inference is never retried automatically, since a retry would run on the GPU again. To retry one yourself, pass the same `request_id` to `deploy.infer` within five minutes: the gateway returns the stored result instead of running it again. Choose the `request_id` yourself on the first call, since a call that fails does not return the one `deploy.infer` generated. `model.create_client()` is not retried either: each call opens a key set, so a retry after a lost answer would leave a key set whose token nobody holds. If it fails, call it again; an unused key set expires with its token. ## Capacity A deploy's runtime handles one inference at a time; concurrent requests wait for it. A request waits up to 30 seconds for the runtime, then fails with 429 and `retry_after: 30`. Each inference gets 300 seconds of GPU time. --- # The Belfort ML client The Belfort ML client is the end client's side of Belfort and holds the secret key. `belfort-ml-client` ships it as the `belfort-ml-client` command and the `belfort_ml_client.Client` class. Your end clients run it on their own machine, on the client token you mint with `model.create_client`. It talks to the gateway itself and never receives an API key. ```mermaid sequenceDiagram participant C as belfort-ml-client participant G as Gateway participant S as Storage C->>G: GET /client/code G-->>C: download links, one per file, and their digest C->>S: download the client C++ Note over C: check the digest, build the client, generate a key pair C->>G: POST /client/keys G-->>C: upload links, one per evaluation key, each bound to its size C->>S: upload the evaluation keys C->>G: POST /client/keys/verify loop until a deploy holds the keys C->>G: GET /client/status end Note over C: encrypt the input C->>G: POST /client/infer G-->>C: encrypted answer Note over C: decrypt opt without BELFORT_CLIENT_SECRET_SEED C->>G: DELETE /client end ``` ## Running it ```bash belfort-ml-client infer --token "$BELFORT_CLIENT_TOKEN" --input input.json ``` | Flag | Environment | Default | What | |---|---|---|---| | `--token` | `BELFORT_CLIENT_TOKEN` | required | The client token from the provider | | `--cache-dir` | `BELFORT_CLIENT_CACHE` | `$XDG_CACHE_HOME/belfort-ml-client`, or `~/.cache/belfort-ml-client` | Where the linked client code is kept, per model and compile | | `--input` | | required | A JSON file with the input, or `-` for stdin | | `--timeout` | | 3600 s | How long to wait for a deploy to load the keys | | | `BELFORT_CLIENT_SECRET_SEED` | none | A 32-byte seed (64 hex characters) the keys derive from; a later run with the same seed resumes the key set. Without it, the run releases the key set at exit | The client downloads only its own code: the generated client code and the runtime headers it compiles. It never receives the model weights or the server code. The input file holds `{"values": [...]}`: the model input flattened in row-major order, with as many entries as the example input you compiled with. For a model with several arguments, it holds `{"inputs": [[...], [...]]}` with one list per argument; only one of them may be encrypted. The decrypted answer goes to stdout as JSON, flattened the same way: ```json title="Output" {"values": [...], "request_id": "req_...", "latency_ms": 847, "encrypt_ms": 12, "decrypt_ms": 3} ``` `latency_ms` is the time the gateway's call to the GPU runtime took: the ciphertext transfers between the gateway and the runtime, the wait in the runtime, and the GPU computation. It does not include the transfers between this client and the gateway, or the gateway's own work. A replayed request reports the time of its first run. `encrypt_ms` and `decrypt_ms` are the time this client took to encrypt the input and to decrypt the answer. Progress goes to stderr. On failure the command prints `belfort-ml-client: ` and exits with status 1. The machine needs a C++20 compiler to build the client code for each model. The build runs `clang++`; set `CXX` to use another compiler, such as `g++`. The platform wheels carry the Cyclops client library ([Installation](/docs/getting-started/installation.md)). No GPU is needed. ## What one run does 1. `GET /client/code` names the model's compile, one presigned download URL per file, and the digest the gateway verified. The client downloads the files, checks them against the digest, and links the client binary. The linked code is cached per model and compile, so the next run on the same compile skips the download and the build. 2. The client binary generates the key pair and keeps the secret key in its own memory. `POST /client/keys` names the evaluation keys and returns one presigned upload URL per key; `POST /client/keys/verify` checks they all arrived. The gateway takes at most 1000 keys, each at most 1 GiB (1,073,741,824 bytes) and 64 GiB in all. 3. `GET /client/status` reports `awaiting_keys`, `awaiting_upload`, `no_deploy`, `loading`, `ready` or `failed`. Once the keys are verified, the gateway loads them onto the deploy you started for the key set. `no_deploy` keeps the client waiting, since you may start the deploy after handing over the token; `failed` ends the run. It means every deploy of the key set failed to load the keys three times, the newest deploy of the key set failed to start, or verify found a key of the wrong size. 4. `POST /client/infer` sends one ciphertext and returns one. A request that could not connect is sent again. One whose answer was lost after sending raises `inference_outcome_unknown`, since the inference may have run. A token opens one key set, and its keys can be named once. The secret key ends with the process, so the next run needs a new token from you, unless both runs set the same `BELFORT_CLIENT_SECRET_SEED`. To run `belfort-ml-client infer` more than once on a token, generate a seed before the first run and set it on every run: ```bash export BELFORT_CLIENT_SECRET_SEED=$(openssl rand -hex 32) ``` The seed is the end client's secret key: anyone who holds it can decrypt the end client's data. Keep it where the end client keeps other secrets, never send it to the provider or to Belfort, and use a new one with each new token. ## From Python `belfort_ml_client.Client` does the same in steps, and runs as many inferences as needed on one key pair while its process lives: ```python title="end_client.py" from belfort_ml_client import Client with Client(token) as client: # or BELFORT_CLIENT_TOKEN client.prepare() # code, keys, and a deploy holding them answer = client.infer([values]) # one list per model argument ``` Leaving the `with` block, or `close()`, stops the key-pair process and keeps the key set. `Client(token, secret_seed=client.secret_seed)` in a later process restores the key pair and skips the key upload; the [Quickstart](/docs/getting-started/quickstart.md)'s `client.py` keeps the seed in a file per token for this. `release()` ends the client at the gateway: `DELETE /client` unloads its key set and deletes its keys, which ends the token. The deploy serves no other key set, so stop it. Otherwise the gateway deletes the key material once the token expires (see [Deploys and credits](/docs/concepts/deploys-and-credits.md)). If the deploy holding the keys stops while a `Client` runs, `infer()` waits for the gateway to load them onto the next ready deploy you start for the key set, for up to an hour, and then sends the same ciphertext again, at most three times per `infer()`. It does so only when the gateway answered that no deploy holds the keys before the request could run. If an earlier attempt lost its answer, the inference may have run on the deploy that stopped, and `infer()` raises `ClientError` with `code` `inference_outcome_unknown` instead of running it a second time. It raises the same after a 5xx answer to the inference, which can mean the GPU ran it and the gateway lost the result. | Method | What it does | |---|---| | `Client(token=, base_url=, secret_seed=, cache_dir=, http_transport=)` | `secret_seed` resumes this token's key set; reads `BELFORT_CLIENT_TOKEN` and `BELFORT_CLIENT_CACHE` for what is not passed, and calls the gateway the token names unless `base_url` is passed; `http_transport` replaces httpx's transport for the gateway calls | | `prepare(timeout=3600)` | `fetch_code()`, `publish_keys()`, `_warm_up()` then `wait_until_ready()` | | `fetch_code()` | `GET /client/code`, download, check the digest, link; or reuse the cached link | | `publish_keys()` | Generate the key pair, `POST /client/keys`, upload, `POST /client/keys/verify` | | `status()` | `GET /client/status` | | `wait_until_ready(timeout=3600)` | Poll `status()` until `ready`; raise `ClientError` on `failed` or the timeout | | `secret_seed` | The seed the secret key derives from, once the keys exist | | `release()` | `DELETE /client`: end the key set and the token | | `close()` | Stop the key-pair process; the key set stays | | `infer(arguments)` | Encrypt one input, `POST /client/infer`, decrypt; returns the dict the command prints | A refused or failed gateway call, a failed presigned transfer, a digest mismatch and a timeout raise `belfort_ml_client.ClientError`, whose message says what to do next. A gateway that stays unreachable after the retries raises `httpx.TransportError`, except during `infer()`, which raises `ClientError` with `code` `inference_outcome_unknown`. A failed link or client binary raises `RuntimeError` (or `OSError` when it cannot start). A `Client` is not safe to share between threads. Clients on the same compile share its cached code; one that has to link it again, after an upgrade of `belfort-ml-client` or the Cyclops library, waits until the others close. ## Retries The client makes up to 4 attempts at each gateway call and each presigned transfer. It retries a dropped connection and the statuses 429, 500, 502, 503 and 504, waiting 1, 2, then 4 seconds, or what `Retry-After` asks for, at most 30 seconds. A presigned transfer is also sent again on a 400 `RequestTimeout`, which the bucket answers when an upload stalls for about 20 seconds. It also waits out 409 `request_in_progress` and `verify_in_progress`, which mean an earlier attempt of the same call is still running: a verify for up to 10 minutes, an inference for up to 420 seconds. Releasing waits up to 10 minutes for a key load in flight. An inference is sent again only after a connection that failed before sending, or a 429; after a 5xx or a lost answer it raises `inference_outcome_unknown`, since the GPU may have run it. ## Timing Building the client code, generating keys and uploading the evaluation keys each take minutes for a large model, and the upload is bounded by the end client's bandwidth.