--- description: Running the Belfort ML client from belfort-ml-client on a client token. --- # The Belfort ML client The Belfort ML client is the end client's side of Belfort and holds the secret key. `belfort-ml-client` ships it as the `belfort-ml-client` command and the `belfort_ml_client.Client` class. Your end clients run it on their own machine, on the client token you mint with `model.create_client`. It talks to the gateway itself and never receives an API key. ```mermaid sequenceDiagram participant C as belfort-ml-client participant G as Gateway participant S as Storage C->>G: GET /client/code G-->>C: download links, one per file, and their digest C->>S: download the client C++ Note over C: check the digest, build the client, generate a key pair C->>G: POST /client/keys G-->>C: upload links, one per evaluation key, each bound to its size C->>S: upload the evaluation keys C->>G: POST /client/keys/verify loop until a deploy holds the keys C->>G: GET /client/status end Note over C: encrypt the input C->>G: POST /client/infer G-->>C: encrypted answer Note over C: decrypt opt without BELFORT_CLIENT_SECRET_SEED C->>G: DELETE /client end ``` ## Running it ```bash belfort-ml-client infer --token "$BELFORT_CLIENT_TOKEN" --input input.json ``` | Flag | Environment | Default | What | |---|---|---|---| | `--token` | `BELFORT_CLIENT_TOKEN` | required | The client token from the provider | | `--cache-dir` | `BELFORT_CLIENT_CACHE` | `$XDG_CACHE_HOME/belfort-ml-client`, or `~/.cache/belfort-ml-client` | Where the linked client code is kept, per model and compile | | `--input` | | required | A JSON file with the input, or `-` for stdin | | `--timeout` | | 3600 s | How long to wait for a deploy to load the keys | | | `BELFORT_CLIENT_SECRET_SEED` | none | A 32-byte seed (64 hex characters) the keys derive from; a later run with the same seed resumes the key set. Without it, the run releases the key set at exit | The client downloads only its own code: the generated client code and the runtime headers it compiles. It never receives the model weights or the server code. The input file holds `{"values": [...]}`: the model input flattened in row-major order, with as many entries as the example input you compiled with. For a model with several arguments, it holds `{"inputs": [[...], [...]]}` with one list per argument; only one of them may be encrypted. The decrypted answer goes to stdout as JSON, flattened the same way: ```json title="Output" {"values": [...], "request_id": "req_...", "latency_ms": 847, "encrypt_ms": 12, "decrypt_ms": 3} ``` `latency_ms` is the time the gateway's call to the GPU runtime took: the ciphertext transfers between the gateway and the runtime, the wait in the runtime, and the GPU computation. It does not include the transfers between this client and the gateway, or the gateway's own work. A replayed request reports the time of its first run. `encrypt_ms` and `decrypt_ms` are the time this client took to encrypt the input and to decrypt the answer. Progress goes to stderr. On failure the command prints `belfort-ml-client: ` and exits with status 1. The machine needs a C++20 compiler to build the client code for each model. The build runs `clang++`; set `CXX` to use another compiler, such as `g++`. The platform wheels carry the Cyclops client library ([Installation](../getting-started/installation.md)). No GPU is needed. ## What one run does 1. `GET /client/code` names the model's compile, one presigned download URL per file, and the digest the gateway verified. The client downloads the files, checks them against the digest, and links the client binary. The linked code is cached per model and compile, so the next run on the same compile skips the download and the build. 2. The client binary generates the key pair and keeps the secret key in its own memory. `POST /client/keys` names the evaluation keys and returns one presigned upload URL per key; `POST /client/keys/verify` checks they all arrived. The gateway takes at most 1000 keys, each at most 1 GiB (1,073,741,824 bytes) and 64 GiB in all. 3. `GET /client/status` reports `awaiting_keys`, `awaiting_upload`, `no_deploy`, `loading`, `ready` or `failed`. Once the keys are verified, the gateway loads them onto the deploy you started for the key set. `no_deploy` keeps the client waiting, since you may start the deploy after handing over the token; `failed` ends the run. It means every deploy of the key set failed to load the keys three times, the newest deploy of the key set failed to start, or verify found a key of the wrong size. 4. `POST /client/infer` sends one ciphertext and returns one. A request that could not connect is sent again. One whose answer was lost after sending raises `inference_outcome_unknown`, since the inference may have run. A token opens one key set, and its keys can be named once. The secret key ends with the process, so the next run needs a new token from you, unless both runs set the same `BELFORT_CLIENT_SECRET_SEED`. To run `belfort-ml-client infer` more than once on a token, generate a seed before the first run and set it on every run: ```bash export BELFORT_CLIENT_SECRET_SEED=$(openssl rand -hex 32) ``` The seed is the end client's secret key: anyone who holds it can decrypt the end client's data. Keep it where the end client keeps other secrets, never send it to the provider or to Belfort, and use a new one with each new token. ## From Python `belfort_ml_client.Client` does the same in steps, and runs as many inferences as needed on one key pair while its process lives: ```python title="end_client.py" from belfort_ml_client import Client with Client(token) as client: # or BELFORT_CLIENT_TOKEN client.prepare() # code, keys, and a deploy holding them answer = client.infer([values]) # one list per model argument ``` Leaving the `with` block, or `close()`, stops the key-pair process and keeps the key set. `Client(token, secret_seed=client.secret_seed)` in a later process restores the key pair and skips the key upload; the [Quickstart](../getting-started/quickstart.md)'s `client.py` keeps the seed in a file per token for this. `release()` ends the client at the gateway: `DELETE /client` unloads its key set and deletes its keys, which ends the token. The deploy serves no other key set, so stop it. Otherwise the gateway deletes the key material once the token expires (see [Deploys and credits](../concepts/deploys-and-credits.md)). If the deploy holding the keys stops while a `Client` runs, `infer()` waits for the gateway to load them onto the next ready deploy you start for the key set, for up to an hour, and then sends the same ciphertext again, at most three times per `infer()`. It does so only when the gateway answered that no deploy holds the keys before the request could run. If an earlier attempt lost its answer, the inference may have run on the deploy that stopped, and `infer()` raises `ClientError` with `code` `inference_outcome_unknown` instead of running it a second time. It raises the same after a 5xx answer to the inference, which can mean the GPU ran it and the gateway lost the result. | Method | What it does | |---|---| | `Client(token=, base_url=, secret_seed=, cache_dir=, http_transport=)` | `secret_seed` resumes this token's key set; reads `BELFORT_CLIENT_TOKEN` and `BELFORT_CLIENT_CACHE` for what is not passed, and calls the gateway the token names unless `base_url` is passed; `http_transport` replaces httpx's transport for the gateway calls | | `prepare(timeout=3600)` | `fetch_code()`, `publish_keys()`, `_warm_up()` then `wait_until_ready()` | | `fetch_code()` | `GET /client/code`, download, check the digest, link; or reuse the cached link | | `publish_keys()` | Generate the key pair, `POST /client/keys`, upload, `POST /client/keys/verify` | | `status()` | `GET /client/status` | | `wait_until_ready(timeout=3600)` | Poll `status()` until `ready`; raise `ClientError` on `failed` or the timeout | | `secret_seed` | The seed the secret key derives from, once the keys exist | | `release()` | `DELETE /client`: end the key set and the token | | `close()` | Stop the key-pair process; the key set stays | | `infer(arguments)` | Encrypt one input, `POST /client/infer`, decrypt; returns the dict the command prints | A refused or failed gateway call, a failed presigned transfer, a digest mismatch and a timeout raise `belfort_ml_client.ClientError`, whose message says what to do next. A gateway that stays unreachable after the retries raises `httpx.TransportError`, except during `infer()`, which raises `ClientError` with `code` `inference_outcome_unknown`. A failed link or client binary raises `RuntimeError` (or `OSError` when it cannot start). A `Client` is not safe to share between threads. Clients on the same compile share its cached code; one that has to link it again, after an upgrade of `belfort-ml-client` or the Cyclops library, waits until the others close. ## Retries The client makes up to 4 attempts at each gateway call and each presigned transfer. It retries a dropped connection and the statuses 429, 500, 502, 503 and 504, waiting 1, 2, then 4 seconds, or what `Retry-After` asks for, at most 30 seconds. A presigned transfer is also sent again on a 400 `RequestTimeout`, which the bucket answers when an upload stalls for about 20 seconds. It also waits out 409 `request_in_progress` and `verify_in_progress`, which mean an earlier attempt of the same call is still running: a verify for up to 10 minutes, an inference for up to 420 seconds. Releasing waits up to 10 minutes for a key load in flight. An inference is sent again only after a connection that failed before sending, or a 429; after a 5xx or a lost answer it raises `inference_outcome_unknown`, since the GPU may have run it. ## Timing Building the client code, generating keys and uploading the evaluation keys each take minutes for a large model, and the upload is bounded by the end client's bandwidth.