The Belfort ML client

The Belfort ML client is the end client's side of Belfort and holds the secret key. belfort-ml-client ships it as the belfort-ml-client command and the belfort_ml_client.Client class. Your end clients run it on their own machine, on the client token you mint with model.create_client. It talks to the gateway itself and never receives an API key.

sequenceDiagram
    participant C as belfort-ml-client
    participant G as Gateway
    participant S as Storage
    C->>G: GET /client/code
    G-->>C: download links, one per file, and their digest
    C->>S: download the client C++
    Note over C: check the digest, build the client, generate a key pair
    C->>G: POST /client/keys
    G-->>C: upload links, one per evaluation key, each bound to its size
    C->>S: upload the evaluation keys
    C->>G: POST /client/keys/verify
    loop until a deploy holds the keys
        C->>G: GET /client/status
    end
    Note over C: encrypt the input
    C->>G: POST /client/infer
    G-->>C: encrypted answer
    Note over C: decrypt
    opt without BELFORT_CLIENT_SECRET_SEED
        C->>G: DELETE /client
    end

Running it#

belfort-ml-client infer --token "$BELFORT_CLIENT_TOKEN" --input input.json

The client downloads only its own code: the generated client code and the runtime headers it compiles. It never receives the model weights or the server code.

The input file holds {"values": [...]}: the model input flattened in row-major order, with as many entries as the example input you compiled with. For a model with several arguments, it holds {"inputs": [[...], [...]]} with one list per argument; only one of them may be encrypted. The decrypted answer goes to stdout as JSON, flattened the same way:

{"values": [...], "request_id": "req_...", "latency_ms": 847, "encrypt_ms": 12, "decrypt_ms": 3}

latency_ms is the time the gateway's call to the GPU runtime took: the ciphertext transfers between the gateway and the runtime, the wait in the runtime, and the GPU computation. It does not include the transfers between this client and the gateway, or the gateway's own work. A replayed request reports the time of its first run. encrypt_ms and decrypt_ms are the time this client took to encrypt the input and to decrypt the answer.

Progress goes to stderr. On failure the command prints belfort-ml-client: <message> and exits with status 1.

The machine needs a C++20 compiler to build the client code for each model. The build runs clang++; set CXX to use another compiler, such as g++. The platform wheels carry the Cyclops client library (Installation). No GPU is needed.

What one run does#

1. GET /client/code names the model's compile, one presigned download URL per file, and the digest the gateway verified. The client downloads the files, checks them against the digest, and links the client binary. The linked code is cached per model and compile, so the next run on the same compile skips the download and the build.
2. The client binary generates the key pair and keeps the secret key in its own memory. POST /client/keys names the evaluation keys and returns one presigned upload URL per key; POST /client/keys/verify checks they all arrived. The gateway takes at most 1000 keys, each at most 1 GiB (1,073,741,824 bytes) and 64 GiB in all.
3. GET /client/status reports awaiting_keys, awaiting_upload, no_deploy, loading, ready or failed. Once the keys are verified, the gateway loads them onto the deploy you started for the key set. no_deploy keeps the client waiting, since you may start the deploy after handing over the token; failed ends the run. It means every deploy of the key set failed to load the keys three times, the newest deploy of the key set failed to start, or verify found a key of the wrong size.
4. POST /client/infer sends one ciphertext and returns one. A request that could not connect is sent again. One whose answer was lost after sending raises inference_outcome_unknown, since the inference may have run.

A token opens one key set, and its keys can be named once. The secret key ends with the process, so the next run needs a new token from you, unless both runs set the same BELFORT_CLIENT_SECRET_SEED.

To run belfort-ml-client infer more than once on a token, generate a seed before the first run and set it on every run:

export BELFORT_CLIENT_SECRET_SEED=$(openssl rand -hex 32)

The seed is the end client's secret key: anyone who holds it can decrypt the end client's data. Keep it where the end client keeps other secrets, never send it to the provider or to Belfort, and use a new one with each new token.

From Python#

belfort_ml_client.Client does the same in steps, and runs as many inferences as needed on one key pair while its process lives:

from belfort_ml_client import Client

with Client(token) as client:          # or BELFORT_CLIENT_TOKEN
    client.prepare()                   # code, keys, and a deploy holding them
    answer = client.infer([values])    # one list per model argument

Leaving the with block, or close(), stops the key-pair process and keeps the key set. Client(token, secret_seed=client.secret_seed) in a later process restores the key pair and skips the key upload; the Quickstart's client.py keeps the seed in a file per token for this. release() ends the client at the gateway: DELETE /client unloads its key set and deletes its keys, which ends the token. The deploy serves no other key set, so stop it. Otherwise the gateway deletes the key material once the token expires (see Deploys and credits).

If the deploy holding the keys stops while a Client runs, infer() waits for the gateway to load them onto the next ready deploy you start for the key set, for up to an hour, and then sends the same ciphertext again, at most three times per infer(). It does so only when the gateway answered that no deploy holds the keys before the request could run. If an earlier attempt lost its answer, the inference may have run on the deploy that stopped, and infer() raises ClientError with code inference_outcome_unknown instead of running it a second time. It raises the same after a 5xx answer to the inference, which can mean the GPU ran it and the gateway lost the result.

A refused or failed gateway call, a failed presigned transfer, a digest mismatch and a timeout raise belfort_ml_client.ClientError, whose message says what to do next. A gateway that stays unreachable after the retries raises httpx.TransportError, except during infer(), which raises ClientError with code inference_outcome_unknown. A failed link or client binary raises RuntimeError (or OSError when it cannot start). A Client is not safe to share between threads. Clients on the same compile share its cached code; one that has to link it again, after an upgrade of belfort-ml-client or the Cyclops library, waits until the others close.

Retries#

The client makes up to 4 attempts at each gateway call and each presigned transfer. It retries a dropped connection and the statuses 429, 500, 502, 503 and 504, waiting 1, 2, then 4 seconds, or what Retry-After asks for, at most 30 seconds. A presigned transfer is also sent again on a 400 RequestTimeout, which the bucket answers when an upload stalls for about 20 seconds. It also waits out 409 request_in_progress and verify_in_progress, which mean an earlier attempt of the same call is still running: a verify for up to 10 minutes, an inference for up to 420 seconds. Releasing waits up to 10 minutes for a key load in flight. An inference is sent again only after a connection that failed before sending, or a 429; after a 5xx or a lost answer it raises inference_outcome_unknown, since the GPU may have run it.

Timing#

Building the client code, generating keys and uploading the evaluation keys each take minutes for a large model, and the upload is bounded by the end client's bandwidth.