This page as Markdown · All pages (llms.txt)

Troubleshooting

Compiling#

CompileError from belfort_ml.compile. The model could not be traced or lowered, usually because of an operation the compiler does not support. The message names it and the line of your model that uses it. Replace the operation, or check What compiles.

CompileError from provider.models.create. Belfort's compiler rejected the model. The gateway fails a compile after 60 minutes with compile_timeout. A compile whose weights total over 80 GiB, or that writes any other file over 1 GiB, fails with artifacts_too_large. The message starts with the model's id; quote it when you contact Belfort.

TimeoutError from provider.models.create. The compile took longer than timeout. It may still finish, and provider.models.list(status="compiling") finds the model. To keep the model in hand for long compiles, create it with wait=False and call model.wait_until_ready(timeout=...).

ConflictError with code name_taken from provider.models.create. Your account already has a model with this name. Names are unique per account and case-sensitive. Delete the old model with model.delete(), or choose another name. To reuse the model when it was built from the same MLIR, call provider.models.get(name=..., create_from=artifacts) instead of create; it raises name_taken only when the MLIR differs.

The Belfort ML client fails to build, naming a C library function. The model's name is also the name of a generated C++ function, so a name such as exp or log collides with the C library. Choose another name.

Making your model more FHE friendly#

When a model does not compile, or compiles too slowly, these changes often help:

  • Pack less data into each ciphertext. A ciphertext holds at most 32,768 values. When a layer does not fit, reduce its parameters, for example increase the stride of a convolution, or downsample the input. The same model can fail at 64×64 and compile at 32×32.
  • Replace an unsupported layer. For example, replace nn.MaxPool2d with nn.AvgPool2d. What compiles lists the layers that do not compile and what to use instead.
  • Use fewer layers. A shallower model compiles faster, produces smaller artifacts, and needs fewer bootstraps when it runs.

Running the Belfort ML client#

belfort-ml-client infer fails with No cyclops build available. The Belfort ML client needs the Cyclops client library to build your model's code. The platform wheels of belfort-ml-client contain it.

the downloaded artifacts hash to .... The files the client downloaded do not match what the compiler produced. The client fails before it generates keys, so it can run again on the same token.

could not reach the gateway <url> that this token names (...). The client reads the gateway's address from the token. Check that the address is the gateway you expect, and that this machine can reach it. A token minted on a local gateway names http://localhost:8000/api/v1; mint a new token on the gateway the end client can reach.

the gateway answered 401 (invalid_credential: ...). The client token has expired, or its key set was deleted. Mint a new one with model.create_client.

the gateway answered 409 (keys_already_named: ...). The token was already used for a run. A token serves one key pair, and the secret key ended with that run's process; mint a new token, or rerun with that run's secret seed (BELFORT_CLIENT_SECRET_SEED, or Client(token, secret_seed=...)).

no deploy held this client's keys within ...s (last state 'no_deploy'). No ready deploy served the client's key set while the client waited. Start one with model.deploy(key_set=minted, ...), with the token you handed over. A deploy started for another token serves only that token's key set. The client waits an hour by default (--timeout).

the gateway answered 413 (key_too_large: ...) or (key_set_too_large: ...). One evaluation key is over 1 GiB (1,073,741,824 bytes), or the keys total over 64 GiB. Contact Belfort; no current model produces keys that large.

inference_outcome_unknown. The deploy stopped before the answer to an inference arrived, the gateway answered it with a 5xx, or it refused a retry after the first answer was lost, so the inference may have run. The client does not send it a second time on its own. Send it again if running it twice is acceptable.

the gateway reports failed: .... Loading the end client's keys onto the deploy failed; the message is the deploy's own. Quote it when you contact Belfort.

the gateway reports failed: deployment ... failed to start (...). The deploy you started for the end client's token failed, with the code in parentheses. With wait=True, the default, model.deploy(...) raised the full error on your side. Start a new deploy with the same token, then run the end client again.

Loading keys yourself#

These apply when you load a key set onto a deploy with deploy.load_keys.

ConflictError with code key_set_mismatch. A deploy serves only the key set it was started for. Stop it and start a deploy for the other key set.

ConflictError with code keys_already_loaded. The deploy holds or loads its key set already. deploy.refresh().loaded_keys names the loaded keys.

Deploying and inferring#

InsufficientCredits from model.deploy. The credits for the whole max_lifetime are reserved up front. Shorten it, or use Request more credits on the console's Credits page. provider.credits.get() shows the balance.

DeployLimitReached from model.deploy. An account runs one deploy at a time, and another is starting or ready. Stop it with deploy.stop(), or find it with provider.deploys.list().

Encrypted answers far from the plaintext ones. An approximated activation received values outside its calibrated range. Calibrate with real samples that cover the inputs your end clients send.