--- description: Common errors encountered while compiling and running an encrypted model --- # Troubleshooting ## Compiling **`CompileError` from `belfort_ml.compile`.** The model could not be traced or lowered, usually because of an operation the compiler does not support. The message names it and the line of your model that uses it. Replace the operation, or check [What compiles](../concepts/compilation-pipeline.md#what-compiles). **`CompileError` from `provider.models.create`.** Belfort's compiler rejected the model. The gateway fails a compile after 60 minutes with `compile_timeout`. A compile whose weights total over 80 GiB, or that writes any other file over 1 GiB, fails with `artifacts_too_large`. The message starts with the model's id; quote it when you contact Belfort. **`TimeoutError` from `provider.models.create`.** The compile took longer than `timeout`. It may still finish, and `provider.models.list(status="compiling")` finds the model. To keep the model in hand for long compiles, create it with `wait=False` and call `model.wait_until_ready(timeout=...)`. **`ConflictError` with code `name_taken` from `provider.models.create`.** Your account already has a model with this name. Names are unique per account and case-sensitive. Delete the old model with `model.delete()`, or choose another name. To reuse the model when it was built from the same MLIR, call `provider.models.get(name=..., create_from=artifacts)` instead of `create`; it raises `name_taken` only when the MLIR differs. **The Belfort ML client fails to build, naming a C library function.** The model's name is also the name of a generated C++ function, so a name such as `exp` or `log` collides with the C library. Choose another name. ## Making your model more FHE friendly When a model does not compile, or compiles too slowly, these changes often help: - **Pack less data into each ciphertext.** A ciphertext holds at most 32,768 values. When a layer does not fit, reduce its parameters, for example increase the stride of a convolution, or downsample the input. The same model can fail at 64×64 and compile at 32×32. - **Replace an unsupported layer.** For example, replace `nn.MaxPool2d` with `nn.AvgPool2d`. [What compiles](../concepts/compilation-pipeline.md#what-compiles) lists the layers that do not compile and what to use instead. - **Use fewer layers.** A shallower model compiles faster, produces smaller artifacts, and needs fewer bootstraps when it runs. ## Running the Belfort ML client **`belfort-ml-client infer` fails with `No cyclops build available`.** The Belfort ML client needs the Cyclops client library to build your model's code. The platform wheels of `belfort-ml-client` contain it. **`the downloaded artifacts hash to ...`.** The files the client downloaded do not match what the compiler produced. The client fails before it generates keys, so it can run again on the same token. **`could not reach the gateway that this token names (...)`.** The client reads the gateway's address from the token. Check that the address is the gateway you expect, and that this machine can reach it. A token minted on a local gateway names `http://localhost:8000/api/v1`; mint a new token on the gateway the end client can reach. **`the gateway answered 401 (invalid_credential: ...)`.** The client token has expired, or its key set was deleted. Mint a new one with `model.create_client`. **`the gateway answered 409 (keys_already_named: ...)`.** The token was already used for a run. A token serves one key pair, and the secret key ended with that run's process; mint a new token, or rerun with that run's secret seed (`BELFORT_CLIENT_SECRET_SEED`, or `Client(token, secret_seed=...)`). **`no deploy held this client's keys within ...s (last state 'no_deploy')`.** No ready deploy served the client's key set while the client waited. Start one with `model.deploy(key_set=minted, ...)`, with the token you handed over. A deploy started for another token serves only that token's key set. The client waits an hour by default (`--timeout`). **`the gateway answered 413 (key_too_large: ...)` or `(key_set_too_large: ...)`.** One evaluation key is over 1 GiB (1,073,741,824 bytes), or the keys total over 64 GiB. Contact Belfort; no current model produces keys that large. **`inference_outcome_unknown`.** The deploy stopped before the answer to an inference arrived, the gateway answered it with a 5xx, or it refused a retry after the first answer was lost, so the inference may have run. The client does not send it a second time on its own. Send it again if running it twice is acceptable. **`the gateway reports failed: ...`.** Loading the end client's keys onto the deploy failed; the message is the deploy's own. Quote it when you contact Belfort. **`the gateway reports failed: deployment ... failed to start (...)`.** The deploy you started for the end client's token failed, with the code in parentheses. With `wait=True`, the default, `model.deploy(...)` raised the full error on your side. Start a new deploy with the same token, then run the end client again. ## Loading keys yourself These apply when you load a key set onto a deploy with `deploy.load_keys`. **`ConflictError` with code `key_set_mismatch`.** A deploy serves only the key set it was started for. Stop it and start a deploy for the other key set. **`ConflictError` with code `keys_already_loaded`.** The deploy holds or loads its key set already. `deploy.refresh().loaded_keys` names the loaded keys. ## Deploying and inferring **`InsufficientCredits` from `model.deploy`.** The credits for the whole `max_lifetime` are reserved up front. Shorten it, or use **Request more credits** on the console's Credits page. `provider.credits.get()` shows the balance. **`DeployLimitReached` from `model.deploy`.** An account runs one deploy at a time, and another is starting or ready. Stop it with `deploy.stop()`, or find it with `provider.deploys.list()`. **Encrypted answers far from the plaintext ones.** An approximated activation received values outside its calibrated range. Calibrate with real samples that cover the inputs your end clients send.