Deploys and credits

Deploys#

A deploy is one GPU instance serving one compiled model. Each deploy starts a fresh instance, which takes a few minutes.

from datetime import timedelta

minted = model.create_client(label="acme-clinic")
# hand minted.token to the client

with model.deploy(key_set=minted, max_lifetime=timedelta(hours=1)) as deploy:
    ...  # keep the deploy up while the client runs

A deploy stops at the first of:

• leaving the with block, or deploy.stop()
• max_lifetime after it started
• provider.deploys.stop_all()

The with form stops the instance even when your code raises. If the instance does not start within timeout, model.deploy() stops it and raises belfort_ml.TimeoutError. A Ctrl-C while the instance starts stops it too; if that stop fails, it prints the deploy id and a python -c command that stops it, since the instance keeps billing. A Python process that is killed does not stop its deploy; it runs until max_lifetime unless you call provider.deploys.stop_all().

A deploy serves one key set for its whole life: the one you name in key_set. A key set holds the evaluation keys of one private key, so the deploy holds one set of weights and one set of evaluation keys, and never takes another key set. The gateway loads the key set's keys once the deploy is ready and the end client has uploaded them. If the keys are in when the deploy turns ready, model.deploy starts that load itself and returns once the GPU holds them. A load that fails there only logs a warning, since the end client's next poll tries it again. The keys stay there until the Belfort ML client ends, you unload them, or the deploy stops.

To keep a deploy running longer, deploy.extend(timedelta(minutes=30)). The total lifetime is capped at 5 hours from start.

deploy.terminates_at says when a deploy will stop. provider.deploys.list() and the console show what is running.

An account runs one deploy at a time. Since a deploy serves one key set, that is one end client at a time. While a deploy is starting or ready, model.deploy() raises belfort_ml.DeployLimitReached (a ConflictError, with .limit and .running); stop that deploy first. A deploy that is stopping does not count.

Developer mode#

Developer mode skips the generation and upload of the end client's evaluation keys, which can take minutes for a large model. Use it to try a model, not to serve one.

with model.deploy(dev_mode=True, max_lifetime=timedelta(minutes=30)) as deploy:
    deploy.client_token  # for the end client, as BELFORT_CLIENT_TOKEN
    deploy.secret_seed   # 64 hex characters, as BELFORT_CLIENT_SECRET_SEED

The gateway opens a key set and chooses its secret seed, and the GPU generates the evaluation keys from it. Once the deploy is ready, it reports secret_seed and a client_token for that key set. An end client started with both resumes the key set, as in The secret key's lifetime, and starts inferring without uploading keys.

Belfort can decrypt

The gateway stores the secret seed, so Belfort can decrypt everything a developer-mode deploy serves. Never send real data to one.

To reuse the same seed on a later deploy, pass that deploy's c_id as key_set, without dev_mode; the deploy runs in developer mode because its key set does. The key set and its tokens expire 24 hours after the first deploy, and the gateway deletes the seed when the key set expires or the end client releases it. Passing both key_set and dev_mode=True, or neither, raises ValueError.

Running continuously#

A deploy lives at most 5 hours. Serving an end client around the clock means starting a new deploy for its key set when the last one ends:

import time
from datetime import timedelta

while True:
    with model.deploy(key_set=minted, max_lifetime=timedelta(hours=5)) as deploy:
        while deploy.refresh().status == "ready":
            time.sleep(60)

What happens to the end client across deploys:

• Its key set outlives the deploy. When the end client next polls its status, the gateway loads the key set onto the next ready deploy that serves it.
• An infer() that finds no deploy holding its keys waits for the next deploy and sends the same ciphertext again, if the first attempt cannot have run. Otherwise it reports the outcome as unknown.
• The token must outlive the deploys: mint it with a ttl_seconds long enough. When it expires, the gateway deletes the key set's keys.
• A long-running Client needs no new token until its token expires, after 7 days at most. belfort-ml-client infer needs a new token for every run unless every run sets the same BELFORT_CLIENT_SECRET_SEED.

The end client cannot run inferences while the next deploy starts, which takes a few minutes, and while its keys load. An account runs one deploy at a time, so the next deploy starts only once the current one is stopping. A 5-hour deploy reserves 30 credits, three times the grant of a new account, so continuous serving needs more credits than the grant.

Credits#

Credits pay for deploy time. Compiling, uploading keys and running inferences are free.

When a deploy starts, ceil(max_lifetime / 10 min) credits are reserved, so the shortest deploy, 30 minutes, reserves 3; if your balance is too low, model.deploy raises InsufficientCredits (with .required and .available) and nothing starts. When it stops, you are charged for the time it ran from becoming ready, rounded up to 10 minutes, and the rest is released; a deploy that fails to start costs nothing. extend reserves the difference. Example: max_lifetime=timedelta(hours=1) reserves 6 credits; stopped after 25 minutes, it costs 3. provider.credits.get()["credit_minutes"] gives the length of a credit.

provider.credits.get()           # {"granted": 10, "used": 3, "reserved": 6, "available": 1, "credit_minutes": 10}
provider.credits.transactions()  # one page, newest first