Startup Behavior

C3 may run jobs on capacity that is already available, or it may provision new capacity when existing machines are busy or a requested hardware profile is not immediately available.

Cold Provisioning

Traditional cloud compute workflows require spinning up a fresh VM for every job:

1. Request VM from cloud provider
2. Wait for allocation (1-10 min)
3. Boot the VM (1-5 min)
4. Initialize drivers and the C3 agent (1-10 min)
5. Download your code (seconds to minutes)
6. Finally run your job

This can add 5-45 minutes before your code starts, depending on provider availability and hardware type.

Available Capacity

C3 can assign jobs to already-available machines. These machines are:

• Already booted and initialized
• Drivers and runtime configured for their hardware profile
• C3 agent running and polling for work
• Network configured with fast access to our control plane

When you submit a job, the scheduler first looks for compatible available capacity, then provisions new capacity if needed:

┌─────────────────────────────────────────────────────────────────────────────┐
│                           C3 CAPACITY ARCHITECTURE                          │
└─────────────────────────────────────────────────────────────────────────────┘

                                    ┌─────────────┐
                                    │   Your Job  │
                                    │  c3 deploy  │
                                    └──────┬──────┘
                                           │
                                           ▼
┌──────────┐                      ┌─────────────────┐
│          │    Job submitted     │                 │
│   You    │ ──────────────────▶  │  C3 Control     │
│          │   allocation < 1s    │  Plane          │
└──────────┘                      └────────┬────────┘
                                           │
                                           ▼
                              ┌──────────────────────┐
                              │ HARDWARE PROVIDER(S) │
                              │  ┌────┐ ┌────┐       │
                              │  │VM  │ │VM  │  ...  │
                              │  │ ✓  │ │ ✓  │       │
                              │  └────┘ └────┘       │
                              │  Available Capacity  │
                              └──────────┬───────────┘
                                         │
                                         ▼
                              ┌───────────────────┐
                              │  Job runs on      │
                              │  first available  │
                              │  hardware         │
                              └───────────────────┘

Available vs New Capacity

Metric	Available capacity	New capacity
Allocation time	seconds	5-45 minutes
Total startup	short startup after assignment	5-45 minutes
When used	Compatible machine already available	No compatible machine is available

Assignment Flow

When compatible capacity is available, assignment is fast:

1. You submit → Job hits the C3 control plane
2. Assignment → Scheduler finds compatible idle hardware and assigns your job
3. Agent pickup (~5s) → Agent picks up the job on its next heartbeat
4. Code download → Bundle is fetched and extracted
5. Your code runs → Execution begins, logs stream back immediately

Capacity Scaling

C3 capacity scales with demand. The control plane tracks pending jobs and available provider inventory:

• Low demand: Existing capacity is reused when available
• High demand: Capacity scales up to meet job volume, subject to provider availability and spend controls
• Burst load: Overflow goes to cold provisioning when the provider can supply the requested hardware

When New Provisioning Happens

Sometimes jobs need new capacity:

• Current capacity busy: Compatible machines are already running other jobs
• Rare hardware type: Specialized hardware is not immediately available
• Unusual demand patterns: Spikes exceed current available capacity

Cold provisioning normally takes longer than assignment to an existing machine, but a catalogued hardware profile does not guarantee that its provider has live capacity. C3 automatically falls back to cold provisioning when needed. CPU profiles currently use experimental cold provisioning from a C3-managed CPU image rather than a persistent warm pool; the experimental tag does not change billing or the current high/medium/low availability tier.

When Provider Capacity Is Unavailable

C3 distinguishes a normal queue from a provider-capacity failure:

• A disabled capacity pool fails immediately with CAPACITY_POOL_DISABLED.
• Provider-confirmed out-of-stock capacity follows the request's .c3 capacity.on_unavailable policy. The default wait action marks an accepted job PROVISIONING_BLOCKED; fail rejects the submission or terminates an existing job immediately with PROVIDER_CAPACITY_UNAVAILABLE.
• An ambiguous create result, unknown capacity signal, or repeated create failures marks the job PROVISIONING_BLOCKED regardless of the authoritative-signal action.
• Each blocked job uses its own capacity.max_wait_minutes deadline (default 60 minutes, configurable from 1–360). C3 continues checking the same selected pool; if usable capacity does not register before that deadline, the job fails with PROVIDER_CAPACITY_UNAVAILABLE.

The configured limit applies only after C3 detects a blocked provisioning episode. It is not a general maximum for jobs waiting behind healthy but busy capacity, and it does not enable failover.

When both provider and hardware are explicitly pinned, wait can admit an otherwise eligible route even if the stock oracle already reports it out of stock. The job begins in PENDING/PROVISIONING_BLOCKED and retries that same selected pool. This admission exception is stock-only: it does not bypass a disabled/manual-target-zero pool, disabled offering, missing provider configuration, stale price, or spend guard. Provider-only, hardware-only, and fully automatic requests retain their normal routing/admission behavior.

Separately, every job has an unconditional six-hour queue ceiling. A job that remains PENDING beyond that ceiling fails with QUEUE_DEADLINE_EXCEEDED, even if C3 never detected a blocked provisioning episode. This outcome is terminal and non-retryable within the job. Because no compute ran, C3 refunds the full reserve. Submit again later or choose a different hardware profile or provider.

All three terminal capacity codes are non-retryable within the same job. If no compute ran, C3 refunds the job's reserved credits. Submit again later or choose a different hardware profile or provider.

Best Practices

Keep Jobs Easy to Place

1. Use common hardware profiles — Common profiles are easier to place quickly.
2. Keep jobs small — Finish faster and free capacity for other work
3. Use --follow — See real-time logs as your job runs

Understand the Timing

• Allocation time: How long from submission until hardware is assigned
• Total startup: Time from submission until your code starts running
• Runtime: Your actual code execution
• Total time: Everything from submit to completion

The Development Experience

The normal workflow is:

┌─────────────────────────────────────────────────────────────────┐
│                    ITERATIVE COMPUTE DEVELOPMENT                │
└─────────────────────────────────────────────────────────────────┘

    ┌──────────┐      ┌──────────┐      ┌──────────┐
    │  Edit    │      │  Submit  │      │   See    │
    │  Code    │ ───▶ │   Job    │ ───▶ │ Results  │
    │ Locally  │      │ c3 deploy│      │  quickly │
    └──────────┘      └──────────┘      └──────────┘
          ▲                                   │
          │                                   │
          └───────────────────────────────────┘
                    Iterate

This is useful for:

• ML experimentation — Try different hyperparameters quickly
• Debugging — Add print statements, see output fast
• Prototyping — Test ideas without provisioning overhead
• Education — Learn GPU and scientific-computing workflows interactively