Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Managing Compute SettingsBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

As an organization admin, use the Compute tab in Organization Settings to choose which instance sizes your organization can use, how many spare instances to keep ready for each size, and which size is the default.

About This Task​

A spare instance lets a new service start on that instance size right away. Turning a size on lets services use it. It also lets indexes use it as their pod size. If you keep spare instances at 0 for a size, the first service that needs that size starts one from scratch instead of using a spare.

Spare instances use credits for every hour they run, even before a service uses them. The Credits per month column on the Compute tab shows what your spare instance counts cost, so you can weigh faster service starts against credit use.

The same size name means different compute for services and for indexes, so each size on the Compute tab lists both. Before you move a service to a size, check its memory. Services deployed before instance sizes existed ran with 8 GiB memory, which matches Large. If you move one of these services to Medium, its memory drops to 4 GiB, so choose Large to keep 8 GiB.

You must be an organization admin to view and change these settings.

Access Compute Settings​

  1. Click your profile icon and choose Settings.
  2. Go to Organization.
  3. Click the Compute tab.

Update Instance Sizes and Spare Instances​

You can choose one of the following sizes. Each size lists the compute a service replica gets and the compute an index gets:

  • Small: Services 1 vCPU, 2 GiB memory. Indexes 0.4+ vCPU, 4 GiB memory.
  • Medium: Services 1 vCPU, 4 GiB memory. Indexes 0.9+ vCPU, 8 GiB memory.
  • Large: Services 2 vCPU, 8 GiB memory. Indexes 1.7+ vCPU, 16 GiB memory.
  • Extra large: Services 4 vCPU, 16 GiB memory. Indexes 3.5+ vCPU, 32 GiB memory.

For services, vCPU is a limit. For indexes, the + means the indexing pod can use more CPU when it's free, but memory is a hard limit.

  1. Enable the sizes you want to make available to your organization.
    Disabled Sizes in Use

    When you turn off a size, it becomes unavailable to services and indexes. However, if the size is in use by a service or index, it remains available to it.

  2. Choose the number of spare instances you want to keep ready for each size. Spare instances are always-on instances that are kept running and unassigned, so a new service on that size starts without waiting for a new instance to start. Setting the number to 0 keeps the size enabled but doesn't keep any instances ready in advance. Maximum is 10 spare instances per size. If you enter a higher number, we keep 10 instances ready. This cap protects your organization from starting a large number of instances by mistake. To see what your choices cost in credits, see Check the Credits Spare Instances Use.
  3. Choose the default size for new services. For details, see Set a Default Size.
  4. Click Save changes. The message "Your compute settings are saved." confirms the update. If the save fails, see Turn Off a Size That's in Use.

Check the Credits Spare Instances Use​

Before you save, use the Credits per month column to check what your spare instance counts cost.

  • Credits per month shows the credits that the spare instances of one size use in a month. The value is the number of spare instances multiplied by 720 hours, because a month counts as 30 days of always-on spares. Every size currently uses 1 credit per instance per hour.
  • Total is the sum of the Credits per month values for all enabled sizes.

Both values update as you edit the spare instance numbers, before you save. For example, two spare instances of one size show 1,440 credits per month. The values show credits only, not money amounts.

If the total is higher than you want, lower the spare instance count for a size or set it to 0. Setting a size to 0 trades credit use for slower first starts.

If loading your current settings fails, click Retry.

Turn Off a Size That's in Use​

Turning off an instance size removes its pool of instances. To protect running workloads, you can't turn off a size while any of the following still use it:

  • A deployed service on serverless hosting. Services that are undeployed or on managed hosting don't count.
  • A pipeline index with a pod size set, whether the index is deployed or not. An undeployed index still claims its size the next time it's deployed. Indexes without a pod size use the default pod size and never block a size.
  • A fleet environment set to that size, even if no service runs in it. Services in an environment run on the environment's size.

When you switch off a size that's in use, a warning lists each environment and service on that size, with the workspace of each service. You can't save until you switch the size back on or free it.

To free a size so you can turn it off:

  1. Move each service or environment listed in the warning to another size, or undeploy the services. To change a service's size, see Scale the Service and Control Cost. To change an environment's size, see Manage Fleet Environments. Undeploying an index doesn't free its size. Move each index to another size, or reset it to the default pod size. For steps, see Edit an Index.
  2. Reopen the Compute tab. The list of what uses each size refreshes only when you open the tab, so a stale warning clears after you reopen it.
  3. Turn the size off and click Save changes.

Saving fails if something still uses a size you turned off, for example a service deployed on it after you opened the tab. An error message names what still uses the size, and your settings stay unchanged. A size that was already off never blocks a save.

Set a Default Size​

The Default size is the size we preselect when you create a service in the UI, and the size we use for Studio runs. Until your organization sets a default, everything runs on Medium. Services created through the API without an instance size also use Medium, regardless of the default.

  1. On the Compute tab, make sure the size you want is turned on in the Enabled column. Only enabled sizes appear in the Default size list.
  2. In Default size, select the size you want. The list shows the compute of each size for a service replica, for example Medium (1 vCPU, 4 GiB memory).
  3. Click Save changes. The message "Your compute settings are saved." confirms the update.

Keep these rules in mind:

  • The Enabled switch for the current default size is locked. To turn that size off, first choose a different default size.
  • If you turn Medium off and haven't chosen a default, the Default size field asks you to choose a size, and you can't save until you do.
  • Clearing Default size returns the default to Medium.

Fix a No Free Instance Error​

When your organization has no free instance of an instance size, a request that needs that size fails with an HTTP 503 error instead of waiting for an instance to start. This happens when the platform has no free instance of that size.

In the UI, the error notification names the size that ran out and tells you to ask your admin to add capacity. If you're an organization admin, raise the spare instances for that size on the Compute tab.

In the REST API, the error response includes reason set to fleet_no_capacity and instance_size, the size that ran out. If the response has a Retry-After header, wait that many seconds before you retry. Pipeline runs that stream their results don't name the size in their error yet.

When a streaming run in the UI has to wait for an instance, it starts one and retries until the instance is ready. This can take a few minutes, so keep the run open instead of starting it again.

To fix the error:

  1. Retry the request after a short wait, or after the number of seconds in the Retry-After header. If the request succeeds, no further action is needed.
  2. If the error keeps coming back, ask an organization admin to raise the number of spare instances for the size named in the error. For steps, see Update Instance Sizes and Spare Instances.