Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Managing Compute SettingsBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

As an organization admin, use the Compute tab in Organization Settings to choose which instance sizes your organization can use, how many warm instances to keep ready for each size, and which size is the default.


About This Task​

A warm instance is always on, so a new service on that instance size starts instantly when traffic spikes. Making a size available lets services use it. It also lets indexes use it as their pod size. If you keep warm instances at 0 for a size, the first service that needs that size starts one from scratch instead of using a warm instance.

Warm instances use credits for every hour they run, even when idle. To weigh faster service starts against credit use, see Check the Credits Warm Instances Use.

The same size name means different compute for services and for indexes, so each size on the Compute tab lists both. Before you move a service to a size, check its memory. Services deployed before instance sizes existed ran with 8 GiB memory, which matches Large. If you move one of these services to Medium, its memory drops to 4 GiB, so choose Large to keep 8 GiB.

You must be an organization admin to view and change these settings.

Access Compute Settings​

  1. Click your profile icon and choose Settings.
  2. Go to Organization.
  3. Click the Compute tab. If you don't see it, you need the organization admin role.

Update Instance Sizes and Warm Instances​

You can choose one of the following sizes. Each size shows its vCPU and memory under its name. Hover over the spec to see the compute a service replica gets and the compute an index gets:

  • Small: Services 1 vCPU, 2 GiB memory. Indexes 0.4+ vCPU, 4 GiB memory.
  • Medium: Services 1 vCPU, 4 GiB memory. Indexes 0.9+ vCPU, 8 GiB memory.
  • Large: Services 2 vCPU, 8 GiB memory. Indexes 1.7+ vCPU, 16 GiB memory.
  • Extra large: Services 4 vCPU, 16 GiB memory. Indexes 3.5+ vCPU, 32 GiB memory.

For services, vCPU is a limit. For indexes, the + means the indexing pod can use more CPU when it's free, but memory is a hard limit.

  1. In the Available column, turn on the sizes you want to make available to your organization.
    Sizes in Use

    When you turn off a size, it becomes unavailable to services and indexes. However, if the size is in use by a service or index, it remains available to it.

  2. In the Warm instances column, use the − and + buttons to set how many warm instances to keep ready for each size. Setting the number to 0 keeps the size available but doesn't keep any instances ready in advance. Maximum is 10 warm instances per size. If you enter a higher number, we keep 10 instances ready. This cap protects your organization from starting a large number of instances by mistake. To see what your choices cost in credits, see Check the Credits Warm Instances Use.
  3. In the Default column, select the default size for new services. For details, see Set a Default Size.
  4. Click Save changes. The message "Your compute settings are saved." confirms the update. If the save fails, see Turn Off a Size That's in Use.

Check the Credits Warm Instances Use​

Before you save, check what your warm instance counts cost.

  • Credits / mo shows the credits that the warm instances of one size use in a month. The value is the number of warm instances multiplied by 720 hours, because a month counts as 30 days of always-on instances. Every size currently uses 1 credit per instance per hour. A size that's off shows a dash.
  • The summary bar next to Save changes shows the monthly total in credits per month, which is the sum of the Credits / mo values for all available sizes. It also estimates the credits the warm instances use by the end of the billing cycle, if your plan has a billing cycle end date. These credits are on top of your plan's credits. To see your credit use, click Usage.

The values update as you change the warm instance numbers, before you save. For example, two warm instances of one size show 1,440 credits per month. The values show credits only, not money amounts.

If the total is higher than you want, lower the warm instance count for a size or set it to 0. Setting a size to 0 trades credit use for slower first starts.

If loading your current settings fails, click Retry.

Turn Off a Size That's in Use​

Turning off an instance size removes its pool of instances. To protect running workloads, you can't turn off a size while any of the following still use it:

  • A deployed service on serverless hosting. Services that are undeployed or on managed hosting don't count.
  • A pipeline index with a pod size set, whether the index is deployed or not. An undeployed index still claims its size the next time it's deployed. Indexes without a pod size use the default pod size and never block a size.
  • A fleet environment set to that size, even if no service runs in it. Services in an environment run on the environment's size.

When you switch off a size that's in use, a warning lists each environment and service on that size, with the workspace of each service. You can't save until you switch the size back on or free it.

To free a size so you can turn it off:

  1. Move each service or environment listed in the warning to another size, or undeploy the services. To change a service's size, see Scale the Service and Control Cost. To change an environment's size, see Manage Fleet Environments. Undeploying an index doesn't free its size. Move each index to another size, or reset it to the default pod size. For steps, see Edit an Index.
  2. Reopen the Compute tab. The list of what uses each size refreshes only when you open the tab, so a stale warning clears after you reopen it.
  3. Turn the size off and click Save changes.

Saving fails if something still uses a size you turned off, for example a service deployed on it after you opened the tab. An error message names what still uses the size, and your settings stay unchanged. A size that was already off never blocks a save.

Set a Default Size​

The default size is the size that new services, Studio runs, and MCP servers use. Existing services keep their current size. Until your organization sets a default, everything runs on Medium. Services created through the API without an instance size also use Medium, regardless of the default. Indexes use the available sizes too: an index's Instance size list shows only the sizes you turned on, plus the size the index already uses. An index without a size uses Medium. For details, see Configure Index Settings.

  1. On the Compute tab, make sure the size you want is turned on in the Available column. Only available sizes can be the default.
  2. In the Default column, select the radio button in the row of the size you want.
  3. Click Save changes.

Keep these rules in mind:

  • The Available switch for the current default size is locked. To turn that size off, first select another size as the default.
  • If you turn Medium off and haven't chosen a default, you must select a default size before you can save.

Fix a No Free Instance Error​

When your organization has no free instance of an instance size, a request that needs that size fails with an HTTP 503 error instead of waiting for an instance to start.

In the UI, the error notification names the size that ran out and tells you to ask your admin to add capacity. If you're an organization admin, raise the warm instances for that size on the Compute tab.

In the REST API, the error response includes reason set to fleet_no_capacity and instance_size, the size that ran out. If the response has a Retry-After header, wait that many seconds before you retry. Pipeline runs that stream their results don't name the size in their error yet.

When a streaming run in the UI has to wait for an instance, it starts one and retries until the instance is ready. This can take a few minutes, so keep the run open instead of starting it again.

To fix the error:

  1. Retry the request after a short wait, or after the number of seconds in the Retry-After header. If the request succeeds, no further action is needed.
  2. If the error keeps coming back, ask an organization admin to raise the number of warm instances for the size named in the error. For steps, see Update Instance Sizes and Warm Instances.