Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Tutorial: Connecting OpenAI-Compatible Clients to a ServiceBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Use your service as a hosted model in the chat clients and frameworks your team already knows. You get the connection details from Gateway once and reuse them in Open WebUI, Mozilla Thunderbolt, and the Vercel AI SDK.

  • Level: Basic
  • Time to complete: 20 minutes
  • Prerequisites:
    • A service with a published pipeline version that is active. The pipeline must return a generated answer or a chat response. For details, see Create a Service and Publish a Pipeline to a Service.
    • A workspace API key. For instructions, see Generate API Keys.
    • Access to at least one of the clients in this tutorial. You can follow only the sections for the clients you use.
  • Goal: After completing this tutorial, you will have connected your service to Open WebUI, Thunderbolt, and an app built with the Vercel AI SDK, and sent it a message from each.

What You're Building​

Your service offers the Completions API, which follows the OpenAI Chat Completions format. Any client that works with OpenAI-compatible providers can treat your service as a model. Your pipeline runs behind it, so users get answers from your own data and logic in a tool they already use.

Every client needs the same three values:

  • A base URL that points to the deployments in your workspace
  • A model ID that identifies your service
  • A workspace API key

You collect these values once, test them in Haystack Enterprise Platform, and then add them to each client.

Get the Connection Details​

You find all connection details on the Connect tab of your service.

  1. In Haystack Enterprise Platform, click Gateway and open your service.

  2. Click Connect.

  3. Under API format, select Completions API.

    If you only see Platform API, your pipeline doesn't return a generated answer or a chat response. Connect shows the Completions API tab only for these pipelines.

  4. In the Endpoint card, copy the URL. It ends with /chat/completions, for example: https://api.cloud.deepset.ai/api/v1/workspaces/my_workspace/deployments/v1/chat/completions.

  5. Remove the chat completions path from the end of the URL to get the base URL, for example: https://api.cloud.deepset.ai/api/v1/workspaces/my_workspace/deployments/v1. Clients append their own paths for chat completions and model listing, such as /chat/completions and /models, or /v1/chat/completions and /v1/models.

  6. In the Request example, find the model value and copy it. This is your model ID. It has the form {workspace_name}/{deployment_id}.

  7. In the Authentication section, click Manage API keys and create or copy a workspace API key. Keep the key secret and don't paste it into shared documents or client-side code.

Result: You have a base URL, a model ID, and an API key.

Test the Service​

Before you set up a client, send a test message from the Connect tab. This way, you know the service works and any later problem is in the client setup.

  1. On the Connect tab, go to the Try it (optional) card. The request runs against the live service and counts toward your quota.
  2. Click Send request.
  3. Check the Response panel. A chat.completion object with an assistant message means the service answered.

If the request fails, the card shows a message that explains why. If the service is waking up or restarting, wait a moment and send the request again. For a list of error codes and what they mean, see Chat Completions.

Result: Your service answers a Completions API request.

Connect Open WebUI​

Open WebUI is a self-hosted chat interface for OpenAI-compatible APIs. You add your service as a connection, and it appears in the model list.

  1. In Open WebUI, go to Settings > Admin > Connections.

  2. In Manage OpenAI API Connections, click the plus icon to add a connection.

  3. In the URL field, type your base URL.

  4. In the API Key field, type your workspace API key.

  5. In the Model IDs field, type your model ID and click the plus icon next to it.

    If you leave this field empty, Open WebUI loads the list of models from your workspace and shows all your services.

  6. Click Verify Connection.

  7. Click Save.

  8. Start a new chat, select your service from the model list, and send a message.

If the settings look different in your version, see the Open WebUI guide for OpenAI-compatible providers.

Result: You can chat with your service in Open WebUI.

Connect Thunderbolt​

Mozilla Thunderbolt is an open source, cross-platform AI client. It works with any OpenAI-compatible endpoint as a custom provider.

  1. In Thunderbolt, go to Settings > Models.
  2. Add a model and select Custom as the provider.
  3. In the Endpoint URL field, replace the pre-filled address with your base URL.
  4. In the API key field, type your workspace API key.
  5. Type your model ID as the model identifier. A custom provider doesn't load a list of models, so you type the ID yourself.
  6. Click Test Connection. Thunderbolt saves a model only after a successful connection test.
  7. Save the model.
  8. Start a chat, select your service as the model, and send a message.

For more about adding models, see the Thunderbolt documentation.

Result: You can chat with your service in Thunderbolt.

Connect an App Built with the Vercel AI SDK​

The Vercel AI SDK is a TypeScript library for building AI apps. Its OpenAI-compatible provider lets your app call your service the same way it calls any other model.

  1. In your app, add the @ai-sdk/openai-compatible provider package.
  2. Store your workspace API key in an environment variable, for example, HAYSTACK_API_KEY.
  3. Create the provider with createOpenAICompatible() and set:
    • name: Any name for the provider, for example, deepset-service.
    • baseURL: Your base URL.
    • apiKey: The environment variable that holds your API key.
  4. Pass your model ID to the provider to get a chat model, and use it with generateText() or streamText().
  5. Run your app and send a question. The answer comes from your service.

For setup details and code examples, see the Vercel AI SDK documentation for OpenAI-compatible providers.

Using Services in Clients
  • Each client sends the full conversation with every request, so your service sees the chat history without any extra setup.
  • If your pipeline uses an Agent with tools, the Agent runs its tools inside the service and the client gets the final answer. Clients can't send their own tools to the service.
  • To let clients set sampling parameters, such as temperature, map generation_kwargs in your pipeline YAML. For details, see Passing Extra OpenAI Parameters Through Chat Completions.

Result: Congratulations! You have connected your service to Open WebUI, Thunderbolt, and your own app. Your team can now use your pipeline from the tools they already work with.

What's Next​