Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Create an Index

Define the processing steps for your files to prepare them for search. You use Pipeline Builder to create your indexes.

About This Task​

You create indexes just like you create pipelines. Indexes are a series of connected components, each performing a preprocessing step on your files. To learn more, see Indexes.

Indexes are specific to a workspace. A single query pipeline can have multiple indexes.

You can use Pipeline Builder to design your indexes.

Using Pipeline Builder

About Builder​

Builder is an easy way to build and visualize your pipelines. In Builder, you simply drag components from the components library and drop them onto a canvas, where you can customize their parameters and define connections. It helps you visualize your pipeline and offers guidance on component compatibility. You can also switch to the YAML view anytime; everything you do in Builder is synchronized with the pipeline YAML configuration.

Using Builder​

This image shows how to access the basic functionalities in Builder. The numbers in the list below correspond to the numbers in the image.

The Builder canvas with component cards and component connections shown and the versions and editor switch buttons
  1. Switch between Builder, Playground (where you can test your pipeline), pipeline analytics (where you can see pipeline details, including search history and logs), and settings (where you can expose your pipeline as an MCP tool, enable GPU or scaling, and more).
  2. Switch between YAML editor and visual canvas.
  3. Manage pipeline versions and enable autosave.
  4. Run the pipeline right in the Builder without needing to deploy it.
  5. A component card. Click the component to open its configuration. Hover over the component icon to check its type and location.
  6. Component connections. Drag a line between components to connect them or use the Connections panel. You can also use the Command Palette to create connections. Press CMD + K to open it and type connect to see the available commands.
  7. Run and More Actions. Click the Run button to run the component in isolation and the More Actions button to access the component's menu.
  8. Use this menu to:
    • Zoom in and out of the canvas.
    • Group components. You can collapse and expand groups to reduce visual clutter. Groups are saved with your pipeline and restored when you reload the builder.
    • Return to the default positioning of components.
    • Expand or collapse components.
  9. Issues and Connections. Expand the view to see all validation and runtime issues in one place. Click Inspect next to an issue to jump directly to the affected component or connection and fix it. Expand the Connections view to manage, inspect, and edit component connections.
  10. Add components. Expand a component group and drag a selected component onto the canvas to add it to your pipeline. You can also use the search bar to find components. The search bar includes an AI-powered semantic search, so you can search for components by their name or functionality. By default, the component library hides legacy components. To show them, click More Actions next to the search field and choose Show deprecated components. You can also pin the component library to keep it always visible.

Running Components and Pipelines​

You can run single components to understand their inputs and outputs and check if they work correctly before you save and deploy the whole pipeline. Click Run on a component card.

You can also run the whole pipeline without needing to deploy it first. Just click the Run Pipeline button on the canvas.

After you save an index, you must enable it to start indexing the files in your workspace. Files uploaded after an index was enabled, are automatically added to the enabled index. The query pipeline can access all files only after indexing is complete.

Prerequisites​

Create an Index in Pipeline Builder​

  1. Log in to Haystack Enterprise Platform, go to Indexes and choose Create Index.

  2. Choose a template to start with or click Build your own to create an index from scratch.

    The Build your own index button on the Idexes page

  3. Give your index a name and add a meaningful description.

  4. Click Create index. You're redirected to Pipeline Builder.

  • If you're creating the index from a template, edit the template if needed and save your index.

  • If you're creating an index from an empty file:

    1. Open the Inputs group and drag the FilesInput component onto the canvas. This is always the first component of an index. It represents the files your index will process.
    2. The second component is often FileTypeRouter. It's useful if you're planning to index files of different types. You can set it to identify the file type and route it to an appropriate converter.
    3. Choose Converters for the file types you want to index.
    4. If you're using multiple Converters, add a DocumentJoiner to join the converted documents into a single list.
    5. Add Preprocessors as needed.
    6. Connect the components by clicking one component's input and then another component's output. The connection is automatically created and validated.
      Tip: Hover your mouse over the connection point to see compatible connections.
    7. Add DocumentWriter as the last component of your index. It writes the processed documents into the document store where a query pipeline can access it.
    8. Add a DocumentStore that will use this index and connect it to DocumentWriter.
      Tip: OpenSearchDocumentStore is the core document store.
    9. Save your index.

What To Do Next