Skip to main content

Create an Index

Define the processing steps for your files to prepare them for search. You use Pipeline Builder to create your indexes.

About This Task​

You create indexes just like you create pipelines. Indexes are a series of connected components, each performing a preprocessing step on your files. To learn more, see Indexes.

Indexes are specific to a workspace. A single query pipeline can have multiple indexes.

You can use Pipeline Builder to design your indexes.

Using Pipeline Builder

About Builder​

Builder is an easy way to build and visualize your pipelines. In Builder, you simply drag components from the components library and drop them onto a canvas, where you can customize their parameters and define connections. It helps you visualize your pipeline and offers guidance on component compatibility. You can also switch to the YAML view anytime; everything you do in Builder is synchronized with the pipeline YAML configuration.

Using Builder​

This image shows how to access the basic functionalities in Builder. The numbers in the list below correspond to the numbers in the image.

The Builder canvas with component cards and component connections shown and the versions and editor switch buttons
  1. Component library. Expand a component group and drag a selected component onto the canvas to add it to your pipeline. You can also use the search bar to find components.
  2. A component card. Click the component name to change it.
  3. Component connections. Click an input connection on one component and an output connection on another to link them. Hover over a connection point next to an input or output to see a list of popular and compatible components you can connect.
    The Builder canvas with component cards and component connections shown
  4. Click a component card to access the menu for deleting, running, duplicating, and accessing the component's documentation.
    Tip: Use the Run action to run a component in isolation and check if it works correctly.
  5. Use this menu to:
    • View, restore, and deploy pipeline versions.
    • Run the pipeline without needing to deploy it.
    • Delete all components from the canvas.
    • Download pipeline YAML or Python configuration.
    • Switch between YAML and visual editors.

After you save an index, you must enable it to start indexing the files in your workspace. Files uploaded after an index was enabled, are automatically added to the enabled index. The query pipeline can access all files only after indexing is complete.

Running Individual Components​

You can run single components to understand their inputs and outputs and check if they work correctly before you save and enable the whole index. For information on how it works, see Run Components in Isolation.

Prerequisites​

Create an Index in Pipeline Builder​

  1. Log in to deepset AI Platform, go to Indexes and choose Create Index.

  2. Choose a template to start with or click Build your own to create an index from scratch.

    The Build your own index button on the Idexes page

  3. Give your index a name and add a meaningful description.

  4. Click Create index. You're redirected to Pipeline Builder.

  • If you're creating the index from a template, edit the template if needed and save your index.

  • If you're creating an index from an empty file:

    1. Open the Inputs group and drag the FilesInput component onto the canvas. This is always the first component of an index. It represents the files your index will process.
    2. The second component is often FileTypeRouter. It's useful if you're planning to index files of different types. You can set it to identify the file type and route it to an appropriate converter.
    3. Choose Converters for the file types you want to index.
    4. If you're using multiple Converters, add a DocumentJoiner to join the converted documents into a single list.
    5. Add Preprocessors as needed.
    6. Connect the components by clicking one component's input and then another component's output. The connection is automatically created and validated.
      Tip: Hover your mouse over the connection point to see compatible connections.
    7. Add DocumentWriter as the last component of your index. It writes the processed documents into the document store where a query pipeline can access it.
    8. Add a DocumentStore that will use this index and connect it to DocumentWriter.
      Tip: OpenSearchDocumentStore is the core document store.
    9. Save your index.

What To Do Next