About This Task

You create indexes just like you create pipelines. Indexes are a series of connected components, each performing a preprocessing step on your files. To learn more, see Indexes.

Indexes are specific to a workspace. A single query pipeline can have multiple indexes.

You can use Pipeline Builder to design your indexes.

Using Pipeline Builder

About Pipeline Builder

Pipeline Builder is an easy way to build and visualize your pipelines. In Pipeline Builder, you simply drag components from the components library and drop them onto a canvas, where you can customize their parameters and define connections. It helps you visualize your pipeline and offers guidance on component compatibility. You can also switch to the YAML view anytime; everything you do in Pipeline Builder is synchronized with the pipeline YAML configuration.

Using Pipeline Builder

This image shows how to access the basic functionalities in Pipeline Builder. The numbers in the list below correspond to the numbers in the image.

1. Component library. Expand a component group and drag a selected component onto the canvas to add it to your pipeline. You can also use the search bar to find components.

2. A component card. Click the component name to change it.

3. Component connections. Click an input connection on one component and an output connection on another to link them. Hover over a connection point next to an input or output to see a list of popular and compatible components you can connect.

   

4. Click a component card to access the menu for deleting, running, duplicating, and accessing the component's documentation.
   Tip: Use the Run action to run a component in isolation and check if it works correctly.

5. Use this menu to:

   • View, restore, and deploy pipeline versions.
   • Run the pipeline without needing to deploy it.
   • Delete all components from the canvas.
   • Download pipeline YAML or Python configuration.
   • Switch between YAML and visual editors.

After you save an index, you must enable it to start indexing the files in your workspace. Files uploaded after an index was enabled, will be automatically added to the enabled index. The query pipeline can access all files only after indexing is complete.

Running Individual Components

You can run single components to verify their input and output and check if they work correctly before saving and enabling the whole index. For information on how it works, see Run Components in Isolation.

Prerequisites

• Understanding indexes. To learn more, see Indexes.
• Understanding pipelines and components. For details, see Pipelines and Pipeline Components.

Create an Index in Pipeline Builder

1. Log in to deepset AI Platform, go to Indexes and choose Create Index.

2. Choose a template to start with or click Build your own to create an index from scratch.

   

3. Give your index a name and add a meaningful description.

4. Click Create index. You're redirected to Pipeline Builder.

• If you're creating the index from a template, edit the template if needed and save your index.
• If you're creating an index from scratch:

1. Open the Inputs group and drag the FilesInput component onto the canvas. This is always the first component of an index. It represents the files your index will process.
2. The second component is often FileTypeRouter. It's useful if you're planning to index files of different types. You can set it to identify the file type and route it to an appropriate converter.
3. Choose Converters for the file types you want to index.
4. If you're using multiple Converters, add a DocumentJoiner to join the converted documents into a single list.
5. Add Preprocessors as needed.
6. Connect the components by clicking one component's input and then another component's output. The connection is automatically created and validated.
   Tip: Hover your mouse over the connection point to see compatible connections.
7. Add DocumentWriter as the last component of your index. It writes the processed documents into the document store where a query pipeline can access it.
8. Add a DocumentStore that will use this index and connect it to DocumentWriter.
   Tip: OpenSearchDocumentStore is the core document store.
9. Save your index.

What To Do Next

• Enable the index to start indexing files. Click Enable on the index page.
• Add the index to your query pipelines. For details, see Edit a Pipeline or Create a Pipeline in Pipeline Builder.
• Click the index name to open the Index Details page where you can see the index information, logs, and processed files.