About This Task

What Are Custom Components

A component is a Python code snippet that follows our template and performs a specific task on your data. When you create and upload a custom component to deepset AI Platform, it becomes accessible to your entire organization. Any member can then use the component in their pipelines.

Custom components are based on Haystack components. Haystack is deepset's open source AI framework, which also powers deepset AI Platform. To learn more, visit the Haystack website.

Custom Components Template

You can find custom component templates and examples in our Custom Components Library.

Currently, you can't delete custom components.

Custom Components in Pipeline Builder

Custom components uploaded to deepset AI Platform are shown in the Custom tab in Pipeline Builder. They're grouped automatically based on their path in the custom components package.

For example, if your component is located at /dc-custom-component-template/src/dc_custom_component/components/generators in the package, it will show up under the Generators group within the Custom tab.

Prerequisites

• Read the following resources before completing this task:
  • Haystack Components
  • Custom Components
  • Pipelines
• You should have a basic understanding of working with GitHub repositories.
• Generate a deepset API key to upload the created component to your workspace. For instructions, see Generate an API Key.

Create a Component

Prepare the Template

1. Fork the dc-custom-component-template GitHub repository. This will let you version control your changes.
2. Navigate to the directory where you cloned the repository and open the ./dc-custom-component-template/src/dc_custom_component/example_components/ directory. The preprocessors and rankers folders are examples you can modify or remove as needed.
3. Create a new folder or rename one of the example folders to match your custom component's name and open it. There's a .py file inside. This is where you'll write your component code. You can rename this file as well.
   Example:
   To create a custom component called WelcomeTextGenerator:
   1. Rename ./dc-custom-component-template/src/dc_custom_component/example_components/preprocessors to ./dc-custom-component-template/src/dc_custom_component/components/generators.
   2. Open the generators folder and rename the example file character_splitter.py to welcome_text_generator.py.
   3. Delete the rankers folder if it's not needed. If you're creating multiple components, use the folder structure to keep them organized.
      Note: You can create all your components in one .py file or you can have a separate folder with a .py file in it for each custom component. That's up to you.

      📘

      Component Folder as Group Name

      The folder name containing your component code becomes the component group name in Pipeline Builder. For example, if you place your component in ./dc-custom-component-template/src/dc_custom_component/components/generators, it will appear in the Generators group in the Pipeline Builder component library within the Custom tab.

Set Up a Virtual Environment

Creating a virtual environment isolates your project's dependencies from other Python projects and your system Python installation.

The template uses Hatch, a Python project manager, to set up virtual environments. For details, see the Hatch website.

1. Install Hatch by running: pip install hatch. This installs all the necessary packages, including pytest.
2. Create a virtual environment by running: hatch shell.

Implement the Component

1. Write the component code in the .py file. Use the recipe below as a starting point:

📖 Tutorial: Creating a Custom Component

2. If your component has dependencies, add them in the ./dc-custom-component-template/pyproject.toml file in the dependencies section:

dependencies = [
  "haystack-ai>=2.0.0"
]

Note: Do not modify versions of dependencies already listed in this file.

3. From the project root directory, run the hatch run code-quality:all command to format your code.
4. Update the component version in the ./dc-custom-component-template/src/dc_custom_component/__about__.py file. You can specify version numbers in any way you like, but we suggest you adopt the major.minor.micro format, for example, 1.1.0. Have a look at Hatch versioning for guidelines.
   The version number applies to all components. Even if you have multiple components, you only need to specify one version number.

Components Connecting to Third-Party Providers

If your component connects to a third-party provider that requires authentication, store the API key in an environment variable and then retrieve it in the component's run() method. You can also add a secret on the Secrets page and give it the same name as the environment variable. For details, see Add Secrets to Connect to Third Party Providers.

For components that need to connect to external APIs, you can use environment variables to securely store credentials.

Here's an example of how to use environment variables in your component code:

import os

class MyComponent:
    def __init__(self, api_key: str = None):
        self.api_key = api_key or os.getenv("MY_API_KEY")

Using Secrets for Secure Connections

When working with sensitive information like API keys, it's important to use secure storage methods.

You can store secrets in deepset AI Platform and reference them in your components using environment variables.

For detailed instructions on managing secrets, see the Add Secrets to Connect to Third Party Providers documentation.

Test Your Component

We recommend that you test your component before uploading.

1. Add unit and integration tests in the ./dc-custom-component-template/tests folder to ensure everything works fine.
2. Run your tests using: hatch run tests. If the tests pass, your component is ready.

What to Do Next

Upload your component to deepset AI Platform to use it in your pipelines.