FirecrawlWebSearch

Search the web and return results as Haystack Documents using the Firecrawl Search API.

Key Features

• Searches the web and returns structured page content and links as Haystack Documents.
• Follows the standard Haystack WebSearch component interface.
• Returns both Document objects and a list of result URLs.
• Supports both synchronous and asynchronous execution.
• You need a Firecrawl API key from firecrawl.dev.

Configuration

1. Sign up at firecrawl.dev and get an API key.
2. Set the FIRECRAWL_API_KEY environment variable. For instructions, see Create Secrets.
3. Configure FirecrawlWebSearch in your pipeline YAML with the desired top_k.

Connections

FirecrawlWebSearch takes a search query string as query input and outputs a list of Document objects (documents) and a list of result URLs (links).

Connect the pipeline's query input to the query input. Connect the documents output to a PromptBuilder or retriever for further processing.

Source Code

To check this component's source code, open firecrawl_websearch.py in the Haystack Core Integrations repository.

Usage Examples

Basic Configuration

  firecrawl_websearch:
    type: haystack_integrations.components.websearch.firecrawl.firecrawl_websearch.FirecrawlWebSearch
    init_parameters:
      api_key:
        type: env_var
        env_vars:
        - FIRECRAWL_API_KEY
        strict: false
      top_k: 5

Using the Component in a Pipeline

# haystack-pipeline
components:
  firecrawl_websearch:
    type: haystack_integrations.components.websearch.firecrawl.firecrawl_websearch.FirecrawlWebSearch
    init_parameters:
      api_key:
        type: env_var
        env_vars:
        - FIRECRAWL_API_KEY
        strict: false
      top_k: 5
  prompt_builder:
    type: haystack.components.builders.prompt_builder.PromptBuilder
    init_parameters:
      template: |
        Given these web search results:
        {% for doc in documents %}{{ doc.content }}{% endfor %}
        Answer: {{ query }}
  llm:
    type: haystack.components.generators.chat.openai.OpenAIChatGenerator
    init_parameters:
      model: gpt-4o-mini

connections:
- sender: firecrawl_websearch.documents
  receiver: prompt_builder.documents
- sender: prompt_builder.prompt
  receiver: llm.messages

inputs:
  query:
  - firecrawl_websearch.query
  - prompt_builder.query

outputs:
  replies: llm.replies

max_runs_per_component: 100

metadata: {}

Parameters

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
api_key	Secret	Secret.from_env_var("FIRECRAWL_API_KEY")	Firecrawl API key.
top_k	int | None	10	Maximum number of search results to return. Can be overridden by a limit key in search_params.
search_params	dict | None	None	Additional parameters passed to the Firecrawl search API. Supported keys include tbs, location, scrape_options, sources, categories, and timeout. See the Firecrawl search API reference for details.

Related Information

• FirecrawlCrawler
• Pipeline Components
• Firecrawl documentation