MarkdownToDocument

Convert Markdown files into text documents your pipeline can query. Use this component in indexes to write Markdown file contents into a document store.

---

Key Features

• Converts Markdown files to plain text documents.
• Optionally converts table contents to a single line.
• Displays a progress bar when processing files.
• Accepts both file paths and ByteStream objects as input.
• Attaches optional metadata to the resulting documents.
• Integrates with FileTypeRouter and DocumentJoiner in multi-format indexing pipelines.

Configuration

1. Drag the MarkdownToDocument component onto the canvas from the Component Library.
2. Click the component to open the configuration panel.
3. Configure the parameters as needed.

Connections

MarkdownToDocument accepts a list of file paths or ByteStream objects (sources) as input, along with optional metadata (meta). It outputs a list of converted documents (documents).

Typically, MarkdownToDocument receives Markdown files routed from FileTypeRouter and sends its output to DocumentJoiner, which combines documents from multiple converters before passing them downstream for splitting and indexing.

Usage Example

Using the Component in an Index

In this index, MarkdownToDocument receives Markdown files from FileTypeRouter and sends them to DocumentJoiner.

components:
  FileTypeRouter:
    type: haystack.components.routers.file_type_router.FileTypeRouter
    init_parameters:
      mime_types:
      - text/text
      - text/markdown
      - application/json
      additional_mimetypes:
  TextFileToDocument:
    type: haystack.components.converters.txt.TextFileToDocument
    init_parameters:
      encoding: utf-8
      store_full_path: false
  MarkdownToDocument:
    type: haystack.components.converters.markdown.MarkdownToDocument
    init_parameters:
      table_to_single_line: false
      progress_bar: true
      store_full_path: false
  DocumentJoiner:
    type: haystack.components.joiners.document_joiner.DocumentJoiner
    init_parameters:
      join_mode: concatenate
      weights:
      top_k:
      sort_by_score: true
  DocumentSplitter:
    type: haystack.components.preprocessors.document_splitter.DocumentSplitter
    init_parameters:
      split_by: word
      split_length: 200
      split_overlap: 0
      split_threshold: 0
      splitting_function:
      respect_sentence_boundary: false
      language: en
      use_split_rules: true
      extend_abbreviations: true
  DocumentWriter:
    type: haystack.components.writers.document_writer.DocumentWriter
    init_parameters:
      policy: NONE
      document_store:
        type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
        init_parameters:
          hosts:
          index: aga-index
          max_chunk_bytes: 104857600
          embedding_dim: 768
          return_embedding: false
          method:
          mappings:
          settings:
          create_index: true
          http_auth:
          use_ssl:
          verify_certs:
          timeout:
  JSONConverter:
    type: haystack.components.converters.json.JSONConverter
    init_parameters:
      jq_schema:
      content_key:
      extra_meta_fields:
      store_full_path: false

connections:
- sender: FileTypeRouter.text/text
  receiver: TextFileToDocument.sources
- sender: FileTypeRouter.text/markdown
  receiver: MarkdownToDocument.sources
- sender: TextFileToDocument.documents
  receiver: DocumentJoiner.documents
- sender: MarkdownToDocument.documents
  receiver: DocumentJoiner.documents
- sender: DocumentJoiner.documents
  receiver: DocumentSplitter.documents
- sender: DocumentSplitter.documents
  receiver: DocumentWriter.documents

- sender: FileTypeRouter.application/json
  receiver: JSONConverter.sources
- sender: JSONConverter.documents
  receiver: DocumentJoiner.documents

max_runs_per_component: 100

metadata: {}

inputs:
  files:
  - FileTypeRouter.sources

Parameters

Inputs

Parameter	Type	Default	Description
sources	List[Union[str, Path, ByteStream]]		List of file paths or ByteStream objects to convert.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	None	Optional metadata to attach to the Documents. This value can be either a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced Documents. If it's a list, the length of the list must match the number of sources, because the two lists will be zipped. If sources contains ByteStream objects, their meta will be added to the output Documents.

Outputs

Parameter	Type	Default	Description
documents	List[Document]		Converted documents.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
table_to_single_line	bool	False	If True converts table contents into a single line.
progress_bar	bool	True	If True shows a progress bar when running.
store_full_path	bool	False	If True, the full path of the file is stored in the metadata of the document. If False, only the file name is stored.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
sources	List[Union[str, Path, ByteStream]]		List of file paths or ByteStream objects to convert.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	None	Optional metadata to attach to the Documents. This value can be either a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced Documents. If it's a list, the length of the list must match the number of sources, because the two lists will be zipped. If sources contains ByteStream objects, their meta will be added to the output Documents.