JSONConverter

Convert JSON files into documents your pipelines can query.

Basic Information

• Type: haystack.components.converters.json.JSONConverter
• Components it can connect with:
  • FileTypeRouter: JSONConverter can receive JSON files from FileTypeRouter.
  • DocumentJoiner: JSONConverter can send converted documents to DocumentJoiner. This is useful if you have other converters in your pipeline and want to join their output with JSONConverter's output before sending it further down the pipeline.

Inputs

Parameter	Type	Default	Description
sources	List[Union[str, Path, ByteStream]]		A list of file paths or ByteStream objects to convert.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	None	Optional metadata to attach to the documents. This value can be either a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced documents. If it's a list, the length of the list must match the number of sources. If sources contain ByteStream objects, their meta will be added to the output documents.

Outputs

Parameter	Type	Default	Description
documents	List[Document]		A list of created documents.

Overview

When configuring JSONConverter, you can use jq_schema to filter the JSON source files and extra_meta_fields to extract from the filtered data.

The jq_schema parameter is used to filter the JSON source files and extract only specific parts. For information on the filter syntax, see the jq documentation. If jq_schema is not specified, the whole JSON file is used to extract information.

You can use the content_key parameter to choose which key from the extracted object becomes the document's content. You must set either jq_schema, content_key, or both.

• If both are set, content_key is looked up in the data extracted by jq_schema.
• If only content_key is set, the source JSON file must be a JSON object, else it is skipped.
• If only jq_schema is set, the extracted data must be a scalar value. If it's a JSON object or array, it is skipped.

Usage Example

Initializing the Component

components:
  JSONConverter:
    type: haystack.components.converters.json.JSONConverter
    init_parameters:

Using the Component in an Index

In this index, JSONConverter receives JSON files from FileTypeRouter and sends them to DocumentJoiner that combines documents from different Converters.

components:
  FileTypeRouter:
    type: haystack.components.routers.file_type_router.FileTypeRouter
    init_parameters:
      mime_types:
      - text/text
      - text/markdown
      - application/json
      additional_mimetypes:
  TextFileToDocument:
    type: haystack.components.converters.txt.TextFileToDocument
    init_parameters:
      encoding: utf-8
      store_full_path: false
  MarkdownToDocument:
    type: haystack.components.converters.markdown.MarkdownToDocument
    init_parameters:
      table_to_single_line: false
      progress_bar: true
      store_full_path: false
  DocumentJoiner:
    type: haystack.components.joiners.document_joiner.DocumentJoiner
    init_parameters:
      join_mode: concatenate
      weights:
      top_k:
      sort_by_score: true
  DocumentSplitter:
    type: haystack.components.preprocessors.document_splitter.DocumentSplitter
    init_parameters:
      split_by: word
      split_length: 200
      split_overlap: 0
      split_threshold: 0
      splitting_function:
      respect_sentence_boundary: false
      language: en
      use_split_rules: true
      extend_abbreviations: true
  DocumentWriter:
    type: haystack.components.writers.document_writer.DocumentWriter
    init_parameters:
      policy: NONE
      document_store:
        type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
        init_parameters:
          hosts:
          index: aga-index
          max_chunk_bytes: 104857600
          embedding_dim: 768
          return_embedding: false
          method:
          mappings:
          settings:
          create_index: true
          http_auth:
          use_ssl:
          verify_certs:
          timeout:
  JSONConverter:
    type: haystack.components.converters.json.JSONConverter
    init_parameters:
      jq_schema:
      content_key:
      extra_meta_fields:
      store_full_path: false

connections:
- sender: FileTypeRouter.text/text
  receiver: TextFileToDocument.sources
- sender: FileTypeRouter.text/markdown
  receiver: MarkdownToDocument.sources
- sender: TextFileToDocument.documents
  receiver: DocumentJoiner.documents
- sender: MarkdownToDocument.documents
  receiver: DocumentJoiner.documents
- sender: DocumentJoiner.documents
  receiver: DocumentSplitter.documents
- sender: DocumentSplitter.documents
  receiver: DocumentWriter.documents

- sender: FileTypeRouter.application/json
  receiver: JSONConverter.sources
- sender: JSONConverter.documents
  receiver: DocumentJoiner.documents

max_runs_per_component: 100

metadata: {}

inputs:
  files:
  - FileTypeRouter.sources

Parameters

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
jq_schema	Optional[str]	None	Optional jq filter string to extract content. If not specified, whole JSON object will be used to extract information.
content_key	Optional[str]	None	Optional key to extract document content. If jq_schema is specified, the content_key will be extracted from that object.
extra_meta_fields	Optional[Union[Set[str], Literal['*']]]	None	An optional set of meta keys to extract from the content. If jq_schema is specified, all keys will be extracted from that object.
store_full_path	bool	False	If True, the full path of the file is stored in the metadata of the document. If False, only the file name is stored.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
sources	List[Union[str, Path, ByteStream]]		A list of file paths or ByteStream objects to convert.
meta	Optional[Union[Dict[str, Any], List[Dict[str, Any]]]]	None	Optional metadata to attach to the documents. This value can be either a list of dictionaries or a single dictionary. If it's a single dictionary, its content is added to the metadata of all produced documents. If it's a list, the length of the list must match the number of sources. If sources contain ByteStream objects, their meta will be added to the output documents.