Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Generate a Retrieval Dataset from an IndexBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Score how well your pipeline retrieves documents without labeling any data. The platform samples passages from an index, has an LLM write one question per passage, and saves each question with the passage it should find.


About This Task​

Retrieval Metrics such as Recall, MRR, and nDCG compare the documents your pipeline retrieved with the documents a test case expects. Most teams have no such labeled data. A generated Dataset gives you a starting point: every test case is a question plus the passage it was written from.

The LLM is prompted to paraphrase the passage instead of copying its words. A question that reuses the passage's words favors keyword search and makes retrieval look easier than it is.

Generation runs in your browser and calls the LLM once per question. Those calls are paid through your workspace's connection to the model.

Prerequisites​

  • An index with indexed files. Only the first 100 indexes of the workspace are listed.
  • An LLM connected in your workspace to write the questions.
  • The pipeline you want to score searches this index.

Generate the Dataset​

  1. Open a pipeline, click Evaluation, and go to Datasets.
  2. Click Generate from index.
  3. In Index, choose the index your pipeline searches.
  4. In LLM that writes the questions, choose a connected LLM.
  5. In Questions, enter how many questions you want. The default is 20. A Dataset holds at most 100 test cases, so 100 is the maximum.
  6. In Dataset name, keep the default, Generated from <index>, <date>, or type your own name.
  7. Check the cost note. It shows how many LLM calls the run makes and which connection pays for them.
  8. Click Generate, and keep the window open until the run finishes. A progress bar shows how many questions are written. When the run finishes, the platform creates the Dataset and opens it.

Each test case sends the question as inputs.query and expects the passage's ID in expected.document_ids, the column the retrieval Metrics read.

How the Passages Are Chosen​

The platform reads the first 400 chunks of the index and keeps those with at least 200 characters. It takes chunks from one file at a time in turn, so no single file fills the Dataset. If the index has fewer suitable chunks than you asked for, the Dataset has fewer test cases.

If the LLM writes the same question for several chunks, the platform saves one test case that expects all of them. Two separate rows would score a correct hit as a miss.

A chunk whose question fails is left out of the Dataset.

Score Your Pipeline on the Dataset​

Add the retrieval Metrics to a Metric Group and run an experiment on the new Dataset. For the steps, see Add Built-In Metrics and Run an Experiment.

Limits​

  • Each question has one known answer, so nDCG is binary on a generated Dataset: a document is either the expected one or it isn't.
  • Chunk IDs change when you rebuild the index. A generated Dataset belongs to the index as it was when you generated it. After a rebuild, generate a new Dataset.
  • The Dataset name is the only record of where the Dataset came from, so keep the index name in it.

Fix Generation Problems​

  • The window closed or you left the page. Closing the dialog, leaving the page, or switching workspaces stops the run and saves nothing. Start again and keep the window open.
  • The LLM list is empty. Connect an LLM in this workspace, then open the dialog again.
  • The index has no suitable passages. The index has no chunks of at least 200 characters among its first 400. Choose an index with indexed files.
  • Every question failed. The dialog shows the LLM's error and saves nothing. Fix the cause the error names, for example the model connection, and click Generate again.
  • The save failed. The written questions are kept, and the button changes to Save Dataset. Click it to save again. This makes no new LLM calls.