What's Generative QA?

Generative QA uses large language models to generate human-like responses to user queries in question answering apps. Instead of simply extracting answers from existing documents, generative systems create a new text based on instructions provided in the prompt. A prompt is a specific instruction given to the model in natural language to help it understand the task and generate an appropriate response.

To build a basic generative QA system, you need a large language model (LLM), like GPT-4, and a simple prompt, such as a user query. LLMs are trained to predict the next word in a sequence and generate answers token by token. Based on the prompt, the model generates an answer.

Advantages

Generative QA offers several advantages compared to extractive QA:

• Extracting information from multiple sources: The system can extract and combine information from various sources to produce coherent and informative answers from scratch.
• Generating original content: LLMs go beyond extracting existing answers and produce more creative and personalized responses. You can use them to create a system that matches your brand voice or feels like talking to a real human.
• Language flexibility: LLMs are trained on large amounts of text, allowing them to generate natural language responses that include idioms or language variations.
• Reasoning capabilities: LLMs have some reasoning capabilities, which make it possible for them to compare facts or draw conclusions.

Challenges

Generative QA systems also have limitations that are important to consider:

• Cost: Proprietary LLMs often come with a price tag. The pricing models are based on the number of tokens processed or per query.
• Limited context length: Models have a maximum token limit they can handle. This includes both the prompt and the generated output. When handling long documents, the model may truncate the generated text to fit within the allowed context length.
• Factual accuracy: Models may generate fictional or incorrect outputs with a high level of confidence. This is known as hallucinations. Hallucinations can occur because of biases in training data or the model's inability to differentiate between factual and fictional information.
• Latency: Generative QA systems are generally slower compared to extractive QA systems.
• Output control: LLMs can sometimes generate harmful, inappropriate, or biased content.
• Evaluation: Evaluating generative models remains challenging. Some of the reasons for that include a lack of objective metrics. Evaluation often involves subjective human judgment, and the diverse outputs make establishing a single ground truth difficult.

Generative QA on Your Data

To address concerns about hallucinations and unreliable answers, you can run a generative QA system on your own data using a retrieval-augmented generation (RAG) approach. This involves limiting the context to a predefined set of documents and adding a Retriever component to your QA system.

In deepset Cloud, you can pass your documents in the prompt. When given a query, the Retriever finds the most relevant documents, injects them in the prompt, and passes them on to the generative model. The model uses these documents to produce the final answer.

This approach makes your system more reliable and ensures it runs on your data.

Applications

Generative QA systems excel in scenarios where you need novel, natural-language answers and not verbatim from existing documents. Some popular applications include:

• Chatbots, AI assistants, and customer support applications offering personalized assistance across various questions.
• Writing aids and content generation apps automating content creation tasks and assisting with content curation.
• Learning assistants in educational applications, providing explanations and summaries of content.
• Translation aids.

Generative QA in deepset Cloud

Generative QA pipelines in deepset Cloud can:

• Detect hallucinations using the HallucinationDetector node.
• Work on your own data (retrieval-augmented generation). You can pass the documents in the prompt and use a Retriever node to filter them.

deepset Cloud supports you throughout the process of creating a generative QA system by:

• Providing a pipeline template ready to use as it is.
• Providing Prompt Explorer, which is a sandbox environment for testing out prompts.
• Offering a library of tested and curated prompts you can reuse and modify, as needed.

deepset Cloud uses PromptNode for building generative QA pipelines. By default, it uses the free google/flan-t5-base model, but you can replace it with other models, like GPT-4 by OpenAI, Claude by Anthropic, Command by Cohere, and more.

Take a look at Use Case: Generative AI Systems to learn when it's best to use a generative system and what data it needs.

And if you want to see what it looks like in production, test our Generative QA pipeline that runs on 10-K annual reports of the largest companies.