HuggingFaceLocalChatGenerator
Generate chat responses using models from Hugging Face that run locally.
Key Features
- Chat generation using locally loaded Hugging Face models supporting the ChatML format
- Supports popular models such as
HuggingFaceH4/zephyr-7b-betaandmeta-llama/Llama-2-7b-chat-hf - Streaming support for real-time token-by-token responses
- Tool/function calling support with custom tool parsing
- Configurable generation parameters and pipeline settings
- Note: locally running LLMs may require powerful hardware depending on the model
Configuration
- Drag the
HuggingFaceLocalChatGeneratorcomponent onto the canvas from the Component Library. - Click on the component to open the configuration panel.
- On the General tab:
- Enter the model name or path (for example,
HuggingFaceH4/zephyr-7b-beta). The model must support the ChatML messaging format. - Enter your Hugging Face API token for remote file authorization. For details, see Use Hugging Face Models.
- Optionally, select a device for loading the model. If not set, the component selects the default device automatically.
- Enter the model name or path (for example,
- Go to the Advanced tab to configure task type, chat template, generation kwargs, pipeline kwargs, stop words, streaming, tools, and other settings.
Connections
HuggingFaceLocalChatGenerator accepts a list of ChatMessage objects through its messages input and outputs generated responses as replies (a list of ChatMessage instances).
Connect ChatPromptBuilder's prompt output to this component's messages input. Connect the replies output to DeepsetAnswerBuilder through OutputAdapter.
Source Code
To check this component's source code, open hugging_face_local.py in the Haystack repository.
Usage Examples
Basic Configuration
HuggingFaceLocalChatGenerator:
type: haystack.components.generators.chat.hugging_face_local.HuggingFaceLocalChatGenerator
init_parameters:
model: HuggingFaceH4/zephyr-7b-beta
token:
type: env_var
env_vars:
- HF_API_TOKEN
- HF_TOKEN
strict: false
Using the Component in a Pipeline
This is an example RAG pipeline with HuggingFaceLocalChatGenerator and DeepsetAnswerBuilder connected through OutputAdapter:
components:
bm25_retriever: # Selects the most similar documents from the document store
type: haystack_integrations.components.retrievers.opensearch.bm25_retriever.OpenSearchBM25Retriever
init_parameters:
document_store:
type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
init_parameters:
hosts:
index: 'Standard-Index-English'
max_chunk_bytes: 104857600
embedding_dim: 768
return_embedding: false
method:
mappings:
settings:
create_index: true
http_auth:
use_ssl:
verify_certs:
timeout:
top_k: 20 # The number of results to return
fuzziness: 0
query_embedder:
type: deepset_cloud_custom_nodes.embedders.nvidia.text_embedder.DeepsetNvidiaTextEmbedder
init_parameters:
normalize_embeddings: true
model: intfloat/e5-base-v2
embedding_retriever: # Selects the most similar documents from the document store
type: haystack_integrations.components.retrievers.opensearch.embedding_retriever.OpenSearchEmbeddingRetriever
init_parameters:
document_store:
type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
init_parameters:
hosts:
index: 'Standard-Index-English'
max_chunk_bytes: 104857600
embedding_dim: 768
return_embedding: false
method:
mappings:
settings:
create_index: true
http_auth:
use_ssl:
verify_certs:
timeout:
top_k: 20 # The number of results to return
document_joiner:
type: haystack.components.joiners.document_joiner.DocumentJoiner
init_parameters:
join_mode: concatenate
ranker:
type: deepset_cloud_custom_nodes.rankers.nvidia.ranker.DeepsetNvidiaRanker
init_parameters:
model: intfloat/simlm-msmarco-reranker
top_k: 8
meta_field_grouping_ranker:
type: haystack.components.rankers.meta_field_grouping_ranker.MetaFieldGroupingRanker
init_parameters:
group_by: file_id
subgroup_by:
sort_docs_by: split_id
answer_builder:
type: deepset_cloud_custom_nodes.augmenters.deepset_answer_builder.DeepsetAnswerBuilder
init_parameters:
reference_pattern: acm
ChatPromptBuilder:
type: haystack.components.builders.chat_prompt_builder.ChatPromptBuilder
init_parameters:
template:
- _content:
- text: "You are a helpful assistant answering the user's questions based on the provided documents.\nIf the answer is not in the documents, rely on the web_search tool to find information.\nDo not use your own knowledge.\n"
_role: system
- _content:
- text: "Provided documents:\n{% for document in documents %}\nDocument [{{ loop.index }}] :\n{{ document.content }}\n{% endfor %}\n\nQuestion: {{ query }}\n"
_role: user
required_variables:
variables:
OutputAdapter:
type: haystack.components.converters.output_adapter.OutputAdapter
init_parameters:
template: '{{ replies[0] }}'
output_type: List[str]
custom_filters:
unsafe: false
HuggingFaceLocalChatGenerator:
type: haystack.components.generators.chat.hugging_face_local.HuggingFaceLocalChatGenerator
init_parameters:
model: HuggingFaceH4/zephyr-7b-beta
task:
device:
token:
type: env_var
env_vars:
- HF_API_TOKEN
- HF_TOKEN
strict: false
chat_template:
generation_kwargs:
huggingface_pipeline_kwargs:
stop_words:
streaming_callback:
tools:
tool_parsing_function:
async_executor:
connections: # Defines how the components are connected
- sender: bm25_retriever.documents
receiver: document_joiner.documents
- sender: query_embedder.embedding
receiver: embedding_retriever.query_embedding
- sender: embedding_retriever.documents
receiver: document_joiner.documents
- sender: document_joiner.documents
receiver: ranker.documents
- sender: ranker.documents
receiver: meta_field_grouping_ranker.documents
- sender: meta_field_grouping_ranker.documents
receiver: answer_builder.documents
- sender: meta_field_grouping_ranker.documents
receiver: ChatPromptBuilder.documents
- sender: OutputAdapter.output
receiver: answer_builder.replies
- sender: ChatPromptBuilder.prompt
receiver: HuggingFaceLocalChatGenerator.messages
- sender: HuggingFaceLocalChatGenerator.replies
receiver: OutputAdapter.replies
inputs: # Define the inputs for your pipeline
query: # These components will receive the query as input
- "bm25_retriever.query"
- "query_embedder.text"
- "ranker.query"
- "answer_builder.query"
- "ChatPromptBuilder.query"
filters: # These components will receive a potential query filter as input
- "bm25_retriever.filters"
- "embedding_retriever.filters"
outputs: # Defines the output of your pipeline
documents: "meta_field_grouping_ranker.documents" # The output of the pipeline is the retrieved documents
answers: "answer_builder.answers" # The output of the pipeline is the generated answers
max_runs_per_component: 100
metadata: {}
Parameters
Inputs
| Parameter | Type | Default | Description |
|---|---|---|---|
messages | List[ChatMessage] | A list of ChatMessage objects representing the input messages. | |
generation_kwargs | Optional[Dict[str, Any]] | None | Additional keyword arguments for text generation. |
streaming_callback | Optional[StreamingCallbackT] | None | An optional callable for handling streaming responses. |
tools | Optional[Union[List[Tool], Toolset]] | None | A list of tools or a Toolset for which the model can prepare calls. If set, overrides the tools parameter provided during initialization. |
Outputs
| Parameter | Type | Description |
|---|---|---|
replies | List[ChatMessage] | A list containing the generated responses as ChatMessage instances. |
Init Parameters
These are the parameters you can configure in Pipeline Builder:
| Parameter | Type | Default | Description |
|---|---|---|---|
model | str | HuggingFaceH4/zephyr-7b-beta | The Hugging Face text generation model name or path (for example, mistralai/Mistral-7B-Instruct-v0.2). The model must be a chat model supporting the ChatML messaging format. If the model is specified in huggingface_pipeline_kwargs, this parameter is ignored. |
task | Optional[Literal['text-generation', 'text2text-generation']] | None | The task for the Hugging Face pipeline. Options: text-generation (decoder models like GPT). If the task is specified in huggingface_pipeline_kwargs, this parameter is ignored. |
device | Optional[ComponentDevice] | None | The device for loading the model. If None, automatically selects the default device. |
token | Optional[Secret] | Secret.from_env_var(['HF_API_TOKEN', 'HF_TOKEN'], strict=False) | The token to use as HTTP bearer authorization for remote files. |
chat_template | Optional[str] | None | An optional Jinja template for formatting chat messages. Most high-quality chat models have their own templates; use this only for models without a template or when you prefer a custom one. |
generation_kwargs | Optional[Dict[str, Any]] | None | A dictionary with keyword arguments to customize text generation: max_length, max_new_tokens, temperature, top_k, top_p. The only default is max_new_tokens set to 512. See Hugging Face documentation. |
huggingface_pipeline_kwargs | Optional[Dict[str, Any]] | None | Dictionary with keyword arguments to initialize the Hugging Face pipeline. In case of duplication, these kwargs override model, task, device, and token. Can also include model_kwargs. See Hugging Face pipeline documentation. |
stop_words | Optional[List[str]] | None | A list of stop words. If the model generates a stop word, the generation stops. |
streaming_callback | Optional[StreamingCallbackT] | None | An optional callable for handling streaming responses. |
tools | Optional[Union[List[Tool], Toolset]] | None | A list of tools or a Toolset for which the model can prepare calls. |
tool_parsing_function | Optional[Callable[[str], Optional[List[ToolCall]]]] | None | A callable that takes a string and returns a list of ToolCall objects or None. If None, the default tool parser is used. |
async_executor | Optional[ThreadPoolExecutor] | None | Optional ThreadPoolExecutor for async calls. If not provided, a single-threaded executor is initialized. |
Run Method Parameters
These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.
| Parameter | Type | Default | Description |
|---|---|---|---|
messages | List[ChatMessage] | A list of ChatMessage objects representing the input messages. | |
generation_kwargs | Optional[Dict[str, Any]] | None | Additional keyword arguments for text generation. |
streaming_callback | Optional[StreamingCallbackT] | None | An optional callable for handling streaming responses. |
tools | Optional[Union[List[Tool], Toolset]] | None | A list of tools or a Toolset for which the model can prepare calls. |
Related Information
Was this page helpful?