Test Hosted Models on Sample PassagesBeta
Try a hosted embedder or reranker before you add it to a pipeline. Rank a few passages for a query and compare the order that up to three models give. You don't need files, a pipeline, or a connection to a model provider, and nothing you enter is stored.
About This Task
The Embedders and Rerankers tabs on the Models page list the models that deepset hosts. A test sends your query and passages to each selected model once, so you can see which model ranks the right passage first for your kind of questions, languages, or tricky wording, such as negation.
How a model ranks depends on its type:
- A reranker orders the passages by its relevance judgment for the query.
- An embedder embeds the query with a text embedder and the passages with a document embedder, as your query and indexing pipelines do. The passages are ordered by cosine similarity.
The drawer shows ranks (#1, #2, and so on), not scores. Embedding models are trained to rank, so their scores can't be compared across models.
Before You Start
Make sure you have permission to run pipelines in the workspace. Without it, every model column shows an access error.
Test Models on Sample Passages
- Open the Models page and click the Embedders or Rerankers tab.
- Click Test at the end of the row of the model you want to try. In the drawer, make sure Sample passages is selected.
- Enter a query in Query. Enter up to 20 passages in Passages, one per line. To start from a known-answer sample instead, choose one in Example. The examples include a cross-language query and a negation, and each marks the passage that answers the query as the expected answer.
- Optionally, in Compare with, add up to 2 more models of the same type.
- Click Rank passages.
- Compare the columns. The first column is the model you opened. The best model for your use case puts the expected answer first. A passage marked Only this model is in that model's top 3 and in no other model's top 3.
Troubleshooting
If a column shows an error, that model couldn't rank the passages:
- Access error: You need permission to run pipelines in the workspace. Ask a workspace admin for access.
- The request fails or times out: Try again, or use fewer passages. One failing model doesn't affect the other columns.
What To Do Next
- To test a hosted model on the documents your own pipeline retrieves, see Compare Hosted Models on a Deployed Pipeline.
- To add a model to a pipeline, see Copy YAML for a Hosted Embedder or Reranker.
Related Information
Was this page helpful?