To find an answer to a query, your pipeline searches the files you uploaded to deepset AI Platform. When you deploy the pipeline, the files are indexed and ready to be searched. However, data is rarely static, and it's only natural that your files should change: get deleted, added, or updated. Your app should always be in sync with your file storage.

Syncing Your Data

The easiest way to ensure your search runs on the latest version of your files is to create a script that periodically syncs your data with deepset AI Platform. deepset AI Platform provides endpoints for uploading and deleting files so you can easily remove old data and add new ones.

Currently, there is no endpoint for updating files, so if a file that's already in deepset is modified, you must delete it and upload the latest version.

Another option is to use the deepset SDK and upload files with the overwrite option. The SDK has no option to delete files, so you'd need to do that through the API endpoint or the UI.

You can also synchronize your data using Airbyte, a data integration platform designed to help move data from various sources to destinations. deepset AI Platform is one of the available destinations. For details, see Synchronize Data Using Airbyte.

Indexing

Whenever you add a file, it's automatically re-indexed. You don't have to redeploy the pipeline to trigger indexing.