Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for migrating qdrant collections #5

Merged
merged 13 commits into from
Jul 24, 2024
46 changes: 44 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ To migrate from [Pinecone serverless](https://www.pinecone.io/blog/serverless/)

From the Supabase instance we need the connection parameters. Retrive them [here](https://supabase.com/dashboard/project/_/settings/database)

![supabsae connection parameters](/assets/supabase_connection_params.png)
![supabase connection parameters](/assets/supabase_connection_params.png)

And substitute those values into a valid Postgres connection string
```
Expand All @@ -120,7 +120,7 @@ The CLI provies a progress bar to monitor the migration.

On completion, you can view a copy of the Pinecone index data in Supabase Postgres at `vec2pg.<pinecone index name>`

![view results](/assets/view_results.png)
![view results](/assets/pinecone_view_results.png)

From there you can transform and manipulate the data in Postgres using SQL.

Expand All @@ -145,6 +145,48 @@ vec2pg qdrant migrate --help
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```

To migrate from Qdrant collection to Postgres you'll need to log in at https://cloud.qdrant.io/ and collect your:

- Qdrant API Key

![Qdrant api key](/assets/qdrant_api_key.png)

- Qdrant URL and collection name

![Qdrant cluster url](/assets/qdrant_nav_dashboard.png)

The URL is the "Cluster URL". To find the collection name, select "Open Dashboard".

![Qdrant collection name](/assets/qdrant_collection_name.png)

- A Supabase instance

From the Supabase instance we need the connection parameters. Retrive them [here](https://supabase.com/dashboard/project/_/settings/database)
olirice marked this conversation as resolved.
Show resolved Hide resolved
olirice marked this conversation as resolved.
Show resolved Hide resolved

![supabase connection parameters](/assets/supabase_connection_params.png)

And substitute those values into a valid Postgres connection string
```
postgresql://<User>:<Password>@<Host>:<Port>/postgres
```
e.g.
```
postgresql://postgres.ahqsutirwnsocaaorimo:<Password>@aws-0-us-east-1.pooler.supabase.com:6543/postgres
```

Then we can call `vec2pg qdrant migrate` passing our values. You can supply all parameters directly to the CLI, but its a good idea to pass the Qdrant API Key (QDRANT_API_KEY) and Supabase connection string (POSTGRES_CONNECTION_STRING) as environment variables to avoid logging credentials to your shell's history.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Then we can call `vec2pg qdrant migrate` passing our values. You can supply all parameters directly to the CLI, but its a good idea to pass the Qdrant API Key (QDRANT_API_KEY) and Supabase connection string (POSTGRES_CONNECTION_STRING) as environment variables to avoid logging credentials to your shell's history.
Then we can call `vec2pg qdrant migrate` passing our values. You can supply all parameters directly to the CLI, but it's a good idea to pass the Qdrant API Key (QDRANT_API_KEY) and Supabase connection string (POSTGRES_CONNECTION_STRING) as environment variables to avoid logging credentials to your shell's history.


![sample output](/assets/qdrant_to_supabase_output.png)

The CLI provies a progress bar to monitor the migration.
olirice marked this conversation as resolved.
Show resolved Hide resolved

On completion, you can view a copy of the Pinecone index data in Supabase Postgres at `vec2pg.<qdrant collection name>`

![view results](/assets/qdrant_view_results.png)

From there you can transform and manipulate the data in Postgres using SQL.



# Requisites
- Python >= 3.8
Expand Down
File renamed without changes
Binary file added assets/qdrant_api_key.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/qdrant_collection_name.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/qdrant_nav_dashboard.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/qdrant_to_supabase_output.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/qdrant_view_results.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.