Thanks team for the great work! When I go through pinecone's exporte

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

Export from pinecone seems to be a little bit hacky about vector-io HOT 2 CLOSED

xiaofan-luan commented on June 18, 2024

Export from pinecone seems to be a little bit hacky

from vector-io.

Comments (2)

xiaofan-luan commented on June 18, 2024 1

Good to know!
If there are some common storage format that can be shared between different vectordbs that would be great

from vector-io.

dhruv-anand-aintech commented on June 18, 2024

Hi @xiaofan-luan, thanks for raising this.
I flag this to the user in case the random searches approach doesn't work:

vector-io/src/vdf_io/export_vdf/pinecone_export.py

Lines 295 to 299 in 5120875

 f"CAUTION!!: Could not collect all ids after {MAX_TRIES_OVERALL} tries.\n" 

 "Due to the way Pinecone's API is built, there is no way to find all the ids of points that are stored in an index.\n" 

 "Reference: https://community.pinecone.io/t/how-to-retrieve-list-of-ids-in-an-index/380/20\n" 

 "This library uses a random search method to collect ids, but it is not guaranteed to collect all ids.\n" 

 "Please provide range of ids instead. Exporting the ids collected so far."

In my trials, using random searches approach is sufficient to extract all the points in the index when there are up to ~90k points in the index (takes ~20 tries). I have not tried with indexes much larger than that. There is an even more thorough approach which involves marking metadata on the points that have been downloaded (

vector-io/src/vdf_io/export_vdf/pinecone_export.py

Lines 45 to 51 in 5120875

 parser_pinecone.add_argument( 

 "--modify_to_search", 

 type=bool, 

 help="Allow modifying data to search", 

 default=False, 

 action=argparse.BooleanOptionalAction, 

 )

) using --modify_to_search, but it is more time-consuming due to Pinecone's rate-limiting on these operations.

The good news is that Pinecone has recently released a list vectors endpoint (https://docs.pinecone.io/docs/get-record-ids), but it is only for serverless indexes through the REST API for now. When they release it for pod-based indexes as well, and with python client support, I will switch to that method.

from vector-io.

Recommend Projects

Export from pinecone seems to be a little bit hacky about vector-io HOT 2 CLOSED

Comments (2)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	f"CAUTION!!: Could not collect all ids after {MAX_TRIES_OVERALL} tries.\n"
	"Due to the way Pinecone's API is built, there is no way to find all the ids of points that are stored in an index.\n"
	"Reference: https://community.pinecone.io/t/how-to-retrieve-list-of-ids-in-an-index/380/20\n"
	"This library uses a random search method to collect ids, but it is not guaranteed to collect all ids.\n"
	"Please provide range of ids instead. Exporting the ids collected so far."

	parser_pinecone.add_argument(
	"--modify_to_search",
	type=bool,
	help="Allow modifying data to search",
	default=False,
	action=argparse.BooleanOptionalAction,
	)