List All Vector Records in Pinecone DB

List All Vector Records in Pinecone DB

Pinecone DB is a great choice for your vector database, but how do we migrate/update/adjust data for an entire index? After reading the official docs there is no API method for list or get all that can be found. The code below is the best method I have found to make changes to your vector records at the moment. The limitation of this method is the 10000 limit per namespace and will not work for everyone.

def list_all_embeddings():
    stats = pinecone_index.describe_index_stats()
    namespace_map = stats['namespaces']
    ret = []
    for namespace in namespace_map:
        vector_count = namespace_map[namespace]['vector_count']
        res = pinecone_index.query(vector=[0 for _ in range(1536)], top_k=10000, namespace=namespace)
        for match in res['matches']:
            ret.append(match)
    return ret

If you are stuck and cannot get all of your records using this pattern, you could possibly get smaller batch of vectors by filtering by metadata. This requires knowledge of the data so it is not written without knowledge of the data model.