What is Pinecone

Pinecone is a cloud-native, managed vector database that enables the construction of high-performance vector search applications with ease. With a simple API and no complications of infrastructure, Pinecone is designed to be seamless and hassle-free. Its essence lies in providing businesses with the ability to effortlessly search, match, and rank results based on complex vector computations, an essential feature in modern machine learning applications.

Features of Pinecone

Pinecone brings a robust set of features on the table, making it a valuable tool for developers dealing with large-scale machine learning applications:

• Fast
  Pinecone allows ultra-low latency query responses, even when handling tens of billions of items. This makes it ideal for large-scale applications, providing high speed and efficiency irrespective of the scale.

• Fresh
  With Pinecone, you can get live updates of the index whenever data is added, edited, or removed. This ensures that your data is always up-to-date, providing an accurate reflection of your data at any given time.

• Filtered
  Pinecone allows the combination of vector search and metadata filters. This capability results in more accurate and faster outcomes, as it streamlines the search process and provides more relevant results.

• Fully Managed
  Ease of deployment, use, and expansion are key features of Pinecone. It's designed to simplify operations while maintaining smooth and secure operations, freeing up time and resources for developers to focus more on the application development instead of database management.

Concepts in Pinecone

I will introduce some of Pinecone's primary concepts.

Organization Management

• Organization
  An organization is a collection of projects that utilize the same billing. Each project is necessarily part of one organization. The organization has the power to manage the members and quotas of the projects under its umbrella.

• Project
  A project is responsible for managing Pods and Indices. It determines the usage environment (such as cloud services and regions). The API keys are issued at the project level, adding an extra layer of security and manageability.

https://docs.pinecone.io/docs/organizations
https://docs.pinecone.io/docs/projects

Index Management

• Index
  An index is a unit that manages vectors. It defines various attributes like the dimension of the vectors, the metrics used for similarity calculations (cosine similarity, Euclidean distance, dot product), and the types of vector metadata.

• Vector
  A vector is the data stored in the index. These are high-dimensional data points, each representing some item or feature in the application.

• Metadata
  Metadata is the additional information attached to a vector. Pinecone supports various data types like strings, numbers, boolean values, and lists of strings. Metadata can be used to filter vectors during a query, providing more efficient search results.

• Pod
  A Pod is the hardware unit that manages the index. Each index is maintained by one or more Pods. Each Pod has a specific type and size.

• Type
  The type of a Pod is either s1 (for storage efficiency), or p1, p2 (for performance efficiency).

• Size
  The size of a Pod refers to the amount of data it can store. The capacity is determined by the combination of type and size.

• Namespace
  A namespace allows the division of vectors within an index into different collections. Each vector belongs to exactly one namespace. By specifying the namespace during a query, you can search for vectors in a specific namespace.

• Collection
  A collection is a dump of the index at a particular point in time. It can be used for data backup or to transfer data between different Pods. This feature can be particularly useful in disaster recovery scenarios or for maintaining data integrity across different hardware units.

https://docs.pinecone.io/docs/indexes

API Endpoints

Pinecone provides several API endpoints that handle vector operations. These APIs can also be accessed from Python or Node.js client libraries.

https://docs.pinecone.io/reference/describe_index_stats_post

DescribeIndexStats

The DescribeIndexStats API returns statistics about the contents of the index. This includes the count of vectors per namespace and the number of dimensions, allowing users to gain insights about the current state of the index.

$ curl --request POST \
     --url https://index_name-project_id.svc.environment.pinecone.io/describe_index_stats \
     --header 'accept: application/json' \
     --header 'content-type: application/json'

{
  "namespaces": {
    "": {
      "vectorCount": 50000
    },
    "example-namespace-2": {
      "vectorCount": 30000
    }
  },
  "dimension": 1024,
  "index_fullness": 0.4
}

Fetch

The Fetch API is used to retrieve vectors based on their IDs. This operation is particularly useful when you want to access specific vectors from the index without performing a similarity search.

$ curl --request GET \
     --url 'https://index_name-project_id.svc.environment.pinecone.io/vectors/fetch?ids=example-vector-1' \
     --header 'accept: application/json'

{
  "vectors": {
    "additionalProp": {
      "id": "example-vector-1",
      "values": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8],
      "sparseValues": {
        "indices": [1, 312, 822, 14, 980],
        "values": [0.1, 0.2, 0.3, 0.4, 0.5]
      },
      "metadata": {
        "genre": "documentary",
        "year": 2019
      }
    }
  },
  "namespace": "example-namespace"
}

Query

The Query API allows you to find vectors similar to the input vector. You can perform the search operation at the namespace level, or use Metadata for filtering during the search. This enables you to narrow down the search results and retrieve the most relevant vectors.

$ curl --request POST \
     --url https://index_name-project_id.svc.environment.pinecone.io/query \
     --header 'accept: application/json' \
     --header 'content-type: application/json' \
     --data '
{
  "includeValues": "false",
  "includeMetadata": "false"
}
'

{
  "matches": [
    {
      "id": "example-vector-1",
      "score": 0.08,
      "values": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8],
      "sparseValues": {
        "indices": [1, 312, 822, 14, 980],
        "values": [0.1, 0.2, 0.3, 0.4, 0.5]
      },
      "metadata": {
        "genre": "documentary",
        "year": 2019
      }
    }
  ],
  "namespace": "string"
}

Upsert

The Upsert API is used to add or update vectors. An upsert operation performs a full update, replacing the existing vector with the new vector if the ID matches.

$ curl --request POST \
     --url https://index_name-project_id.svc.environment.pinecone.io/vectors/upsert \
     --header 'accept: application/json' \
     --header 'content-type: application/json' \
     --data '
{
  "vectors": [
    {
      "id": "example-vector",
      "values": [
        1,
        2,
        3
      ]
    }
  ]
}
'

{
  "upsertedCount": 1
}

Update

The Update API allows you to partially update vectors. You can specify separate update values for the vector and the Metadata, providing greater flexibility in updating your vector database.

Delete

The Delete API is used to remove vectors from the index. Similar to the Query operation, you can delete vectors at the namespace level or use Metadata for filtering during the deletion.

References

https://docs.pinecone.io/docs/organizations
https://docs.pinecone.io/docs/projects
https://docs.pinecone.io/docs/indexes
https://docs.pinecone.io/reference/describe_index_stats_post