📝 Guest Post: Caching LLM Queries for Improved Performance and Cost Savings*

If you're looking for a way to improve the performance of your large language model (LLM) application while reducing costs, consider utilizing a semantic cache to store LLM responses. By caching LLM responses, you can significantly reduce retrieval times, lower API call expenses, and enhance scalability. Additionally, you can customize and monitor the cache's performance to optimize it for greater efficiency. In this guest post, Chris Churilo from Zilliz introduces GPTCache, an open-source semantic cache designed for storing LLM responses. Read on to discover how caching LLM queries can help you achieve better performance and cost savings, as well as some tips for implementing GPTCache effectively.

Why Use a Semantic Cache for Storing LLMs?

By developing a semantic cache for storing LLM (Large Language Model) responses, you can experience various advantages, such as:

- Enhanced performance: Storing LLM responses in a cache can significantly reduce response retrieval time, mainly when the response is already present from a previous request. Utilizing a cache for LLM responses can enhance your application's overall performance.

- Lower expenses: Typically, LLM services charge fees based on the number of requests and token count. Caching LLM responses can reduce the number of API calls to the service, leading to cost savings. Caching is especially valuable when dealing with high traffic levels, where API call expenses can be significant.

- Improved scalability: Caching LLM responses can increase your application's scalability by reducing the load on the LLM service. Caching helps prevent bottlenecks and ensures that your application can handle a growing number of requests.

- Customization: A semantic cache can be tailored to store responses based on specific requirements, such as input type, output format, or response length. This customization can optimize the cache and improve its efficiency.

- Reduced network latency: A semantic cache closer to the user can reduce the time required to retrieve data from the LLM service. By minimizing network latency, you can enhance the overall user experience.

Therefore, building a semantic cache for storing LLM responses can provide improved performance, reduced expenses, enhanced scalability, customization, and reduced network latency.

What is GPTCache?

GPTCache is an open-source solution created to enhance the speed and effectiveness of GPT-powered applications by implementing a cache system to store language model responses. This tool was inspired by our own needs for a semantic cache when we were building the OSS Chat application - an LLM application that provides a chatbot interface for users to get technical knowledge about their favorite open-source projects. GPTCache allows users to tailor the cache to their requirements with features such as embedding functions, similarity evaluation functions, storage location, and eviction options. Moreover, GPTCache supports the OpenAI ChatGPT interface and the LangChain interface.

Supported Embeddings

GPTCache offers various options for extracting embeddings from requests for similarity search. Additionally, the tool provides a flexible interface that supports multiple embedding APIs, enabling users to select the one that suits their requirements. The supported list of embedding APIs includes the following:

• OpenAI embedding API

• ONNX with the GPTCache/paraphrase-albert-onnx model

• Hugging Face embedding API

• Cohere embedding API

• fastText embedding API

• SentenceTransformers embedding API

GPTCache provides users with various embedding function options that can impact the accuracy and efficiency of the similarity search feature. In addition, GPTCache aims to offer flexibility and accommodate a broader range of use cases by supporting multiple APIs.

Cache Storage and Vector Store

GPTCache offers a variety of features to enhance the efficiency of GPT-based applications. The Cache Storage module supports multiple popular databases, including SQLite, PostgreSQL, MySQL, MariaDB, SQL Server, and Oracle, allowing users to choose the database that best suits their needs.

Additionally, the Vector Store module provides a user-friendly interface for finding the K most similar requests based on extracted embeddings. Milvus, Zilliz Cloud, and FAISS are some of the vector stores supported by GPTCache.

A Cache Manager controls the Cache Storage and Vector Store modules, and users can choose between LRU (Least Recently Used) and FIFO (First In, First Out) eviction policies when the cache becomes full.

Similarity Evaluator

And Finally, Similarity Evaluator. The Similarity Evaluator module determines the similarity between input and cached requests and offers a range of similarity strategies to match different use cases. Overall, GPTCache is an open-source project that offers a variety of features to optimize the use of language models.

In Summary

GPTCache aims to enhance the efficiency of language models in GPT-based applications by reducing the need to generate responses from scratch repeatedly. It achieves this by utilizing cached responses whenever possible. GPTCache is an open-source project, and we welcome you to explore it independently. Your feedback is valuable, and you can also contribute to the project if you wish.

---

*This post was written by Chris Churilo. At Zilliz, she has an integral role in building and growing the Milvus community, creating educational content and resources, and collaborating with users to improve the database. We thank Zilliz for their ongoing support of TheSequence.