Stop Guessing How Much Memory Your AI Needs

Your AI assistants are costing you too much. Every question from a user, every line of code from your coding bot, costs you compute power and memory.

You know you need a cache—a fast memory area that stores past work—to speed things up. But figuring out how much to buy is a complete guessing game.

Buy too little memory, and the AI runs slow. It wastes expensive compute power re-doing work it just did. Buy too much, and you're paying for expensive hardware that just sits there, unused.

This isn't a small problem. It's the difference between a profitable AI service and one that bleeds cash.

What Researchers Discovered

A team of researchers built a tool that ends the guesswork. They figured out how to calculate the exact amount of memory you need to make your AI run fast and cheap.

You can read their full paper here: The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems.

Here’s what they found, in plain terms.

You don't need to save everything. Think of it like a library. You don't need every book you've ever read on your desk. You only need the few you're currently using. The rest can stay on the shelf. The same is true for your AI's "memory." A small, hot set of recent data handles most requests. You can get 95% of the speed benefit with a cache 5x smaller than you might think you need.

Performance gains have a ceiling. Adding more memory helps, but only up to a point. It's like adding monitors to your desk. A second monitor is a huge boost. A tenth monitor does almost nothing. The researchers' tool finds that "sweet spot" where spending more money gives you almost no extra speed.

You can predict the perfect size automatically. Their tool, called KVSET, analyzes real user requests. It tells you the ideal cache size instantly. No more slow, costly manual testing. For a coding agent, they found the memory needed for "good enough" performance (95% coverage) was 5x less than for "near-perfect" (99% coverage). This lets you make a business decision: "Is that last 4% of speed worth 5x the cost?"

How to Apply This Today

You don't have to wait. You can start using these ideas right now to cut costs and plan smarter.

Here are four concrete steps to take this week.

1. Analyze Your Current AI Traffic Logs. The first step is understanding what you're actually asking your AI to do. Export the logs of user prompts and AI responses from the last week. Look for patterns: Are users asking about the same topics? Are coding requests reusing the same files? This data is the fuel for the next steps.

• For example: If you run a customer service bot, export a CSV of all customer questions from the last 7 days. Look for repeats.
• Tool to use: Simple log export from your AI platform (like OpenAI, Anthropic, or vLLM) or your application backend.

2. Use KVSET to Find Your Memory Sweet Spot. Download the open-source KVSET tool from the research paper's repository. Feed it your log data. It will analyze the patterns and tell you the minimum cache size needed to hit performance targets like 90%, 95%, or 99% "hit rates." This number is your new benchmark.

• For example: KVSET might tell you that for your chatbot, a 2 GB cache gets you a 95% hit rate, but to get to 99%, you need 10 GB. Now you have a clear cost/benefit choice.
• Prerequisite: Your engineering team needs basic Python skills to run the tool.

3. Design a Two-Tier Storage Strategy. Stop trying to fit everything into the most expensive memory (like GPU RAM). Use KVSET's output to design a smarter system. Put the small, "hot" dataset it identifies into fast, expensive cache. Put everything else into a larger, slower, cheaper tier (like regular server RAM or even fast disk).

• For example: For a team of 50 developers using a coding assistant, you might provision 500 GB of fast GPU cache for active files and 2 TB of slower SSD storage for the complete project history.
• How to implement: Configure your AI serving system (like vLLM or SGLang) to use a hierarchical cache.

4. Make Cache Size a Business Metric, Not a Tech Guess. Bring the numbers KVSET gives you into your planning meetings. Frame decisions in business terms: "We can double our user capacity for the same cost if we accept a 95% hit rate instead of 99%." This turns infrastructure from a black-box cost center into a lever you can pull.

• For example: Present to leadership: "Our analysis shows moving from a 99% to a 95% performance target reduces our monthly memory cost from $50k to $10k, with a negligible impact on user wait times."

What to Watch Out For

This approach is powerful, but it's not magic. Keep these limitations in mind.

It works with common, but not all, systems. The KVSET tool currently models caches that use the "Least Recently Used" (LRU) eviction policy. This is very common, but if your system uses a different method, the predictions might be less accurate.

It's for planning, not real-time control. KVSET is an analysis and prediction tool. It tells you how much cache you should have. It doesn't automatically resize your cache live as traffic changes. You or your team would need to build that automation on top of its recommendations.

It focuses on today's mainstream AI models. The research analyzed the standard "Transformer" architecture used by most models. Newer, more efficient model types (like Mamba) work differently. If you adopt those, you'll need to re-run your analysis.

Your Next Move

This week, start with step one. Export one week's worth of logs from your most costly AI service. See what the conversation patterns look like.

That single action moves you from guessing to measuring. It's the foundation for every cost-saving decision that follows.

Are you already tracking how often your AI repeats work? What's one cost you could cut if you knew the exact right size for your systems?