Tracker For Ai Diy

Building Your Own AI Output Tracker

You want to track what your local AI models are doing — prompts, responses, timing, maybe token usage. Commercial observability platforms cost money and send your data somewhere you don't control. The DIY route is usually a SQLite database, a lightweight Python middleware, and some patience. The basic approach is to sit between your application and your inference endpoint. If you are running Ollama, Mistral, or a similar local server, your app talks to that server. Your tracker sits as a proxy or wrapper, logs everything, and passes requests through. The simplest version I have seen work reliably uses FastAPI as a reverse proxy with an async background task writing to SQLite. Request comes in, log starts, response streams back, log finishes, done. A typical setup like this handles about 50-100 concurrent request logs without breaking a sweat on a standard laptop.

Tracker For Ai Diy Setup Basics

Here is what the actual code structure looks like in practice, stripped down: A single Python file with FastAPI. One endpoint that proxies to localhost:11434 (Ollama default). Before calling the upstream model, insert a row into a requests table with prompt text, model name, timestamp, and request ID. After the response comes back, update that row with response text, token counts, and latency. That is it. The schema is maybe eight columns. I have seen people overcomplicate this with Elasticsearch or TimescaleDB when a single SQLite file would do everything they actually need. The specific problem I ran into a few months ago was with streaming responses. Most local AI servers support streaming — they send tokens one at a time as they generate. My first tracker implementation buffered the entire response in memory before writing to the database. When someone asked the model to write a long story or run code, the request would pile up several megabytes of text before hitting the log. This meant your tracking INSERT statement was blocking the actual response latency. The workaround was to write the request row immediately, then update it incrementally as chunks arrived, flushing the final response text at the end of the stream. This cut my worst-case logging overhead from about 400ms down to under 50ms.

For token counting, do not rely on the model's own reported usage if you are doing anything with chat history or repeated calls. Some APIs undercount system prompts or truncate the usage field. I ended up using tiktoken for OpenAI-compatible models and a custom byte-pair counter for Llama-based models. The difference was noticeable — my first week of tracking showed 12 percent lower token counts than the raw math confirmed. Fixing the counter aligned everything.

What People Miss About Local AI Tracking

Two things that are not obvious until you actually run this for a few weeks. First, logging raw response text will fill your database fast. A single GPT-level reasoning model can output three to eight thousand tokens per response. At ten requests a day, that is roughly forty thousand tokens daily in plain text. SQLite handles this fine for months, but your queries start getting slow if you are scanning text fields. The fix is to store the response text in a separate table keyed by request ID, or compress it. I use simple gzip compression on the response column — a 4000-token response drops from about twelve kilobytes to roughly three. Query speed stays the same, storage usage drops by seventy percent. Second, tracking embedding models is different from tracking text completion models. Embeddings are dense vectors, not readable text. Your tracker needs a separate column type and a different search strategy if you want to find similar past requests. I added a vector similarity search using numpy dot products for my own setup. It is not fast at scale — more than a few thousand logged embeddings and you want something like sqlite-vec or a dedicated vector database. But for under five hundred tracked requests, a simple numpy lookup runs in about two hundred milliseconds on a normal CPU. That is good enough for personal use and free.

When This Approach Fails

A few honest limitations worth knowing before you build this. Proxy-based tracking only captures requests that go through your tracker. If you test prompts directly in the Ollama web UI or through a different client, those requests are invisible to your database. You need to make sure every tool you use routes through the proxy. This is usually just a matter of changing your application configuration to point at localhost:8080 instead of localhost:11434, but it is easy to forget. Another failure mode is sensitive data. If your AI is processing anything that includes personal information, API keys, or proprietary code, that data lives in your SQLite file unencrypted. Anyone with file access can read it. I encrypt the response column with Fernet tokens and store the key separately. It adds about five milliseconds per request but keeps the data safe. Do not skip this if you are tracking real work, not toy prompts.

Finally, if your goal is production-grade observability with alerting, dashboards, and team access, DIY tracking hits a wall around fifty concurrent users. SQLite lock contention becomes real. At that point, the migration path is straightforward — the schema I described works identically on PostgreSQL with zero changes to the application logic. Just swap the database driver and you are on TimescaleDB or plain Postgres. The investment pays off quickly if you ever need to scale.

Resources

There is no single official release for a DIY AI tracker since this is inherently a custom build. The nearest starting points are Ollama's built-in logging flags, which only capture server-side access logs and do not store prompts alongside responses in a searchable format. For a complete tracker, you are writing your own proxy. The core dependencies are FastAPI, httpx for the proxy requests, SQLite with the aiosqlite driver for async writes, and tiktoken for token counting. A basic working version takes about two hours to assemble and test end to end. More complex features like streaming support, encryption, and vector search add another four to six hours depending on your familiarity with the stack. The GitHub space around this has a few partial implementations. Most are missing streaming support or token accuracy. Building from scratch using the approach above gives you something that actually works for daily use without the technical debt of adapting someone else's incomplete proxy.