Minimalist Ai Tracker

The thing most people get wrong about light-weight AI monitoring

I spent about three months building something that tracks AI inference latency, token throughput, and cost-per-million-tokens without requiring a Grafana dashboard or a five-service microservice mesh. The result was Minimalist Ai Tracker, which is just a single Python package that ships around 400 lines of actual logic and wraps your model calls in lightweight instrumentation. The whole idea started because I was tired of watching my team's OpenAI API bills double every quarter with no visibility into which prompts, which models, or which code paths were responsible. Existing tools either required Kubernetes operators or forced you to instrument every single call site with verbose decorators. Neither option works when you're running a small team on a few EC2 instances.

What Minimalist Ai Tracker actually does

It instruments OpenAI-compatible calls, Anthropic SDK calls, and HuggingFace transformers pipeline inference, collecting structured metrics at the function level. You get per-request latency histograms, token count tallies, error rates broken down by error type, and a cumulative cost estimate based on per-model pricing. It writes to stdout by default and can pipe to a local SQLite file or a Prometheus endpoint if you want. The installation is standard. pip install minimalist-ai-tracker and then you wrap your client initialization. Here is the typical setup for an OpenAI call. Import the tracker, instantiate it with your model list and pricing tiers, wrap the client object, and run as normal. The wrapper intercepts each request and response, extracts token counts from the usage field, computes cost using the rate card you provided, and emits a structured log line. That line looks something like: {"ts": "2025-03-12T14:22:01Z", "model": "gpt-4o-mini", "latency_ms": 342, "input_tokens": 1204, "output_tokens": 89, "cost_usd": 0.00041, "status": "ok"}.

For HuggingFace pipelines, the tracker hooks into the model's forward pass and captures generation time plus input and output token lengths. This is where things get interesting because transformers doesn't always return clean usage dictionaries, so the tracker falls back to counting tokens from the generated text length and the tokenizer's output length. That fallback is usually within 5 percent of the actual value, which is good enough for cost estimation but not for hard billing audits.

How to get started without overcomplicating it

Create a configuration file. Most people skip this and use defaults, which works fine until you need custom pricing. The config supports model overrides, custom endpoints for local deployments, and quiet mode for production environments where logging volume matters. I recommend setting up the config file early even if you just leave it mostly empty. It saves you from having to refactor code later. Wrap your client. If you are using the OpenAI Python SDK version 1.0 or above, the tracker provides a context manager. You pass your client object and a metrics handler. The handler is just a class with a record method that accepts the structured dict I described. By default you get a console handler, but you can swap in a file handler, a prometheus handler, or a webhook handler that POSTs to an internal dashboard. Run a test. Send ten requests to your model with varying prompt lengths. Check the logs. Verify that latency numbers match what you see in your own timer. Verify that token counts are sane. If the input tokens look wrong, check whether your provider is returning usage data. Some providers omit it for streaming responses. The tracker handles this by estimating output tokens from the streamed text, but the estimate has a margin of error that compounds over long generations.

Here is a concrete snippet for the OpenAI setup. import os from openai import OpenAI from minimalist_ai_tracker import Tracker, ConsoleHandler client = OpenAI(api_key=os.environ["OPENAI_API_KEY"]) handler = ConsoleHandler() tracker = Tracker(client, handler=handler, model="gpt-4o-mini") with tracker: response = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": "Explain quantum computing"}], max_tokens=256 ) print(response.choices[0].message.content) That produces a log line after each response. The latency includes network round-trip time, so it is not purely model inference time. If you need pure inference latency, you have to wrap the API call differently or use a local proxy that strips network overhead. I built a small nginx-based proxy that adds a X-Inference-Time header, but that is outside the scope of the tracker itself.

Where it gets messy in practice

I ran into a real problem with batch processing. When you send multiple completions in a single API call, the OpenAI SDK returns one usage object for the entire batch. The tracker was attributing the full latency and cost to the first item in the batch and ignoring the rest. That meant my cost estimates for a data pipeline processing ten thousand samples at once were off by roughly a factor of ten. I fixed it by adding a batch-aware mode that splits the usage object proportionally across batch items based on input token distribution. It is not perfect because the API does not return per-item latencies, but it is close enough. Another issue is streaming. The tracker can instrument streaming responses, but it needs to accumulate the full stream before it can report accurate token counts. This means the first log line you see after a stream completes is the complete record. If you are watching logs in real time expecting per-chunk metrics, you will be disappointed. The tracker intentionally does not emit partial records because partial metrics are misleading. A chunk that never finishes due to a timeout should not be counted as successful usage. There is also the question of local models. If you are running a quantized LLaMA model on a GPU with vLLM or TGI, the tracker can wrap the HTTP client that talks to your local endpoint. But if you are calling the model directly through the vLLM Python API, the tracker has no hook point unless you add instrumentation to the serving layer. I wrote a small middleware for vLLM that patches the completion endpoint and emits tracker-formatted logs. It works, but it requires you to run a custom vLLM image with the middleware pre-installed. That is a deployment decision, not a configuration change.

When Minimalist Ai Tracker is the right call

Use it when you need cheap, low-friction observability for a small AI service. It costs nothing except the logging overhead, which is negligible. It adds roughly 0.3 milliseconds per request due to the instrumentation dict construction and JSON serialization. For a high-throughput system doing ten thousand requests per second, that adds up to three seconds of CPU time per second, which is measurable but usually acceptable. Do not use it when you need sub-millisecond precision on inference timing, when you are processing batches at massive scale and need per-item granularity, or when your infrastructure already has a full observability stack. In those cases, a proper APM tool will serve you better. The tracker is not trying to replace Datadog or OpenTelemetry. It is trying to give you something that works when those tools are overkill. The pricing model is another limitation. The tracker uses static rate tables for popular models. If you are on a custom enterprise contract with OpenAI or Anthropic, your actual per-token cost may differ from what the tracker reports. I built in a config override for custom pricing, but most users do not know about it. Check the documentation if your bill does not match the tracker output.

A practical pitfall beginners miss

Token counting is not the same across providers. The tracker uses tiktoken for OpenAI models, which is the correct tokenizer for GPT families. For other models, it falls back to a character-based heuristic or the model's own tokenizer if available. This means your token counts for a non-OpenAI model may be off by ten to twenty percent. The cost estimates will reflect that discrepancy. If you need accurate token accounting for a non-standard model, you should either provide your own tokenizer in the tracker config or validate the counts against the provider's dashboard for a sample of traffic. Another thing people overlook is the difference between cached and uncached tokens. OpenAI's context caching feature reduces cost for repeated prefixes, but the tracker does not automatically detect cache hits. If your application sends the same system prompt and conversation history repeatedly, your actual costs will be lower than what the tracker reports. You can manually adjust the config to apply a cache discount factor, but most people do not. This is fine for relative comparison between models or prompts, but it will make your absolute cost numbers slightly inflated. If you want the source code, the project is on GitHub under the MIT license. The README has installation instructions, configuration reference, and examples for OpenAI, Anthropic, and HuggingFace. There is also a section on the vLLM middleware I mentioned. The repository is actively maintained, and issues get answered within a day or two on weekdays. I check the repo myself occasionally because I still use it in production for a few internal services.

The bottom line is that Minimalist Ai Tracker solves a narrow problem well. It gives you visibility into AI spending and performance without the bloat of enterprise observability tools. It is not perfect. It will not replace a full metrics pipeline. But for a small team or a solo developer who needs to know whether that new prompt is costing more than the old one, it does exactly what it claims and nothing more.