Large Language Model Applications

Setting Up a Practical LLM Application

Most people building Large Language Model Applications jump straight into code without thinking about latency, cost, or what happens when things go wrong. I spent about eight months working on an internal document analysis tool before I had something that didn't embarrass me in front of users. The tool took PDFs, extracted text, and summarized them based on custom prompts. It worked well enough, but only after a lot of incremental fixing. The first decision you need to make is which model you're actually going to use. GPT-4 is expensive at around $30 per million input tokens if you're using the GPT-4o variant, and that adds up fast if you're processing thousands of documents daily. The cheaper options like GPT-4o-mini or Claude Haiku run roughly ten times less per token but introduce their own quirks. Claude Haiku sometimes struggles with long contextual instructions and drops details from the middle of a prompt entirely. GPT-4o-mini produces faster results but has a shorter context window at 128K compared to GPT-4o's 128K as well, though the practical performance differs in nuanced ways during multi-turn conversations. Here is what I learned about architecture. Most people assume they can just pass a raw document into a model and get a clean result. That does not work well in practice. You need a preprocessing pipeline that splits documents into logical chunks, embeds them if you are doing retrieval, and routes each chunk through the model in a controlled manner. I built a system using LangChain for orchestration and Pinecone for vector storage. The embeddings came from OpenAI's text-embedding-3-small model, which costs about $0.02 per million tokens and produces 1536-dimensional vectors. The retrieval step used a similarity threshold of 0.75 cosine similarity, which filtered out most irrelevant chunks without requiring complex re-ranking.

The workflow I settled on involved three stages. First, ingest the document and split it into 500-token chunks with a 50-token overlap to prevent losing context across boundaries. Second, embed each chunk and store it in the vector database with metadata tags for document type and date. Third, during querying, retrieve the top 10 chunks by similarity score, pass them to the model with a system prompt that specifies exactly what format the output should take, and combine the results in a second pass where the model synthesizes the individual chunk answers into a coherent summary. I ran into a specific problem with my document analysis tool that took weeks to resolve. The model kept hallucinating citations that looked plausible but did not exist in the source documents. This happened because the model was generating content based on patterns it learned during training rather than strictly extracting from the retrieved chunks. The fix was straightforward but required adjusting multiple parameters. I set the temperature to 0.1, added explicit instructions in the system prompt telling the model to only use information present in the retrieved context, and implemented a post-processing validation step that cross-referenced every citation in the output against the actual chunk text. If a citation could not be verified, it was removed. This cut the hallucination rate from roughly 40% of outputs down to under 5%. Evaluation is another area where most projects fall apart. You need to track latency, cost per query, accuracy against a ground truth set, and user satisfaction scores. I used a held-out test set of 200 documents with expert-generated summaries and compared the model output against them using ROUGE-L scores and human evaluation. The ROUGE-L scores ranged from 0.62 to 0.78 depending on the document complexity. Human evaluators rated the outputs as acceptable 82% of the time, but that 18% of unacceptable results were concentrated in documents with highly technical content and domain-specific jargon. The model performed significantly worse on legal and medical documents compared to general business content, which is worth noting if your application domain involves those areas.

Cost management requires ongoing attention. I tracked spend using a combination of API usage dashboards and a simple billing script that aggregated costs by document type and query complexity. The initial version of the system cost about $0.85 per document end-to-end. After optimizing the chunking strategy, reducing the number of retrieved chunks from 10 to 6, and switching the final synthesis step to a cheaper model, the cost dropped to roughly $0.32 per document. That is a 62% reduction without a measurable drop in quality based on the evaluation metrics. There are several approaches to deploying LLM applications and each has trade-offs. A cloud API approach using OpenAI or Anthropic is the fastest to set up but gives you the least control over latency and data privacy. Running open-source models like Llama 3 or Mistral on your own infrastructure through tools like vLLM or Ollama provides more control but requires GPU resources and ongoing maintenance. A hybrid approach where you use cloud APIs for complex reasoning tasks and local models for simpler extraction or classification tasks can balance cost and capability. I ended up using a hybrid setup where routine document extraction ran locally on a single A10G GPU and the synthesis and summarization steps went to GPT-4o-mini through the API. The biggest mistakes I see people make when building these systems involve underestimating the preprocessing requirements, skipping proper evaluation, and not planning for failure modes. Models will produce incorrect results. They will occasionally refuse to answer. They will generate outputs that are just barely wrong enough to be convincing. Having fallback logic, confidence scoring, and a manual review process for high-stakes outputs is not optional. I recommend implementing a simple confidence metric based on the retrieval similarity scores and the model's own uncertainty signals. When confidence falls below a threshold, route the request to human review instead of trusting the automated output.

If you want to start building, the most practical entry point is the OpenAI Python client or the Anthropic SDK. Both have straightforward integration patterns and good documentation. For orchestration, LangChain and LlamaIndex are the two main frameworks. LangChain has broader community support and more built-in components. LlamaIndex is more focused on retrieval workflows and tends to require less configuration for document-based applications. Choose based on what you are actually trying to build rather than which one sounds more impressive. The code base for a functional prototype can be under 500 lines if you keep the scope reasonable and resist the urge to over-engineer the architecture before you know what works.