Language Detector And Translation

Building a Language Detection and Translation Pipeline That Doesn't Suck

Most people think language detection and translation is just calling an API and you're done. It's not that simple. I spent about six months debugging a pipeline that kept misidentifying Spanish as Portuguese on customer support tickets, and honestly the issue was way more boring than anyone expects.

Let me walk through how this actually works when you're building it for production, not a weekend demo. At the core, you're dealing with two separate problems that get glued together. Language detection figures out what tongue you're reading. Translation converts it to something your users understand. The glue is usually where things fall apart. For detection, you have three real paths: n-gram statistical models, character-level neural networks, or using the translation API's built-in detection. The n-gram approach with something like fastText from Meta runs locally, processes a thousand documents per second on a single CPU core, and costs you exactly nothing after you download the model. The neural approaches are more accurate on short text but they need GPU memory and they're slower. Translation APIs like Google's or DeepL give you detection for free if you just send the raw text, but you're locked into their ecosystem and you're paying per character.

I went with fastText for detection and DeepL for translation. The split decision came down to cost at scale and accuracy on domain-specific text. Here's why that combination matters. FastText's pre-trained models come in sizes from 96 languages down to 108. The md model is about 750MB and hits roughly 97% accuracy on text longer than 20 characters. Below that, accuracy drops to somewhere around 80% and that's where my problem started. Customer messages like "ok thanks" or "help" get flagged as English sometimes and Portuguese other times depending on what the rest of the batch looks like. The workaround was setting a confidence threshold at 0.85 and routing anything below it through a secondary check using langdetect, which is the Python port of Google's language-detection library. Both libraries use different algorithms, so combining them as a voting system pushes accuracy on short text up to about 94%. It added roughly 3 milliseconds per document to the pipeline, which was acceptable. For translation, DeepL beats Google on quality for European languages and Russian, but it's worse on Southeast Asian languages and it doesn't support Arabic well. If your user base is global, you need a fallback. I route Chinese, Japanese, and Korean through Google Translate instead of DeepL. The code switch is about ten lines using a language code lookup table.

The detection step matters more than people realize for translation quality. Send French text to a model that thinks it's Italian and you get garbage. I've seen teams skip confidence scoring entirely and just take the top result from the detector. That's how you get 3% of your translations that are completely wrong and you don't notice until a customer complains about a refund that was never processed because the original message was in Romanian and got misread as Romanian-inspired nonsense.

Implementation Details Most Guides Skip

Batching is where you save money and time. Detection and translation APIs charge per request or per character. Sending one document at a time through DeepL's API at their standard pricing runs about $25 per million characters. Batching fifty documents together drops that to roughly $15 per million because you're reducing HTTP overhead and hitting volume tiers. With fastText running locally, batching doesn't affect cost but it does affect throughput. Processing 50 documents in a single numpy operation on the character embeddings is about 4x faster than looping through them one by one. Caching translations is non-negotiable. Same source text plus same target language plus same domain will produce the same output 99 times out of 100. I store cached results in Redis with a key made from the MD5 hash of the source text concatenated with the target language code. Hit rate on our support ticket system was around 73% on a good day. That cut our translation costs by nearly three-quarters and reduced average latency from about 800 milliseconds to roughly 12 milliseconds for cached hits. Here's the part nobody talks about: punctuation and formatting preservation. Translation models mangle HTML tags, product SKUs, and email addresses if you don't handle them separately. The working approach is to extract placeholders before translation, run the translation on the plain text, then inject the original values back. A regex pass for patterns like [A-Z]{2,10}-\d{4,8} catches most product codes. HTML tags get swapped for tokens, translated, then restored. This adds maybe 5 milliseconds per document but prevents the kind of broken output where a price like €19.99 turns into something unreadable because the model tried to translate the euro symbol.

Handling right-to-left scripts requires explicit direction. UTF-8 encoding alone doesn't solve it. You need to set the dir attribute on container elements to rtl and make sure your CSS flexbox or grid layouts don't break when text direction flips. I learned this the hard way when Arabic translations rendered left-to-right and the punctuation ended up on the wrong side of the sentence, which confused about 12% of our Middle Eastern users enough to submit duplicate tickets.

When This Approach Fails Completely

Automatic detection and translation breaks down on code-mixed text, which is extremely common in multilingual markets. A message like "Hey, the delivery took forever pero al menos llegó bien" mixes English and Spanish in ways that no detector handles cleanly. fastText will pick one language and the translation model will produce uneven output. The fix is manual review queues for low-confidence detections combined with mixed-language patterns, but that requires human-in-the-loop infrastructure you might not want to build. Dialect variation is another blind spot. fastText labels all Arabic as ar and then DeepL translates based on Modern Standard Arabic. If your user is writing in Iraqi dialect or Levantine Arabic, the translation will be technically correct but feel stiff and unfamiliar. The same issue shows up with Brazilian Portuguese versus European Portuguese. Google handles this slightly better with regional variants but DeepL treats them as the same language. If you need something more robust than the pipeline I described, the alternative is running a fine-tuned multilingual model like mBART or NLLB-200 locally. These handle some of the code-mixing and dialect issues better because they're trained on parallel corpora that include those variations. The tradeoff is you need a GPU with at least 8GB VRAM and inference takes about 200 milliseconds per document instead of 12 milliseconds from cache. For a small team processing a few hundred documents daily, the GPU cost might justify the quality gain. For high-volume production, the fastText plus DeepL plus caching approach hits the sweet spot between cost and accuracy.

The whole setup runs on a single t3.medium EC2 instance for the detection and caching layer, with API calls going outbound to DeepL and Google. Monthly cost comes to about $45 for compute plus whatever the translation API bills run. On our traffic volume that translates to roughly $0.002 per translated document after caching. Anything significantly under that usually means someone is cutting corners on detection quality or not caching at all.