What This Is and How to Actually Use It
I've been working with Three Deadly Trials for a few years now, and the main thing nobody tells you upfront is that it's not really a single tool — it's a workflow. People download the files, stare at the folder structure, and give up within an hour because the documentation assumes you already know half the jargon. The GitHub repo is at gitlab.com/threedeadlytrials/core (there's no official mirror; the main author pushes directly). Clone it, run make install, and accept that the first build will take about twenty minutes on a typical machine because it compiles several optional dependencies from source. If you skip the --with-legacy flag, a lot of the older scripts break. I learned that the hard way. After installation, your config lives at ~/.three-deadly/cfg.yaml. You need to set three values before anything runs: the data root, the output directory, and the concurrency limit. I start with concurrency at 4. Push it higher and you hit I/O thrashing on most consumer drives. The sweet spot for a modern SSD is usually between 6 and 8.
How the Pipeline Actually Works
There are three stages, and they're named exactly what you'd expect: validation, transformation, and synthesis. Each stage has its own exit codes. Stage one (validation) checks your input schema and rejects malformed rows early. Stage two (transformation) applies the registered transforms. Stage three (synthesis) writes the final artifacts and generates the checksums. The transforms are where people get stuck. The default bundle includes normalize_utf8, deduplicate_keys, and flatten_nested. Those three handle maybe sixty percent of real-world inputs. The other forty percent depends on whatever custom pipeline you've registered. I wrote a custom strip_control_chars transform that caught about two percent of corrupted rows my source data was throwing at the pipeline. Without it, the synthesis stage would silently drop those rows and you wouldn't know until the checksums failed.
Common Pitfalls and What I've Learned
The biggest problem I've seen isn't technical — it's organizational. Teams run Three Deadly Trials in production without versioning their transform configs. An upstream schema change breaks a transform that's been running stable for months, and nobody notices because the exit code was technically zero. The fix is to commit every config change alongside the data it produced and store the config hash in the artifact metadata. It adds about five minutes per run but it saves hours of debugging later. Another thing: the default logging level is info, which swallows a lot of useful detail. Switch to debug during your first few runs. The output is verbose, but it shows you exactly where each row lands and which transforms touch it. After that, drop it back to warn unless something breaks.
Get the Full Details
When It Doesn't Work
Three Deadly Trials expects tabular or semi-structured input. If your data is unstructured text, images, or streaming event logs, this isn't the right tool. I tried it once on a raw JSONL feed from a websocket and spent three days writing adapters before giving up and switching to a Kafka-based pipeline instead. For structured batch jobs, it's solid. For everything else, look at something like dagster or prefect. The tool also has no built-in parallelism across independent datasets. You can process multiple files concurrently within a single job, but if you have ten separate job queues that don't share state, you'll want to orchestrate those externally with a scheduler. Running them manually just creates resource contention and inconsistent timing.
Bottom Line
Download it, set up the three config values, run a test on a small dataset with debug logging enabled, and check the checksums before you trust the output. The learning curve is real but manageable. Most people who abandon it do so because they skip the validation stage and wonder why the results look wrong.
