Makeshift Hash Pipe

Building an Improvised Hash Processing Pipeline

I ran into this problem about two years ago while dealing with a bulk file integrity check on a Linux box that had zero free space for installing extra tools. I needed to hash about forty thousand files and compare them against a reference manifest, but the environment only had the usual busybox and coreutils. So I built something out of a pipe, some awk scripts, and a small inline Python helper. People who've done enough incident response work have probably done the same thing. The results were good enough for triage, even if they weren't pretty. The basic idea is straightforward. You take a list of file paths, run each through a hash function, and stream the output somewhere you can process it. On a normal system you'd use a tool like md5deep or vhash. On a stripped-down or constrained system you just connect shell primitives with pipes and let the OS handle the buffering. That's what I mean by a makeshift hash pipe. It's not a brand name. It's a description of the approach.

What a Makeshift Hash Pipe Actually Is

A makeshift hash pipe is an ad-hoc pipeline that hashes files using whatever tools are available in the environment and passes the results through stdout or a temporary file for comparison. There is no single implementation. The pattern shows up whenever someone needs file hashing without the right binary. It typically involves a loop over paths, a hash command per file or batch, and output formatted as hash_value filename so it matches the standard checksum output format that most diff tools expect. The reason this pattern persists is that many forensics and compliance workflows require SHA-256 or MD5 inventories, but the machines you end up working on are often locked down, minimal, or remotely accessed through a thin shell. You can't just install the fancy tools. You work with what's there. That constraint is what makes the makeshift version useful rather than just a hack.

How I Build One When Time Is Short

My usual starting point is a bash script that reads paths from stdin or a file and runs each through the fastest available hash tool. Here is roughly the skeleton I fall back to: while IFS= read -r f; do
[ -f "$f" ] || continue
shasum -a 256 "$f" 2>/dev/null || sha256sum "$f" 2>/dev/null || md5sum "$f"
done This handles the common case where the target system has GNU coreutils, OpenSSL command-line tools, or Python's hashlib available. The pipeline part comes when you chain it to other operations. I usually append something like | sort -k2 | uniq -f1 --all-checksums | wc -l to count how many unique hashes I produced, or pipe it into an awk script that flags files missing from a baseline manifest. The whole thing runs in a single foreground session and exits when the input is exhausted.

For larger jobs I switch to a Python generator because Python's hashlib module is almost always present and it avoids forking a new process per file. The throughput difference is noticeable. On a typical job with about twenty thousand files, the pure bash version with separate sha256sum invocations takes roughly forty minutes on an SSD. The Python generator version using a single interpreter process finishes in about twelve minutes on the same dataset. The gap widens when you're hashing from HDDs or network mounts because process fork overhead dominates. One thing I learned the hard way is that error handling matters more than the hash algorithm itself. I once ran a makeshift hash pipe over a directory tree that contained a few broken symlinks and one FIFO special file. The script hung for about twenty minutes on the FIFO before I killed it. The fix was adding a [ -f "$f" ] guard before calling the hash command, and wrapping the call in a timeout. OnBusyBox systems without timeout, I use a small background coprocess or just rely on the fact that most hash implementations return quickly on non-regular files. That guard check alone saves you from the common class of "pipeline hung on a socket or device node" mistakes.

Common Pitfalls That Catch People Out

The first issue is encoding. If your file paths contain non-ASCII characters, sha256sum and md5sum output them literally in the checksum filename field, which breaks most downstream parsers that assume pure ASCII. I handle this by running the output through iconv -f utf-8 -t ascii//translit when I need clean output, or by simply quoting the filename field and telling whoever receives the data to handle UTF-8. Neither option is ideal, but picking one consistently prevents half the bugs I see in practice. The second issue is newline handling in paths. Filenames with newlines are rare but legal on most Unix filesystems, and they completely break any pipeline that uses newline-delimited line counting. The workaround is to use null-delimited I/O throughout: find ... -print0 | while IFS= read -r -d '' f; do ...; done. This adds about ten percent overhead because of the extra shell complexity, but it prevents the rare case where a corrupted or deliberately crafted filename causes you to miscount or miss files entirely. I count maybe one in every three hundred incident response jobs where this actually bites me, but when it does, it looks like a data integrity failure until you inspect the paths manually. A third pitfall is not normalizing the output format across different tools. GNU sha256sum outputs binary mode by default with a space separator and optional * prefix for binary mode files. OpenSSL's dgst command uses a different format. Python's hashlib prints uppercase hex by default. If you're comparing results from two different makeshift implementations, the hex strings will match but the surrounding formatting will not, and a naive diff will report everything as different. Always force a single output format. I usually add an awk post-processor that extracts the hex digest and the filename, lowercases the digest, and strips any mode indicator. That takes about five seconds of development time and prevents hours of confusion later.

When a Makeshift Hash Pipe Is the Right Call

The makeshift approach works well when you have limited tooling, a one-time job, or a constrained environment where installing packages is not an option. It also works reasonably well for medium-scale jobs under about fifty thousand files on modern hardware. The main bottleneck is I/O, not the hash computation. SHA-256 on a sequential read from an NVMe drive typically saturates at around two hundred megabytes per second, which means a one-terabyte dataset takes roughly eighty seconds of pure hashing time. The rest of the wall clock is spent on filesystem metadata calls, process creation, and output formatting. The approach breaks down when you need cryptographic-grade certainty about reproducibility across platforms. If you're building a baseline for legal evidence or a compliance audit that must survive cross-platform comparison, you should use a validated tool like HashCalc, Glastopf, or the NIST reference implementation. A makeshift pipe is acceptable for triage and internal comparison on a single system, but it is not a substitute for a validated chain-of-custody tool when the output may be challenged in an audit or proceeding. Another scenario where it fails is when you need recursive hashing with metadata preservation. Makeshift pipes typically only output the hash and the path. They do not capture modification times, ownership, permissions, or extended attributes. If your workflow requires that metadata for change detection, you need a proper inventory tool like pkg or Tripwire, or at minimum a wrapper that runs stat alongside each hash. I sometimes build this wrapper in a single Python script because os.stat and hashlib are both standard library, but it moves the project out of "makeshift" territory and into "custom tool" territory.

What I Do Instead When the Job Gets Serious

If the dataset exceeds about a hundred thousand files, or if I need the output to be reproducible across multiple analysts' machines, I switch to a dedicated tool. My usual fallback is rhash or gdHash, both of which support parallel execution and produce standardized output. For Windows environments I use PDQ Deploy-style scripting with PowerShell's Get-FileHash, which handles the path encoding issues I described earlier. The trade-off is that these tools require installation or at least a known binary to be present, which defeats the purpose of the makeshift approach in the first place. There is also the option of using Docker or a portable runtime to bring a consistent environment with you. I keep a small container image with Python 3, coreutils, and the tools I need for hash verification. When I hit a machine that is too locked down even for that, I fall back to the bare shell pipeline. Most of the time the shell pipeline is enough. The container is my safety net, not my daily driver.

Practical Tips I Actually Use

When building a makeshift hash pipe, I always redirect stderr to a separate log file so that permission-denied errors and read faults do not mix with the checksum output. I pipe stderr through grep -v "Permission denied" when I know some files are expected to be unreadable, but I keep the full log for later review. This habit saved me once when I was diagnosing a false-negative result that turned out to be caused by a read error on a mounted NFS share that had briefly dropped connectivity. The hash output looked clean, but the log told a different story. I also prefer to write the output to a temporary file rather than relying on stdout for large jobs. The reason is that if the pipeline breaks partway through, you lose everything. With a temp file you can resume from the last successful hash by tracking which paths you have already processed. A simple grep -Ff processed.txt input.txt | comm -23 trick does the delta calculation in about three seconds even on a million-line file list. I include this in nearly every makeshift implementation I write now. For parallel execution without external tools, I use GNU parallel when it is available, or a bash background-jobs loop with a fixed job count when it is not. The fixed-count approach uses a simple counter and wait to cap concurrency at about four or eight jobs depending on the number of CPU cores. On a four-core machine this typically cuts runtime by thirty to fifty percent compared to the sequential version, with negligible overhead from job management. I have not seen cases where more than eight parallel workers helped, probably because the bottleneck shifts to disk I/O rather than CPU.

One final note on verification. Always spot-check your makeshift hash pipe output against a known-good run of a validated tool on a small subset of files. I usually hash ten files with both the makeshift pipeline and sha256sum directly, compare the digests, and verify that the formatting is consistent. If the digests match but the formatting differs, I fix the parser before scaling up. If the digests differ, I stop immediately and investigate. A single byte flip in a digest usually means either a different encoding, a truncation bug, or a race condition where the file changed between when you read it and when you hashed it. The last one is more common than you would expect on actively used systems. The makeshift hash pipe is not glamorous. It will not win any awards. But it works when nothing else is available, and it works fast enough for most practical purposes. I have used it on production servers, incident response workstations, and a Raspberry Pi running a headless script to inventory a network share. In every case the pattern was the same: identify the available tools, choose the hash algorithm, format the output consistently, and verify before you trust the result. Everything else is just detail.