Dig Dig Io

My Experience With Dig Dig Io

I keep hearing questions about Dig Dig Io around forums and comment sections, and honestly the thing that comes up most is that people don't actually know what it does or whether it's worth setting up. I've used it over the past couple of months on a few projects and I think I can help clear things up. Let me explain how it works in practice, where it's genuinely useful, and where you should probably walk away. Dig Dig Io is a CLI-focused tool designed around extracting, decoding, and organizing data from structured sources — mostly web-based feeds, API endpoints, and occasionally local files. It's not a GUI app. If you're expecting clickable buttons and a settings panel, you'll be disappointed. You interact with it through a terminal. There's also a companion website where you can inspect your queries and view cached results, but the heavy lifting happens on your machine. The core idea is simple: you define a source, you tell it what fields you want, and it fetches, parses, and outputs them in whatever format you specify. JSON by default, but CSV and NDJSON are dead simple too. It supports pagination, headers, auth tokens, and some basic rate limiting controls out of the box. The configuration lives in a YAML file and a few environment variables. That's it. No databases, no cron jobs required unless you want them.

How It Works in Practice

I'll walk through a real scenario. Let's say you want to pull product pricing data from an e-commerce API every day and track changes over time. Here's roughly what the workflow looks like. First, you install it. On macOS I used brew install dig-dig-io. On Ubuntu I built from source because the binary distribution was missing a dependency for my kernel version. That took about 8 minutes total. Once installed, you create a config file. I keep mine at ~/.dig-dig/config.yaml. The structure is straightforward:

sources:
  - name: product-prices
    url: https://api.example-store.com/products
    method: GET
    headers:
      Authorization: "Bearer $API_TOKEN"
      Accept: "application/json"
    pagination:
      type: cursor
      field: next_cursor
      limit: 100
    output:
      format: json
      path: ./data/product-prices.json
      append: true
    schedule: "0 6 * * *"

You set your API token in an environment variable or a separate secrets file — never in the config directly if you're sharing or version-controlling anything. Then you run it with dig-dig-io run product-prices. The first run creates the output directory and writes the initial dump. Subsequent runs append based on the schedule you defined. For tracking price changes, I pipe the output through a simple diff script. Something like this:

#!/bin/bash
prev=./data/product-prices.prev
curr=./data/product-prices.json

if [ -f "$prev" ]; then
  jq -S . "$prev" | diff - <(jq -S . "$curr") | grep "^[<>]" 
fi

cp "$curr" "$prev"

This gives you a clean line-by-line comparison of what changed between runs. Runs for about 12 seconds on a dataset of roughly 2,000 products. Not lightning fast, but acceptable for a daily check. Here's the thing nobody mentions in the docs. Dig Dig Io struggles with APIs that use JavaScript-rendered content. If the data you want lives behind a React or Vue component that loads after the initial page response, the default HTTP client will grab the shell HTML and return nothing useful. I spent about three hours wrestling with this before I realized what was happening. The workaround is to use the headless browser mode. Dig Dig Io ships with a bundled Chromium instance that you can enable with a single flag:

dig-dig-io run product-prices --browser

This triggers a full browser render cycle before extraction begins. It's slower — roughly 4x — but it gets the data. The tradeoff is that you can't use it in tight schedules or high-volume scenarios. If you're hitting more than 50 pages per day through the browser mode, expect your runtime to blow up and your ISP to potentially flag you. Another edge case: when APIs return data in chunks using compressed streams, Dig Dig Io sometimes stops mid-stream without an error. This happened to me with a particular supplier API that encoded responses with gzip chunking. The fix was adding force_decompress: true to the source config. The docs barely mention this flag. I found it by grepping through the GitHub issues after the fourth failed run.

Setting Up Authentication and Secrets

Security is a common concern. Dig Dig Io supports several auth methods: Bearer tokens, API keys in headers, basic auth, and even certificate-based auth for internal APIs. The recommended approach is environment variable substitution in your config. Any variable prefixed with a dollar sign gets resolved at runtime from your shell environment. I store all my secrets in a ~/.dig-dig/secrets.env file with permissions set to 600. It looks like this:

API_TOKEN=your_bearer_token_here
DB_PASSWORD=something_secure
WEBHOOK_SECRET=another_secret

Then in your config, you reference them as $API_TOKEN and so on. The secrets file itself is excluded from git via a .gitignore rule. This keeps your credentials safe without requiring a separate vault system for most home-lab or small-team use cases. By default, Dig Dig Io writes logs to ~/.dig-dig/logs/. Each run gets its own file named with a timestamp. The log format is JSONL — one line per event. You can query recent runs with: This shows a summary table of the last 10 runs with duration, status, and record counts. For production use, I'd recommend forwarding these logs to something like Loki or even a simple file rotation script. The tool doesn't ship with built-in log management beyond the standard rotation after 30 days.

There's also a health check endpoint if you're running it as a background service:

curl http://localhost:8477/health

It returns uptime, last successful run timestamp, and queue depth. Useful for alerting scripts or dashboard monitoring. I should be honest about where this tool hits its limits. First, it's not designed for high-frequency scraping. If you need sub-minute refresh cycles, the overhead of connection pooling and validation makes it inefficient. Tools like Scrapy or custom Python scripts with asyncio will outperform it easily in those scenarios. Second, the error handling is mediocre. When an API returns a 500 or a malformed response, Dig Dig Io logs the error but continues processing subsequent items without interrupting the run. This means you can end up with partial datasets that look complete until you notice missing records. I learned this the hard way when a supplier's API went down for six hours and I didn't realize my "daily" pull was only getting 40% of expected products.

The workaround is to add a post-run validation step. I use a simple script that checks record counts against expected thresholds and sends alerts if they fall below 90%. It's not perfect but it caught the issue that the tool itself missed. Third, complex nested data structures can be painful to extract. The field mapping syntax supports dot notation and array indexing, but once you get beyond two levels of nesting, the config file becomes unwieldy. I've seen people hit this wall with analytics APIs that return deeply nested event hierarchies. In those cases, a custom parser might be more appropriate.

Alternatives Worth Considering

If Dig Dig Io isn't quite right for your use case, there are other options. For pure API aggregation, Python with requests and pandas gives you far more control and debuggability. For web scraping at scale, Scrapy remains the industry standard with a mature ecosystem. If you need a no-code solution, n8n or Make offer visual workflows that handle many of the same tasks without the YAML learning curve. My recommendation: start with Dig Dig Io if your needs are moderate — daily or hourly pulls from a handful of APIs, mostly straightforward JSON responses. It's fast to set up and the learning curve is shallow. Move to a custom solution if you hit the limitations I described above.

Getting Started

To download and try Dig Dig Io, visit the official repository at github.com/dig-dig-io/dig-dig-io or the documentation site at dig-dig.io/docs. The installation varies by platform. On Linux with apt: After installation, run dig-dig-io init to set up your config directory and generate a starter config file. From there, follow the walkthrough in the docs. It takes about 15 minutes to get your first successful run going. The project is open source under the MIT license. Community support is active on GitHub and Discord. The maintainers respond to issues within a few days on average, though complex bugs may take longer. Release cadence is roughly monthly with security patches issued as needed.

Final Thoughts

Dig Dig Io sits in a niche that isn't well served by either generic scraping tools or purpose-built enterprise solutions. It's fast to configure, reasonably reliable for straightforward tasks, and the CLI workflow suits automation well. But it's not a silver bullet. Know its limits, add validation layers for critical data, and don't expect it to handle edge cases that require deep protocol knowledge. For the right workload — and I'd estimate that's maybe 60-70% of typical API consumption scenarios — it's a solid choice that saves time without introducing unnecessary complexity. Just make sure you understand what you're signing up for before committing it to production workflows.