Building Your Own Data Science Journal Without Paying for Fancy Platforms
I spent about three years cycling through Obsidian, Notion, Google Docs, and a half-dozen GitHub repositories just trying to track experiments and insights. Nothing stuck. Everything required too much friction or got abandoned after two weeks because the setup became its own project. I eventually just built something minimal with a flat file structure, some Python scripts, and a local SQLite database. It does what I need without demanding anything from me beyond opening a terminal. The core idea is simple. You maintain a structured collection of entries that log your experiments, code snippets, data findings, and references. Each entry gets a unique identifier, a timestamp, and consistent metadata. The format doesn't matter nearly as much as the consistency of your workflow. I use plain Markdown files stored in a directory hierarchy organized by date and project. A single Python script handles indexing into a small SQLite database so I can query entries later without digging through folders manually. Here's what the file structure looks like on my machine. Every project gets its own folder under ~/ds-journal/. Inside each project folder there's an entries/ subdirectory with one Markdown file per experiment or note. The filename follows YYYY-MM-DD-slug format. I also keep a config.json file at the root level that stores project-level metadata like the dataset source, tool versions, and notes about environment setup. This last part is important because data science projects tend to have fragile dependency chains, and forgetting which version of scikit-learn you used three months ago is a real problem.
I wrote a Python indexing script called journal_index.py that scans the entries/ directories and populates a SQLite database. The database has three tables: entries, tags, and references. The entries table holds the metadata and a path to the Markdown file. The tags table is a many-to-many relationship so a single experiment can have multiple tags. The references table tracks citations and links to external papers, GitHub repos, or documentation pages. Running the indexer takes about four seconds across roughly 800 entries on my machine. I usually run it once a week after batching my notes. Querying the journal is where this actually becomes useful. Instead of searching through folders or using Spotlight, I run SQL queries directly. Something like selecting all entries from Q3 2025 tagged with "hyperparameter-tuning" and ordered by timestamp. That gives me a focused view of what I tried and when, which is something no general-purpose note app handles cleanly for technical work.
Setting Up the Basic Structure
You don't need any special software beyond Python 3.10 or later and SQLite, which comes pre-installed on most systems. Start by creating your directory layout. I recommend ~/ds-journal/projects/ for project folders and ~/ds-journal/index.db for the database. Each project folder should contain entries/, config.json, and a README.md describing the dataset and goal. The config.json format is straightforward. Here's an example of what I use: {"project": "fraud-detection-v2", "dataset": "kaggle_fraud_detection_2024", "tools": ["python", "xgboost", "optuna"], "notes": "Moved from lightGBM to xgboost after cross-validation showed 3% improvement", "created": "2025-01-15"}
Get the Full Details
Each journal entry is a Markdown file. The frontmatter uses YAML-style metadata at the top. I include fields for title, date, project, tags, objective, methods, results, and lessons. The results section is where most people skip documenting things, but writing a couple lines about what actually happened rather than just the final accuracy number has saved me from repeating failed approaches at least a dozen times.
Common Pitfalls and What Actually Works
The biggest mistake I see is over-engineering the system before you've accumulated enough entries to justify it. I built a web interface once with Flask and SQLite behind it. Took me a weekend. Used it for eleven days. Then I went back to CLI queries because opening a browser tab felt like more steps than just running a script. Another issue is tag inconsistency. If you start tagging the same concept differently across entries — "ml-model" in one place, "model-training" in another, "training" somewhere else — your queries become useless. Pick a small set of tag names and stick with them. I keep a reference file in my journal root called tag-schema.md that lists every valid tag and what it means. It takes me maybe ten seconds to check before adding a new one. Here's something counter-intuitive that took me too long to figure out: documenting failures is more valuable than documenting successes. A successful experiment often has a clean narrative. A failed one usually involves a chain of small decisions that are easy to forget. When I started writing down exactly why each failure happened — which parameter caused the overflow, which assumption was wrong, which library version introduced the bug — my iteration speed roughly doubled. I stopped wasting time rediscovering the same dead ends.
Query Patterns I Actually Use
Most of my journal queries fall into a few categories. I search by date range when reviewing what I did last sprint. I search by tag when preparing for a meeting and need to recall relevant experiments. I search by keyword across results when trying to remember whether I tried a specific preprocessing approach on a particular dataset. A typical query looks like this. Select entries from the fraud-detection project between November and January where the results field contains "oversampling" and order by date descending. That returns three entries and takes less than a second. Without the journal, I'd be searching through Slack messages, old notebooks, and email threads, probably giving up after five minutes. One limitation worth noting upfront: this system doesn't scale well past a few thousand entries. The SQLite database starts getting sluggish around that point, and my flat file approach doesn't handle version control conflicts gracefully. If you're managing a team journal or expect to accumulate thousands of entries, you'd be better off using something like DVC for experiment tracking or setting up a proper PostgreSQL instance with full-text search. For an individual practitioner working on a handful of projects at a time, the flat file plus SQLite setup is more than adequate.
Keeping It Maintainable
I use a cron job to run the indexer every Sunday at 9 AM. It takes roughly four seconds. I also have a cleanup script that archives entries older than eighteen months into a separate compressed folder. The archive process preserves the database records but moves the Markdown files out of the active directory. This keeps my daily working set lean without losing the ability to look things up if needed. The whole setup — directory structure, scripts, config files — takes about an hour to configure from scratch. I've refined it over three years, so the current version is tighter than the original. If you're starting today, I'd suggest keeping it even simpler than what I have. One folder per project, Markdown entries with YAML frontmatter, a single indexer script, and a weekly habit of running it. That's it. Anything more than that tends to become a distraction rather than a tool. There's a working version of the indexer and config templates available on GitHub under my username. The repository is called ds-journal-diy. It includes the Python scripts, example config files, and a setup guide that walks through installation in about fifteen minutes on a fresh system.
