Why most ML project trackers fail before they help
I spent about three years trying to build proper project documentation systems for machine learning work, and the pattern was always the same. People spend more time maintaining the tracker than actually doing the work, so they abandon it, and the knowledge gets lost. I ended up creating something deliberately bare-bones just to keep from losing track of which experiment led to what result. This is essentially a single-page template that forces you to capture the critical decision points of an ML experiment without becoming paperwork. It has six sections: problem statement in one line, data source and size, feature list, model choice with hyperparameters, evaluation metric, and the one-line lesson learned. That's it. Nothing fancy. The reason this works better than a Notion database or a dedicated tool is that it lives in a text file. You can version-control it, search it with grep, and fill it out in under two minutes between training runs. I use a plain markdown file per project, and when I need to find out why I abandoned a particular approach six months ago, I can scroll through the file in about ten seconds.
Here is what each section actually looks like in practice: The problem statement is where most people mess up. They write paragraphs describing their goal. The constraint here is one line, no exceptions. "Predict churn from call center metadata" is sufficient. "Building an ML pipeline to reduce customer attrition by identifying at-risk accounts" is wrong because it describes a business outcome, not a modeling task. Keep it technical. This took me a while to learn the hard way when I was tracking twelve different classification approaches for a delivery routing problem and couldn't tell which one used which formulation by the end of the week. Data source and size needs a timestamp or version ID. If you are pulling from a database table, note the query filter. If you are downloading a dataset, note the URL and the date. Data drifts, schemas change, and people rename columns without telling anyone. I once spent four hours debugging a model that had silently started receiving null values because someone added a new optional field to the upstream table, and my worksheet from two weeks prior would have caught it immediately if I had recorded the schema version.
Feature list should distinguish between engineered features and raw columns. I write it like: "age (raw), days_since_last_purchase (engineered), interaction_term_age_x_days (engineered)." The reason is that when you come back to reproduce results, the difference between a raw column and an engineered one changes everything about how you handle missing values and scaling. Beginners tend to group these together and then wonder why their preprocessing pipeline behaves inconsistently across train and test splits. Model choice and hyperparameters. Don't just write "Random Forest." Write "Random Forest, n_estimators=100, max_depth=None, min_samples_split=5." The difference between max_depth=10 and max_depth=None can change your AUC by point-zero-four on tabular data, and you will not remember which you used unless you wrote it down. I had a situation where I tuned a gradient boosting model across three weekends and could not reproduce my best result because I only noted "good hyperparameters" in my notes. Two days of wasted work. Never again. Evaluation metric matters more than you think. If you are doing binary classification, state whether you are using AUC-ROC, F1, log loss, or something else, and specify which class is positive. "Accuracy" is almost never the right answer, and writing "accuracy" without qualification means someone reading your worksheet later will have to reverse-engineer what you actually optimized for. I used to write vague metrics and then get confused during model comparisons, picking the worse model because I had accidentally compared accuracy on one run to F1 on another.
Get the Full Details
The lesson learned section is the most valuable part of the entire thing. One line, written after the experiment completes, describing what you discovered. "Adding recency features improved AUC by 0.03 but increased inference latency by 40ms." "Principal component analysis made things worse, not better." "The validation set was leaking because I fitted the scaler before splitting." This section builds into a personal knowledge base that accelerates every subsequent project. After about twenty experiments, you start seeing your own patterns repeated.
How to set this up in under five minutes
Create a folder structure like this: projects/delivery_churn_reduction/experiment_log.txt. Put the template inside as a starting point, and duplicate it for each new experiment. I use a simple format that looks like this: Experiment 001: 2025-03-12 Problem: Predict customer churn from call center metadata (binary classification)
Data: call_center_db.customers_2024, 450k rows, 23 columns, snapshot 2024-11 Features: account_age (raw), call_count_30d (raw), tenure_months (raw), avg_call_duration (engineered), ratio_long_short_calls (engineered) Model: XGBoost, n_estimators=200, max_depth=6, learning_rate=0.05, subsample=0.8
Metric: AUC-ROC, positive class = churned Result: AUC 0.847, training time 12 min Lesson: Call count features dominated importance; removing avg_call_duration had no impact on AUC
This takes about ninety seconds to fill out after an experiment. The act of filling it out forces you to articulate what you did, which catches mistakes before they compound. I usually fill it out while the next model is training so I am not wasting idle compute time on documentation.
Where this approach breaks down
A single text file does not scale past roughly fifty experiments per project. After that, searching becomes tedious even with grep. At that scale, you need a lightweight database or a spreadsheet with filters. I transition to a CSV-based tracker at around experiment thirty, keeping the same six fields but adding columns for tags and experiment type so I can sort and filter. The worksheet also does not replace actual experiment management tools if you are working in a team. Dvc, Weights & Biases, or MLflow are necessary when multiple people are running experiments concurrently. This is a personal tracking tool, not an infrastructure solution. I have seen people try to use this as a substitute for proper experiment tracking in team settings, and it falls apart within a week because there is no audit trail or shared state. Another limitation: this approach assumes you are doing tabular or straightforward classification regression work. Computer vision and NLP projects often require logging image samples, token lists, or augmentation pipelines that do not fit cleanly into one line. For those domains, I add supplementary files alongside the worksheet rather than trying to compress everything into the template.
The counter-intuitive part nobody mentions
Most people think the value of a worksheet is in looking back. The actual value is in the writing. The constraint of fitting your experiment into six fields forces you to make decisions you would otherwise skip. You have to choose your metric before you train. You have to enumerate your features explicitly, which catches data leakage when you realize you listed a column that could only be computed after the fact. You have to write a lesson learned, which means you cannot call an experiment a failure and move on without extracting something useful. I noticed this effect consistently across projects. The experiments where I filled out the worksheet thoroughly were the ones where I could actually reproduce results and build on them. The experiments where I skimmed the worksheet were the ones I had to redo from scratch. The discipline of the format is the feature, not the content. If you want to start using this, copy the template above into a text file and use it for your next three experiments. You will probably find yourself adding or removing fields, and that is normal. The point is to begin with the constraint and adjust from there, not to design the perfect system upfront. The perfect system is the one you actually use consistently.
