Diy Data Science Checklist

Getting Started With DIY Data Science Without Losing Your Mind

I built my first real data science checklist after I wasted three weeks debugging a pipeline that turned out to be a timezone mismatch in a PostgreSQL dump. Not dramatic. Just embarrassing. Since then I've been assembling and refining a practical Diy Data Science Checklist for people who want to do this themselves without falling into the same traps I kept hitting. Here is the checklist, roughly in the order it matters during a project. This is not a sequential pipeline because real projects do not respect linear workflows. You will jump around. That is fine. Define the actual question before you touch any data. This sounds obvious and most people skip it. Write one sentence that describes what success looks like. If you cannot write it without using words like leverage or synergize, you do not have a clear enough problem yet. Go back.

Identify your data sources and document where each comes from. This means naming the specific tables, APIs, spreadsheets, or scraped pages. I once had a model trained on a CSV that someone had exported manually three months prior. The column headers had shifted because a colleague added a status field mid-project. The model silently predicted worse for customers who called support. I caught it only because I checked the schema drift report against the original source. Record the exact file paths, API endpoints, and access credentials. Use environment variables, not hardcoded strings. Store the path in a .env file and add that file to your .gitignore. This is not optional. I learned this the hard way when a junior developer pushed a SQLite file containing hashed passwords to a public repo. Decide what data type and volume you actually need. Roughly 10,000 labeled examples is a minimum for most supervised learning tasks unless your signal is very clean. If you are doing anomaly detection on event logs, you might get away with less because the baseline is implicit. If you are predicting churn from transactional data with few events per user, you likely need more. Estimate conservatively.

Phase Two: Data Collection And Cleaning

Pull the data exactly once if possible, then store it as immutable raw data. Do not edit the original. Keep a copy untouched. Write a reproducible ingestion script that logs timestamps, row counts, and any errors. If something breaks mid-pull, you should be able to rerun it and get the same starting point. Create a schema document before you clean anything. List every column, its type, its allowed range, and its source. When cleaning data, you are making decisions. Document each decision with a timestamp and a short reason. A column marked as nullable one week and required the next is a silent bug waiting to surface during production. Handle missing values with intent, not default habits. Do not drop rows with missing values just because it is fast. If 40 percent of a feature is missing, dropping rows might destroy your sample. Impute with median for skewed distributions, mode for categorical features with one dominant class, and model-based imputation when the missingness looks informative. I ran into this with a healthcare dataset where the missingness in a lab result actually indicated the patient skipped the test. Dropping those rows biased the model toward sicker patients.

Check for duplicates, leaks, and target leakage explicitly. A duplicate row is usually a merge error or a log replay. Duplicate detection is cheap compared to a model that looks robust until it is deployed. Target leakage means a feature contains information about the label. Examples include using a post-treatment variable to predict an outcome, or including a transaction ID that maps back to the target. Run a quick correlation check between every feature and the target, then sanity-check the top correlates against your domain knowledge.

Phase Three: Exploratory Data Analysis

Write a short EDA summary instead of running exploratory analysis for hours. Report the distribution of each feature, basic summary statistics, and two or three plots per feature. If a feature has extreme skew, plot it on a log scale. If a categorical feature has 200 classes, show the top ten and group the rest. Long tail distributions are normal. Treating them like mistakes is not. Split by time, not randomly, whenever your data has a temporal component. Random splits leak future information into training. If you are predicting next-month sales using monthly data, train on January through September and validate on October and November. I used random k-fold CV on a time-series dataset once and got an AUC of 0.94. The real-world AUC was 0.61. The leak was obvious in hindsight but invisible during model selection. Baseline everything. Create a trivial model before you build anything fancy. For classification, predict the majority class. For regression, predict the mean. Record the performance. If your model does not beat the baseline by a meaningful margin, you are building complexity for no reason. Most models fail here. That is normal. Move on and fix the data or the question instead of adding layers.

Phase Four: Feature Engineering

Build features that reflect the causal mechanism, not just statistical association. A feature that predicts well but has no logical connection to the outcome tends to break under distribution shift. I had a retail dataset where the number of product descriptions per SKU was a strong predictor of sales. It was also a proxy for how recently the product was added to the catalog. When we moved to a new region with different onboarding speed, the model lost 18 percent of its accuracy in two weeks. Removing the feature cost five percent initially but preserved performance over six months. Normalize or standardize features after the train-validation split. Fit the scaler on training data only, then transform validation and test data. Fitting on the full dataset leaks statistics. This is one of the most common mistakes in production pipelines. You can forget it in a notebook and realize it later when your cross-validation scores look impressive and your holdout performance drops sharply. Use feature stores or versioned feature definitions for anything beyond a single experiment. A feature defined as revenue minus costs means nothing without the exact query, date range, and deduplication logic attached. Write a small YAML or JSON file that captures the definition, the transformation steps, and the data source. I stopped writing ad-hoc SQL for features after a deployment failed because the staging query joined a table that was renamed in production without updating the documentation.

Phase Five: Modeling

Start with simple models and only add complexity when the simple model hits a wall. Logistic regression, random forest, and gradient boosting cover most problems. Neural networks are overkill for tabular data in most cases and add maintenance cost without meaningfully improving accuracy. I see people reach for transformers on datasets with five thousand rows and twenty columns. They should not do that. Use nested cross-validation for hyperparameter tuning when you have limited data. Outer folds estimate generalization error. Inner folds select hyperparameters. Without nesting, your performance estimate is optimistic because the outer fold sees the effects of inner model selection. For datasets under 50,000 rows, nested CV usually adds one to two days of compute but prevents you from publishing a result that will not reproduce in production. Track every experiment with a tool that logs parameters, metrics, and artifacts. MLflow, Weights & Biases, or even a simple CSV log. Without tracking, you will not remember which configuration produced which result. I spent two days recreating a model that I had already built because I forgot to save the random seed and the exact feature list. A one-line logger would have saved that.

Set a maximum training time and memory budget. A model that takes four hours to train is less useful than a model that takes twenty minutes and is nearly as accurate. You will iterate faster, retrain more often, and catch regressions sooner. I use a rule of thumb: if training exceeds ten minutes for a notebook experiment, I refactor the code or reduce the dataset before continuing.

Phase Six: Evaluation

Choose metrics that match the business cost, not the textbook default. Accuracy is almost never the right metric for imbalanced problems. Use precision, recall, F1, ROC AUC, or PR AUC depending on whether false positives or false negatives are more expensive. If a false negative costs ten times a false positive, optimize for recall at a fixed precision threshold. I once optimized a fraud model for AUC alone and deployed it with a threshold that produced too many false alarms. Operations rejected the model within a week. Validate on a holdout set that was never used during development. Reserve ten to twenty percent of your data at the beginning. Do not look at it until the final evaluation. If you peek, you will unconsciously tune toward it. This happens more often than you think. I have done it myself. Report confidence intervals, not just point estimates. A single AUC of 0.87 tells you nothing about stability. Bootstrap the metric across folds or run multiple train-test splits with different seeds. Report the mean and standard deviation. If the standard deviation is larger than 0.02, your result is noisy and you should either collect more data or simplify the model.

Phase Seven: Deployment And Monitoring

Package the model with the exact preprocessing pipeline. A model file alone is not a model. It is a snapshot. Include the scaler, the encoder mappings, the feature list, and the version of every library used. I use joblib for Python scikit-learn pipelines and ONNX for models that need to run across languages. The package should load in under five seconds on a standard CPU instance. Write an inference API that accepts the same input format as your training data. If training used a JSON payload with feature names as keys, the API should use the same payload. Do not switch to a positional array at deployment. I built an API that accepted arrays and spent a week debugging why predictions drifted. The input order had changed between training and inference because the engineering team reordered columns for a batch job. Monitor feature drift, prediction drift, and business outcomes. Set up automated checks that compare incoming feature distributions to the training baseline. If the distribution of a key feature has shifted by more than two standard deviations for three consecutive days, trigger an alert. I use KS tests for continuous features and chi-squared tests for categorical features. A simple drift detection script that runs hourly takes about an hour to write and saves you from flying blind.

Plan for retraining before you deploy. Data drift is not a bug. It is a condition. Define a retraining schedule based on drift signals, not calendar dates. If drift stays below threshold for six months, monthly retraining is wasteful. If drift spikes after a marketing campaign, retrain immediately. I set up a pipeline that triggers retraining when the drift score exceeds a configurable threshold and validates the new model against the current one before swapping.

Common Pitfalls That Are Not Obvious

Overfitting to the wrong thing. People often optimize for a metric that correlates with the true objective but is not the true objective. Click-through rate is not revenue. Conversion rate is not retention. Check the downstream impact of your optimization target before you celebrate a good score. Data leakage through group structure. If your data has repeated entities, such as users, orders, or devices, you must split at the entity level. Random row-level splits will put samples from the same entity in both train and test sets, inflating performance. Use GroupKFold or a custom split that groups by the entity key. I missed this on a customer-level churn project and the model appeared to reach 0.92 AUC in validation. Real performance was 0.74. Assuming a clean pipeline means a clean result. A well-written pipeline can still produce garbage if the question is wrong, the data is biased, or the deployment context differs from the training context. The checklist covers engineering. It does not replace thinking about whether the model should exist in the first place.

When This Approach Fails

This checklist assumes you have enough data to make decisions and enough computational resources to train models that are larger than a logistic regression. If you have fewer than one thousand labeled examples and no synthetic data option, skip the complex modeling stages and focus on data collection and feature quality. If you do not have access to a GPU or a cloud budget, stop trying to train large models and use smaller ones. A random forest with fifty trees often outperforms a poorly tuned neural network on tabular data and runs on a laptop. If your data is highly unstructured, such as images, audio, or long-form text, this checklist is a starting point, not a complete guide. You will need domain-specific preprocessing and different evaluation methods. The core discipline of tracking, validation, and drift monitoring still applies. I keep a living copy of this checklist in a markdown file inside each project. It gets updated whenever I hit a new failure mode. The version from 2023 is already outdated because we moved from on-prem servers to managed cloud and that changed the deployment and monitoring steps significantly. Treat it as a working document, not a law.