Why Most Data Analysis Examples You See Are Useful in Theory and Useless in Practice
Everyone posts a clean DataFrame and three lines of code that produces a beautiful visualization. The data was already tidied. There were no nulls, no weird date formats, no column names that changed depending on which source you pulled from. What you actually need is a look at what happens when the pipeline breaks, because it always breaks. Last quarter I was working with transaction logs from three different regional subsidiaries. Each one stored dates in a slightly different format. One used DD/MM/YYYY, another used MM-DD-YYYY, and a third one had rows where the date column contained the text "TBD" scattered among actual timestamps. A standard tutorial would strip that out and move on. I had to keep those rows because they represented legitimate backlogged entries. The core steps looked like this. Load the raw data. Inspect the shapes and dtypes. Reconcile the date columns into a single standard. Impute the TBD values using forward-fill within each subsidiary group. Then merge on transaction ID rather than on date, because duplicate IDs across subsidiaries were common and merging on date would create a Cartesian explosion. From there, aggregate by region and product category, calculate month-over-month growth rates, and finally export.
I put together a stripped-down version of that workflow so other people can follow the same pattern without reinventing the edge-case handling. It uses pandas, a couple of standard libraries, and the sample dataset is included as a CSV inside the package. You can grab it from the public repository linked at the bottom of this page. The repo contains three files. The first is the raw_transactions.csv file with intentionally messy dates, duplicated IDs, and a few nulls in the amount column. The second is the analysis script that walks through loading, cleaning, merging, aggregating, and visualizing. The third is a Jupyter notebook version for people who prefer cell-by-cell debugging. I wrote the script so you can run it directly without configuring a virtual environment, though using one is still recommended if you plan to modify anything.
What People Usually Get Wrong on the First Pass
The most common mistake is merging on a key that looks unique but isn't. In my experience, transaction IDs repeat across sources because each subsidiary numbers its own ledgers independently. If you just concatenate everything and groupby the ID, you will silently combine rows that belong to different entities. The fix is to add a source prefix to every ID before the merge, or better yet, create a composite key from subsidiary code and transaction ID. A second issue is treating nulls as zeros. I have seen pipelines where a null in a revenue column got filled with zero, which then skewed averages downward and made seasonality plots look flat. Nulls in financial data usually mean missing reports, not zero revenue. Forward-fill, interpolation, or flagging them as missing is safer than imputing a hard zero. The analysis script handles this by converting the amount column to numeric with a coerce flag, which turns non-numeric values into NaN instead of raising an error.
Get the Full Details
How the Script Actually Works, Step by Step
First, the script reads the CSV and prints the shape and dtypes. This is where you catch problems before they propagate. A column that pandas guessed as object when it should be datetime is a red flag. You can force the right type with pd.to_datetime and errors='coerce', which converts unparseable strings into NaT. Next, the date reconciliation block normalizes all date columns. It tries multiple formats in sequence and keeps the first successful parse. If every format fails, the row stays as NaT. This is slower than a single-format parse, but it saves you from losing records that have minor formatting drift. Then the merge step joins the three regions using the composite key. After that, the script drops duplicate rows based on the key and inspects the result size. If the final row count is wildly different from the input total, something is off, and you should inspect the dropped rows rather than blindly proceeding.
The aggregation block groups by region, product category, and month. It calculates sum, mean, and month-over-month growth. Growth rate is calculated as the percentage change from the prior month, with a check to avoid division by zero. Values where the prior month is zero are set to NaN instead of infinite or undefined. Finally, the visualization exports a pair of plots. One is a stacked bar chart showing revenue by category across months. The other is a line chart comparing regional growth trends. Both are saved as PNG files in the output directory.
Where This Approach Breaks Down and What to Use Instead
This script is not built for datasets larger than a few million rows. Pandas holds everything in memory, so as the data grows, the process slows to a crawl and may crash on machines with less than 16 GB of RAM. For larger scales, you should switch to a DuckDB query or a Spark job. The logic is the same. Only the engine changes. Another limitation is that the date reconciliation here is format-order dependent. If your data contains a mix of ambiguous dates like 01/02/2023, the script will default to the first format that parses. In production, you should enforce a single input format at the ingestion layer or tag each source with its expected convention. Otherwise you will get silent misinterpretations that are very hard to trace later.
Download and Run It Yourself
The complete example is available here: github.com/sapiens-ai/data-analysis-example/releases Clone the repo, open the raw_transactions.csv file to see the intentional mess, then run the main script with python. No special dependencies beyond pandas and matplotlib are required. If you want the notebook version, open the .ipynb file in Jupyter and run the cells in order. Output plots and the cleaned CSV are written to the output/ folder. The script is documented inline, so you can follow the logic without reading external material. If you run into an error, check the dtypes output at the top of the console. Almost every failure comes from a type mismatch that shows up in the first few lines. Fix the type, rerun, and the rest usually follows without additional changes.
This is not a polished enterprise solution. It is a working template that shows the parts people forget to include in tutorials. Use it as a starting point, not a finished product.
