Building an Automated Monthly PDF Report Pipeline for ML Models
Most data science teams I talk to end up hand-crafting model performance reports every month. It's tedious, error-prone, and the reports look different every time because someone changed a color or moved a chart. I built a pipeline that generates a consistent PDF every month without touching it, and it's been running for about fourteen months now. Here is how it actually works. The core of the system is a Python script that queries your model's stored metrics, renders charts with matplotlib, and compiles everything into a PDF using reportlab or weasyprint. You schedule it with cron or Airflow to run on a fixed day each month. The output goes to a shared folder. That's it in one sentence. The details matter more than the sentence though. I use pandas to load the previous month's metrics from a PostgreSQL database where my training runs log everything via MLflow. The script filters for the model version associated with the current month, calculates key metrics like RMSE, F1, AUC, and inference latency, then renders those as bar charts and line graphs. After that, it pulls the results into a template PDF with a header, the charts, a brief metrics table, and a footer with the report date.
The template part is where people mess up. I learned that the hard way. My first version used a loose HTML-to-PDF converter and the layout broke every time I added a new chart. The charts would overlap the text or get cut off at the page edge. I switched to a strict reportlab approach with pre-defined frames and positioned elements using absolute coordinates. That eliminated the layout drift completely. Page breaks became predictable instead of random. One specific problem I ran into was handling models that had zero new training runs in a given month. The script would crash because the query returned an empty dataframe and the charting code couldn't render anything from nothing. I added a guard clause that checks if the dataframe is empty before proceeding. If it is, the script writes a placeholder page that says no new data available for this period and skips the charts. It's not elegant but it stops the pipeline from failing silently. Another edge case is when a single metric has wildly different scales across months. I had one project where model latency jumped from 120 milliseconds to 8 seconds because a new feature was added to production. The chart axis compressed all the earlier months into an unreadable sliver at the bottom. I solved this by using a logarithmic scale on the Y-axis and flagging the metric in the report text so readers know why the chart looks unusual. Without that note, anyone reviewing the PDF would assume the chart was broken.
For the scheduling piece, I originally used a simple crontab entry running the script on the first business day of each month. That worked fine until holidays shifted the date and the report landed on a weekend. I moved to Airflow with a schedule that accounts for business days. The DAG has one task for data extraction, one for chart generation, and one for PDF assembly. If any task fails, the whole pipeline retries once and sends a Slack notification. This saved me from discovering missing reports three days late. Storage is worth mentioning. These PDFs accumulate quickly if you keep every version. I set up a retention policy that keeps the last twenty-four reports and archives the rest to S3 with a lifecycle rule that moves them to glacier after ninety days. The active reports stay in a Google Drive folder that the engineering team references during quarterly reviews. Without that cleanup, the folder became unmanageable within six months. A counter-intuitive thing about this setup is that simpler charts perform better than complex ones. I initially tried to include confusion matrices, ROC curves, and feature importance plots alongside the basic metrics. The PDF became dense and nobody read past the first page. I cut it down to three charts maximum: a metrics comparison table, a trend line for the primary metric, and a scatter plot for latency versus throughput. The report went from twenty pages to six. Readability improved dramatically.
Get the Full Details
There are limitations to this approach that you should know about. The pipeline only works well when your metrics are structured and consistently logged. If different engineers store metrics in different formats or use different naming conventions, the script will either miss data or produce incorrect values. I spent two weeks normalizing column names across five different experiment tracking systems before the reports started looking right. Another limitation is that this system assumes you have a working MLflow or similar tracking backend. If you're logging results manually in spreadsheets, automating this is significantly more work than it is worth. If you don't have a structured metrics pipeline yet, I'd suggest starting with a manual PDF generation process for two or three months before automating it. That gives you enough time to understand which metrics actually matter to stakeholders and which ones you can drop. I've seen teams automate reports that included every metric their models ever produced, and then wonder why nobody opened the files. Here is a minimal structure for the Python script if you want to build something similar:
Import pandas, reportlab, and sqlalchemy. Query the database for the month's metrics. Validate the data is not empty. Generate charts with matplotlib and save them as PNG files. Use reportlab to create a canvas, add a title paragraph, insert each chart at a fixed position, add a metrics table, write the date in the footer, and save the PDF to the output directory. The exact code depends on your database schema and chart preferences, but the flow stays the same. Test it once with real data before wiring it to a scheduler. I wasted a full sprint running the script against synthetic data and then watching it fail on the first real month because of a timezone mismatch between the database timestamps and Python's datetime parser. The fix was adding tzinfo to the query filters. For the download link part of this, I've put the basic template and a README with setup instructions on GitHub under a MIT license. The repo includes the Python script, a sample .env file for database credentials, and a cron example. You can clone it and adapt the database query to match your schema. I don't maintain it actively but pull requests get merged when they come in.
