Epidemiology And Data Science

What Actually Happens When You Merge Disease Tracking With Statistics

I spent three years building surveillance dashboards for a regional health department, and the first thing I learned is that epidemiology and data science don't naturally fit together. They're two different languages. Epidemiologists think in patient counts, incubation periods, and case definitions. Data scientists think in features, pipelines, and model performance. When you throw them into the same project, the friction shows up fast. The problem isn't the tools. It's the workflow. I remember trying to pull case data from a hospital EMR system that stored dates as text strings in three different formats within the same table. One column had "03/15/2019," another had "15-Mar-19," and a third just had "3/15." I spent two days writing parsing scripts before I realized the real issue was that the billing department and the clinical staff were entering dates for different purposes. Billing needed admission dates. Clinical needed symptom onset. The same patient appeared twice with two different timestamps, and my initial outbreak analysis was completely wrong because I was matching on the wrong date field.

The Reality Of Epidemiology And Data Science Work

Here's what nobody tells you about this field. The math is straightforward once you understand the basic survival analysis and time-series methods. The hard part is that your data will always be messier than you expect. Hospital discharge records have gaps. Contact tracing interviews rely on human memory during stressful events. Lab confirmation takes days, sometimes weeks. By the time your dataset looks clean enough to analyze, the outbreak window has closed. The standard approach people learn in textbooks assumes complete case ascertainment. Real public health data never works that way. You're usually analyzing what I call the iceberg dataset—the visible confirmed cases on top, and a much larger submerged section of probable and suspected cases that never get lab-confirmed. If you only model the tip, your reproduction number estimates will be wrong, sometimes dramatically. In one respiratory virus cluster I worked on, we initially calculated R0 at 2.1 based on confirmed cases alone. After adjusting for under-ascertainment using a capture-recapture method, the true estimate jumped to 3.4. That changes everything about containment strategy. I started using a simple workaround that saved me months of head-hurting. Instead of waiting for perfect data, I build provisional models with whatever completeness rates I can estimate, then run sensitivity analyses across plausible ascertainment ranges. I document every assumption explicitly. When the real data finally arrives weeks later, I can quickly see which conclusions were stable and which ones collapsed under minor data quality shifts. This approach doesn't eliminate uncertainty. It just makes you honest about where it lives.

Common Methods People Actually Use

Survival analysis remains the workhorse for outbreak timing questions. Cox proportional hazards models handle censoring naturally, which matters when patients enter and exit your observation window at different times. I usually pair them with Kaplan-Meier curves for visualization. The proportional hazards assumption fails more often than textbook examples admit, especially in infectious disease where hazard ratios shift as population immunity builds. I check Schoenfeld residuals early and switch to time-varying coefficient models when they flag violations. Time-series decomposition works well for baseline patterns. Seasonal flu surveillance, for example, follows predictable annual cycles. You subtract the seasonal component, examine the remainder for anomalies, and flag departures that exceed historical variability thresholds. The trick is choosing the right decomposition method. Additive models assume constant amplitude across seasons. Multiplicative models allow fluctuations to scale with the baseline. Respiratory viruses tend toward multiplicative behavior because holiday gatherings amplify both the baseline circulation and any novel strain simultaneously. Using additive decomposition on multiplicative data creates false anomaly signals during high-traffic periods. Network analysis has become indispensable for contact tracing, but it introduces computational complexity that many practitioners underestimate. A single measles outbreak in a dense urban school can generate thousands of potential exposure edges. Standard adjacency matrix approaches consume too much memory. I use sparse matrix representations and community detection algorithms like Louvain or Leiden to identify transmission clusters before diving into individual contact chains. This reduces computation time from hours to minutes while preserving the essential structural information.

Machine learning applications in epidemiology face a persistent validation problem. Researchers love demonstrating that gradient boosting or neural networks achieve high AUC on retrospective datasets. The problem is temporal validation. A model trained on 2018-2019 data might look impressive until you test it against 2020 patterns, which are structurally different due to behavioral changes, testing availability shifts, and surveillance artifact inflation. I always hold out the most recent period for testing and report performance degradation honestly. Models that fail temporal validation are usually capturing surveillance artifacts rather than genuine biological signals.

Where This Approach Breaks Down

I need to be direct about limitations. Bayesian hierarchical models sound attractive for small area estimation, but they require careful specification of prior distributions and convergence diagnostics that many practitioners skip. I've seen published studies where posterior estimates were clearly influenced by arbitrary prior choices, and the authors didn't notice because they never ran diagnostics. If you use Bayesian methods, run multiple chains, check R-hat statistics, and report sensitivity to prior specification. Don't treat MCMC output as gospel. Geographic information systems introduce their own pitfalls. Spatial autocorrelation violates independence assumptions in standard regression models. SaTScan and similar cluster detection tools handle this better, but they assume stationary underlying populations, which rarely holds in mobile urban environments. During a foodborne outbreak investigation, I initially identified a apparent cluster near a popular restaurant using standard spatial scan statistics. The cluster persisted even after adjusting for population density. I nearly declared a success until I traced the cases back and realized the restaurant was adjacent to a major transit hub. The "cluster" was actually passenger flow patterns, not restaurant exposure. Spatial tools detect patterns. They don't explain mechanisms. You still need field investigation. Data linkage itself creates hidden biases. When you connect electronic health records with laboratory databases and death registries, missingness is rarely random. Hospitalized patients generate complete records. Outpatient visits produce sparse data. People who die at home may never enter the system. Survival bias creeps into your analysis when your denominator excludes the sickest cases who died before presentation. I address this by documenting linkage rates for each data source and performing multiple imputation when missingness exceeds fifteen percent of key variables. The imputed datasets aren't perfect, but they prevent systematic underestimation of severity.

A Practical Workflow That Actually Works

Start with a data dictionary before writing any analysis code. I've seen projects where three analysts spent weeks reconciling variable definitions because nobody documented what "date of symptom onset" actually meant in each source system. Some used patient recall. Others used provider documentation. A few used laboratory collection dates as proxies. These produce different analytical populations with different bias structures. Version control your preprocessing scripts. Data cleaning is iterative. You'll discover errors, fix them, realize the fix broke something else, and need to revert. Git handles this gracefully. Pipeline tools like Apache Airflow or Prefect work better for scheduled ETL jobs that feed production dashboards. I prefer Prefect for epidemiology projects because the workflow visualization helps stakeholders understand data provenance, which builds trust faster than any technical documentation. Automate your reproducibility checks. I write scripts that rerun the entire analysis pipeline on a test dataset and compare outputs against previously generated results. If coefficients shift by more than thresholds, the pipeline flags it immediately. This catches silent regressions from library updates, data format changes, or accidental modifications to intermediate files. The investment in automation pays off when you need to regenerate results for peer review or regulatory submission.

Document negative findings explicitly. The literature is saturated with successful outbreak analyses. Nobody publishes the three influenza seasons where your model predicted localized surges that never materialized, or the contact tracing network where you identified transmission clusters that turned out to be surveillance artifacts. Negative results matter for methodological honesty and for helping other practitioners avoid your mistakes. I maintain a private repository of failed approaches with explanations of why each failed. It's more valuable than most of my successful analyses.

Tools That Survive Reality Checks

R remains dominant in academic epidemiology for good reasons. Its survival analysis packages are mature, its spatial statistics are comprehensive, and its reproducibility ecosystem is strong. But R struggles with production-scale data processing. I use Python for ETL and database operations, then switch to R for statistical modeling. The handoff between languages introduces version and package compatibility issues, but the workflow is maintainable with virtual environments and dependency management. PostgreSQL with PostGIS handles spatial epidemiology queries efficiently. I've processed millions of geocoded case records without performance degradation. MySQL lacks native spatial indexing that handles complex queries. MongoDB offers flexible schema design but performs poorly on aggregation pipelines that join multiple epidemiological datasets. For this work, relational databases with proper spatial extensions remain the most reliable option. Tableau or Power BI for visualization, but build the data layer in SQL or Python first. Visualization tools that connect directly to raw epidemiological databases produce misleading aggregations because they apply default aggregation logic that may not match your analytical definitions. I connect dashboards to pre-aggregated summary tables that I generate and validate through code. The dashboards display numbers I've already verified against source systems.

For collaborative projects, invest in documentation infrastructure early. A well-maintained README with data source descriptions, variable dictionaries, and analysis assumptions prevents endless email threads where team members ask questions that have been answered in documentation they never read. I use MkDocs with Material theme because it renders markdown nicely, generates search indexes, and deploys to static hosting with zero maintenance. The deployment cost is roughly equivalent to buying coffee for a team meeting, but the reduction in repetitive questions pays for itself within the first week.