Data Analysis Problems And Solutions

When Your Numbers Stop Making Sense

You spend three days cleaning a dataset only to realize the timestamps are in UTC and the business operates in Eastern time. Or you build a model that hits 97 percent accuracy and it completely fails in production because the test set leaked information from the future. Data analysis is mostly just fixing mistakes other people made while pretending everything is fine. The problems people run into fall into a few patterns I have seen repeated endlessly across different industries. The same core issues show up whether you are working with financial transactions, healthcare records, or clickstream data from a mobile app. This is the first wall everyone hits. You get handed a spreadsheet and roughly a third of the rows have missing values, the column headers change mid-document, and the date formats switch between two different styles in the middle of the file. I once spent a full morning tracking down why a churn prediction model kept failing. The issue was a single column where null values had been replaced with the string "N/A" instead of actual nulls. Python treated those as text, not as missing data, so the imputation step completely ignored them. The model was effectively predicting based on a column full of wrong assumptions.

The fix was straightforward once I found it. I ran a dtype check across every column before doing anything else. Anything returning object or mixed types gets flagged immediately. Then I replaced string placeholders like "N/A," "NULL," and "-" with proper null values using simple replacement logic. For missing numerical data, mean imputation works in most cases, but if you have a skewed distribution, median or mode imputation keeps the outliers from warping your results. For categorical data, creating a separate "missing" category often performs better than trying to force the value into an existing group. The deeper problem here is that dirty data is rarely random. The way values go missing tells you something about your system. If a field is missing more often for a certain user segment, that is not noise. That is information. I learned this the hard way when analyzing subscription cancellations. The "reason for leaving" field was blank for 60 percent of churned users. Instead of dropping those rows, I treated the blank as its own category and it turned out to be the single strongest predictor of whether a customer would come back. People who explicitly chose "too expensive" behaved differently from people who just disappeared.

The Time Series Trap

One of the most common mistakes I see is treating time series data like cross-sectional data. You pull three years of daily sales and run a standard train-test split. Your model looks great on paper and then collapses the moment you try to forecast next month. The problem is temporal leakage. By randomly splitting the data, you have given your model information from the future that would not have been available at prediction time. The solution is time-aware splitting. Use the first 80 percent of observations chronologically for training and the last 20 percent for testing. Do not shuffle the data. If you are doing cross-validation, use TimeSeriesSplit from scikit-learn, which creates multiple train-test pairs that respect the temporal order. This usually drops your validation score compared to a random split, but that dropped score is actually more honest and closer to what your model will achieve in production. I worked on a demand forecasting project where the team had built a surprisingly good model using random splits. When we switched to proper time-based validation, the R-squared value dropped from 0.91 to 0.63. That sounded bad until we deployed the model on holdout weeks and confirmed the 0.63 number was closer to reality. The original 0.91 was never going to happen in the real world.

Feature Engineering Without a Map

Beginners often treat feature engineering as a guessing game. They throw every possible transformation at the problem and hope something sticks. That approach generates hundreds of features and very few of them actually help. Worse, it introduces multicollinearity and overfitting that is hard to spot until deployment. The method I use is more deliberate. First, I understand the business question before touching a single column. If you are predicting customer lifetime value, you need features that capture purchase frequency, average order value, recency of interaction, and product category diversity. Stuffing in things like "day of week account was created" rarely adds signal. I start with domain knowledge, then validate each feature's relationship to the target using correlation analysis and mutual information scores. Features that show weak or nonsignificant relationships get dropped early, not after model training. One specific trick that saves a lot of time is using a decision tree as a feature selector. Fit a simple tree on your features and target, then look at the feature importance output. Anything below the 20th percentile of importance can usually go. Trees catch nonlinear relationships that correlation matrices miss. This approach cut my feature count from 340 down to about 45 on a recent customer segmentation project without meaningfully affecting the clustering quality.

Causal Inference vs Correlation

This is the problem that costs companies real money. You see a strong correlation between two variables and the executive team decides to invest heavily in a initiative based on that relationship. Six months later, nothing changed. The correlation was spurious or the causation went the other direction. I ran into this with an e-commerce client who believed that sending a follow-up email after a support ticket increased repeat purchase rates. The correlation was strong. But when I looked at the data more carefully, customers who received follow-up emails were already more engaged. They had higher ticket values and longer account histories. The email was a consequence of engagement, not the cause. We used propensity score matching to create a comparable control group and found the email had virtually no effect on repeat purchases. The real driver was pre-existing customer quality. The practical takeaway is that whenever someone claims X causes Y, ask what would happen if you intervened. Can you run a controlled experiment? If not, look for natural experiments, instrumental variables, or difference-in-differences designs. If none of those are available, present your findings as associations, not causal claims. The data will tell you what it knows, and usually it knows less than you think.

The Visualization Lie

Bad visualizations are not always accidental. Sometimes they are deliberate attempts to make a weak result look strong. A truncated y-axis can turn a two percent change into what looks like a dramatic swing. Stacked bar charts can obscure the actual trend by hiding the component that is driving the change. I have sat in meetings where a slide showed revenue growth that looked incredible until someone pointed out the chart started at $95 million instead of zero. The fix is mostly discipline. Start axes at zero unless you have a specific and justified reason not to. Always label your axes and include the actual numbers, not just tick marks. Use consistent scales across comparative charts. When you show a trend over time, include the absolute values alongside percentages so the viewer can judge the real magnitude of the change. One practical rule I follow is the ink-to-data ratio. If a visual element does not convey information, it should not be there. Gridlines can stay if they help readers extract values. Background colors, decorative icons, and 3D effects on bar charts should go. They add noise, not clarity. A clean line chart with proper labels communicates more in three seconds than a heavily decorated dashboard that takes thirty seconds to decode.

When the Model Works but the Business Does Not Care

The most frustrating problem in data analysis is not technical. It is the gap between what the model predicts and what the stakeholder actually needs. I built a churn prediction model that was genuinely useful. The precision-recall curve was solid, the lift chart showed clear improvement over random guessing, and the business team could have used it to target retention campaigns effectively. The model was rejected because the marketing team did not trust the output. They wanted to see individual customer names, not probability scores, and they wanted explanations they could understand in a two-page summary. The workaround was to add SHAP values to the model output. SHAP explains each prediction by showing which features pushed the probability up or down for that specific customer. It also produces a global view of feature importance. I spent an afternoon building a simple dashboard that showed each flagged customer with their top three drivers and the direction of the effect. Once the team could see why the model made each prediction, trust appeared almost overnight. The model had not changed. The interpretability did. This is the part nobody teaches in introductory courses. A model is only as good as its ability to drive a decision. If the people who need to act on the output cannot understand it or do not trust it, the best model in the world is worthless. Invest as much time in explaining your work as you do in building it.

Performance and Scaling

Most data analysis projects in industry do not hit true big data problems. The datasets are large enough to slow things down on a laptop but small enough to fit in memory. The real performance bottleneck is usually inefficient code, not insufficient hardware. Nested loops over pandas DataFrames, repeated groupby operations, and loading entire CSV files into memory when you only need a subset are the usual suspects. The quick win is to switch from loops to vectorized operations. A groupby-agg on a pandas DataFrame with a custom loop can take ten minutes on a dataset of five hundred thousand rows. The same operation with pandas' built-in aggregation functions runs in under a minute. For larger datasets, consider using Dask or Polars instead of pandas. Polars is particularly fast because it uses parallel processing and lazy evaluation by default. A query that takes thirty seconds in pandas often completes in under three seconds in Polars on the same machine. I encountered a pipeline that was supposed to process one month of transaction data every night. It took four hours to run and frequently failed due to memory errors. After refactoring it to use Polars with chunked reading and parallel aggregation, the same process completed in eighteen minutes. The fix was not a better server. It was better use of the tools already available.

Documentation and Reproducibility

The problem that haunts every data team is the analysis that cannot be reproduced six months later. You spend weeks building a complex preprocessing pipeline, the results look great, and then you hand it off or move to the next project. When someone asks for the model two months later, you cannot remember which version of the library you used, what parameters you tuned, or why you dropped three columns in step four. The work becomes a black box. The minimal viable documentation is not much. Record your environment using a requirements file or conda environment export. Comment your code so that each major step has a one-sentence explanation of what it does and why. Save intermediate outputs at key stages so you can verify each step independently. This takes roughly ten minutes extra per session and prevents hours of reconstruction work later. I keep a simple template notebook for every project. The first cell documents the date, the objective, and the data source. The second cell captures the environment details. Each subsequent section has a header explaining the step and a comment noting any deviations from the standard approach. It sounds tedious until you need to rerun something and your memory is not reliable. Most of my past work I can reconstruct in an afternoon because I wrote the damn notes.