Data Science In Financial Services

What Actually Happens When You Build a Model in Finance

The first thing you learn is that data science in financial services is 10% modeling and 90% infrastructure, compliance, and explaining to people who don't trust the model why the model said what it said. Everyone expects the hard part to be the algorithm. It isn't. The hard part is getting clean enough data out of systems that weren't built for analytics and then convincing auditors the thing you built doesn't violate some regulation you've never heard of. I'm going to walk through a practical workflow using credit risk modeling as the concrete example, because that's the closest thing to a canonical project in this space. You can apply the same structure to fraud detection, portfolio optimization, or anything else. The structure holds. The specifics will hurt differently.

Data Ingestion Is Where Projects Die

Before you write a single line of modeling code, you need a dataset that isn't a liability. Financial data lives in places that make data engineers weep. Core banking systems, legacy mainframes, third-party API feeds, manual entry into spreadsheets. The data exists, but it's scattered across systems that were never designed to talk to each other. My standard starting point is to map every data source against the target variable. If you're building a default prediction model, you need payment history, credit utilization ratios, inquiry counts, employment stability, and account age. The problem is that payment history data often comes in monthly statements formatted differently across four or five regional banking platforms. You'll spend two weeks just aligning date formats and resolving missing values that represent actual zeros versus genuinely unknown information. This distinction matters enormously for logistic regression. There's no shortcut here. You either build a proper data pipeline or you end up manually cleaning CSVs every week and pretending it's temporary. The manual path wins sometimes, but only for small projects that never get scaled. If your model ever becomes production-grade, the manual path will sink you within six months.

Feature Engineering That Actually Matters

Raw features from financial databases are almost never useful directly. A transaction amount tells you nothing without context. A balance means nothing without the credit limit. The value lives in derived features. Standard transformations I rely on include ratio features like debt-to-income, utilization rates, and payment-to-balance ratios. Rolling statistics over 30, 60, and 90-day windows capture trend behavior. Time-since-last-activity and recency scores are deceptively powerful predictors. Interaction terms between income level and credit utilization tend to surface non-linear relationships that individual features miss. Here's the counter-intuitive part that people miss: simpler engineered features often outperform complex ones. A well-calculated payment-to-limit ratio will beat a neural network input layer full of raw transaction data 9 times out of 10 in credit risk. The signal is already encoded in the ratio. You're just asking the model to rediscover arithmetic.

I ran into a specific edge case last year where our loan default model was performing great in backtesting but completely failed in production. The feature distributions in the training set looked normal, but the live scoring environment had a completely different distribution for the delinquency_days feature. Something in the data pipeline was truncating negative values, which meant everyone who was slightly behind on payments looked like they were current. The model had never seen negative delinquency in training, so it assigned them confidence scores that made no sense. We caught it because we were monitoring PSI values religiously, but it cost us three weeks of debugging. The fix was a simple data quality check at the ingestion layer that validates sign consistency on numeric fields. I implement that check on every project now, regardless of how obvious the field seems.

Model Selection Without Overthinking It

Gradient boosting dominates this space. XGBoost, LightGBM, CatBoost. They handle tabular financial data better than deep learning approaches for the vast majority of use cases. The interpretability is reasonable with SHAP values, the training speed is acceptable, and the performance is competitive without requiring GPU clusters. Logistic regression still deserves a seat at the table, particularly for models that face regulatory scrutiny. When a borrower is denied credit, you often need to provide adverse action reasons. A logistic regression model with clear coefficients is easier to explain to a compliance team than a black box with 200 interaction terms. The performance gap between logistic regression and gradient boosting on well-engineered financial data is usually within 2-4% AUC. That's a small price for interpretability. I've seen teams waste months building custom neural network architectures for credit scoring. The models performed marginally better than a properly tuned LightGBM baseline. The extra complexity introduced maintenance burdens, longer retraining times, and compliance headaches that outweighed the performance gains. The baseline beats the custom approach most of the time unless you have genuinely non-structural data like images or unstructured text.

Validation That Doesn't Lie to You

Random k-fold cross-validation is the standard beginner approach and it's almost always wrong for financial data. Financial datasets have temporal structure. Customer behavior changes over time. Economic conditions shift. A model trained on 2019 data tested on 2023 data is evaluating something completely different from a model tested on holdout data from the same period. Time-series split validation is the minimum standard. Train on older data, validate on newer data, repeat with expanding windows. This gives you a realistic estimate of how the model will perform when deployed. The performance will almost always be worse than random-split CV, and that's the point. Random-split CV gives you optimistic estimates that disappear the moment you deploy. PSI, or Population Stability Index, is the standard metric for monitoring feature drift between training and production. A PSI above 0.1 indicates moderate drift, above 0.25 indicates significant drift that requires investigation. I calculate PSI weekly for all production models and set up alerts at the 0.2 threshold. This caught another production issue recently where a new loan product launch changed the customer mix dramatically. The model was still technically valid, but the underlying population had shifted enough that recalibration was needed before the next quarterly review.

Deployment Realities

The model you build in a notebook is not the model that ships to production. There's a gap between the two that involves model serialization, scoring API development, latency optimization, and monitoring infrastructure. Most data science teams underestimate this gap by a factor of three. PMML remains the most practical serialization standard for financial services because it's widely supported across banking infrastructure and doesn't tie you to a specific framework. ONNX is gaining traction but adoption in legacy banking systems is inconsistent. Whatever format you choose, document the schema, the expected input types, and the version dependencies. Production environments will break in unpredictable ways if you skip this. Monitoring is where most projects fail after deployment. You need to track prediction distribution shifts, feature PSI, model performance metrics against actual outcomes, and data quality scores at the ingestion layer. I've found that a simple dashboard with PSI trends and prediction histograms catches 80% of production issues before they become incidents. The remaining 20% requires A/B testing frameworks and shadow mode deployments where new models run alongside existing ones without affecting decisions.

The most honest assessment I can give: building a model that works in production is significantly harder than building one that works in a notebook. The gap exists because financial data is messy, regulatory requirements are strict, and infrastructure is often legacy. The models that deliver value are the ones where the team invested equally in data quality, validation rigor, and monitoring. Anything less produces something that looks good on paper and breaks in the real world within six months.