A Second Course In Statistics Regression Analysis

Getting Past the Basics of Regression

Most introductory statistics courses cover OLS regression in a very compressed way. You learn the normal equations, you plug numbers into a calculator, and you get out a coefficient. That foundation is useful, but it leaves you unprepared for the actual work of modeling real data. The second course is where things get honest. I spent three semesters teaching this material at a mid-sized public university, and I can tell you exactly where students break. It is not the algebra. It is the moment they realize that adding one more predictor can silently destroy the interpretability of every other variable in the model. That realization usually hits during week five, right when we start talking about multicollinearity and variance inflation factors.

A Second Course In Statistics Regression Analysis

The structure of a proper intermediate regression course assumes you already know how to read a scatterplot and compute a correlation. What it does not assume is that you have ever cleaned a messy dataset yourself. The gap between textbook examples and real data is enormous, and the course exists to bridge that gap with mathematical rigor rather than hand-waving. Here is what the curriculum actually covers in my experience. You start with multiple regression, which sounds simple until you encounter a situation where two predictors are correlated at r equals 0.92 and your standard errors blow up. Then you move into diagnostic techniques: residual plots, leverage, Cook's distance, the DFBETAS measure. After that comes model selection, which is where most people get lost because stepwise regression is technically available but almost always the wrong choice. I remember a specific project from 2019 when a graduate student brought me a logistic regression model with twelve binary predictors and a McFadden R-squared of 0.41. The model predicted perfectly, which in this context means it was overfitting catastrophically. The training accuracy was 98 percent, the test accuracy dropped to 63 percent, and the coefficients were so large they made the optimization algorithm unstable. We fixed it by applying L1 regularization, specifically the LASSO method, and reducing the predictor set to four variables through cross-validation. The final model had 81 percent test accuracy and coefficients that actually meant something.

This is the kind of practical problem that second-course material forces you to confront. Textbook datasets are clean, balanced, and generously distributed. Real datasets are not. They have missing values that are not missing at random. They have outliers that are actually valid observations. They have categorical variables with rare levels that collapse your degrees of freedom.

The Core Tools You Actually Need

Matrix notation appears early and stays with you. You stop thinking of regression as a collection of separate calculations and start seeing it as a single linear algebra operation. The closed-form solution for the coefficient vector is beta hat equals X transpose X inverse times X transpose y. This formula is elegant, but it is also computationally expensive when X is large, which is why QR decomposition and singular value decomposition exist as practical alternatives. Understanding what happens under the hood matters more than memorizing the formula. When X transpose X is nearly singular, the inverse becomes numerically unstable. Small changes in the data produce massive changes in the coefficients. This is not a theoretical concern. I have seen it happen with as few as fifty observations when two predictors share 97 percent of their variance. The standard error on one coefficient can exceed the estimate itself by an order of magnitude, which means the t-statistic drops below 1 and the variable becomes statistically indistinguishable from noise. Residual analysis is where the diagnostic work happens. A plain residual plot should show no pattern. If you see a funnel shape, heteroscedasticity is present and your standard errors are biased. If you see curvature, the relationship is nonlinear and your linear model is misspecified. If you see a single point with a residual of five standard deviations, you have an outlier that deserves investigation rather than automatic deletion.

Levene's test checks for equality of variance across groups. The Breusch-Pagan test is more directly relevant to regression because it tests whether the variance of the residuals is predictable from the independent variables. Both tests have low power with small samples, so a nonsignificant result does not guarantee homoscedasticity. You need to look at the plots regardless of what the test says.

Model Selection Without the Tricks

Stepwise selection is widely taught and widely abused. It uses the data twice: once to choose the model and once to estimate the coefficients. This double use invalidates the standard error calculations and makes the p-values too small. The effect is most severe when you have many candidate predictors relative to your sample size. With 100 candidates and 200 observations, stepwise regression will routinely produce models that fail replication in entirely predictable ways. Information criteria like AIC and BIC avoid some of these problems but introduce their own assumptions. AIC estimates predictive accuracy on new data and penalizes model complexity with a factor of two times the number of parameters. BIC imposes a heavier penalty that grows with the log of the sample size. Neither criterion guarantees the true model will be selected, and both can fail when the candidate set contains variables that are correlated with the outcome only through mediation rather than direct effect. I use ridge regression and LASSO as my default starting points. Ridge applies an L2 penalty that shrinks coefficients toward zero without setting any exactly to zero. This is useful when you have many correlated predictors and want to stabilize the estimates. LASSO applies an L1 penalty that can set coefficients exactly to zero, which gives you automatic variable selection. The choice between them depends on whether you believe the true model has a few large effects or many small ones.

Elastic net combines both penalties and is available in most statistical software. It is my go-to method when I have more predictors than observations, which happens more often than most people admit. The mixing parameter alpha controls the balance between L1 and L2. Setting alpha to 0.5 usually works well, but you should tune it using cross-validation rather than guessing.

The Problems That Nobody Talks About Enough

Measurement error in the independent variables causes attenuation bias. The estimated coefficient is biased toward zero, and the bias increases with the reliability ratio. If your predictor has a reliability of 0.6, the true effect is approximately 1.67 times larger than what OLS estimates. This is not a minor correction. It changes the interpretation of every coefficient in the model, and most applied researchers ignore it entirely. Omitted variable bias is the other silent killer. If you leave out a variable that is correlated with both the outcome and an included predictor, the included predictor's coefficient absorbs part of the omitted variable's effect. The direction of the bias depends on the sign of the correlation between the omitted variable and the included predictor. I have seen this happen with salary regression when education is included but ability is omitted. The education coefficient is inflated by roughly 15 to 20 percent in most datasets, depending on the correlation between education and unmeasured ability. Endogeneity appears in countless forms. Simultaneity, measurement error, omitted variables, and selection bias all create endogeneity problems that invalidate OLS. The fix is usually instrumental variables, but finding a valid instrument is harder than the textbook examples suggest. A valid instrument must be correlated with the endogenous predictor and uncorrelated with the error term. Both conditions are untestable directly, and researchers often justify instruments based on theoretical reasoning that collapses under scrutiny.

I encountered a particularly ugly case of endogeneity in 2021 when studying the effect of police presence on crime rates. More police obviously respond to higher crime, which creates reverse causality. The naive OLS estimate suggested that more police increased crime, which is obviously wrong but statistically consistent with the bias direction. We used a instruments based on budget cycles from the previous year, which affected current police hiring but not current crime. The IV estimate was half the OLS estimate, which changed the policy recommendation entirely.

What the Course Does Not Cover

Computational statistics has moved fast, and most second-course textbooks have not kept up. Bayesian regression, bootstrapping, and machine learning methods like random forests and gradient boosting are now standard tools in applied research, but they rarely appear in the curriculum. This is a gap that students need to fill themselves if they want to work with real data. Bayesian regression provides a different perspective on the same problem. Instead of point estimates and confidence intervals, it gives you full posterior distributions for every parameter. This is computationally intensive but straightforward with modern MCMC samplers like Stan or PyMC. The results are often more interpretable because you can make direct probability statements about parameters rather than relying on the indirect logic of hypothesis testing. Bootstrap methods provide another fallback when asymptotic theory breaks down. If your sample is small, your residuals are non-normal, or your model is complex, bootstrap confidence intervals are often more accurate than the textbook formulas. The computational cost is usually manageable: ten thousand resamples with modern hardware takes about two minutes for a standard regression model.

The field is not perfect. Regression analysis has well-known limitations that nobody wants to emphasize in an introductory course. Causal inference requires assumptions that are rarely testable. Predictive accuracy degrades when the data generating process changes over time. Model uncertainty is almost never quantified in applied work, which means the reported standard errors are optimistically precise. These limitations do not make regression useless. They make it essential to understand what the method can and cannot do. A second course in statistics regression analysis should teach you both the power and the fragility of the approach, because using regression without understanding its assumptions is worse than not using it at all.