The Distance Between Us

Understanding Distance Metrics in Machine Learning

Distance metrics are one of those things everyone encounters early in data science and then never really thinks about again until something breaks. The Distance Between Us in any given dataset depends entirely on which metric you choose, and picking the wrong one is a common reason models underperform without anyone understanding why. At its core, a distance metric is a function that tells you how far apart two points are in a mathematical space. The most basic version is Euclidean distance, which is just the straight-line distance between two coordinates. You calculate it by subtracting each coordinate, squaring the results, adding them up, and taking the square root. In code that looks roughly like this: That works fine for simple cases, but real data is rarely that clean. I once spent three days debugging a clustering model that kept grouping completely unrelated customers together. The issue was that I was using Euclidean distance on customer data where one feature was annual revenue ranging from 0 to 50 million and another was age ranging from 18 to 95. The revenue feature dominated every distance calculation, making age effectively irrelevant. The workaround was to normalize both features to the same scale using standardization before computing distances. That alone fixed the clustering in about ten minutes.

There are several other metrics worth knowing about, and each one has a different idea of what "near" and "far" mean. Manhattan distance measures distance as if you were navigating a grid, like city blocks. You add up the absolute differences of each coordinate. This is useful when your data represents actual grid-like movement or when you want to penalize diagonal movement. It is less sensitive to outliers than Euclidean distance because it does not square the differences. Cosine similarity does not measure distance in the traditional sense. Instead it measures the angle between two vectors. Two documents can have very different lengths but still point in the same direction if they use similar words. This is why cosine similarity is the default choice for text comparison and recommendation systems. Python makes this straightforward with sklearn:

from sklearn.metrics.pairwise import cosine_similarity

similarity = cosine_similarity([vector_a], [vector_b])[0][0]

Minkowski distance is a generalization that includes both Euclidean and Manhattan as special cases. When the parameter p equals 2, it is Euclidean. When p equals 1, it is Manhattan. For values between 1 and 2, you get something in between. This is handy when you are tuning a model and want to find the sweet spot without switching between two different functions. The metric you pick should match the shape of your data and what you are actually trying to measure. Here is a practical breakdown: One thing beginners often miss is that distance metrics assume your data lives in a metric space, which means the triangle inequality must hold. Most of the common metrics satisfy this, but if you are working with specialized data types like graphs or trees, standard Euclidean or Manhattan distance will give you garbage results. In those cases you need domain-specific distance functions.

If you are building a nearest neighbor search, do not compute all pairwise distances unless your dataset is small. For anything over ten thousand points, use aKD-tree or ball tree structure. Scikit-learn provides these out of the box, and the difference in speed is usually dramatic. A brute-force KNN on fifty thousand ten-dimensional points might take forty seconds. The same query with a ball tree can return results in under two hundred milliseconds. Another practical issue is that distance calculations break down in very high dimensions. As dimensionality increases, the difference between the nearest and farthest points shrinks, and all points start looking equally distant. This is the curse of dimensionality, and it affects everything from clustering to anomaly detection. The usual fix is dimensionality reduction before computing distances. Principal Component Analysis is the most common approach, but for text data, latent semantic indexing or even simple truncation of your vocabulary to the top one thousand terms often works just as well and runs much faster. I ran into this exact problem last year with a semantic search project. We had embeddings in seven hundred sixty-eight dimensions from a transformer model. Nearest neighbor queries were taking too long and the results were getting worse, not better, as we added more training data. The issue was not the model quality, it was the raw dimensionality. We reduced the embeddings to two hundred dimensions using PCA, recomputed the distances, and query time dropped by about sixty percent while accuracy actually improved slightly. The dimensionality reduction removed noise that was confusing the distance metric more than it removed signal.

Common Pitfalls

One of the most frequent mistakes is applying a distance metric without normalizing the data first. If your features have different ranges, the metric will implicitly weight the larger-range features more heavily, regardless of whether those features are actually more important. Always check your feature scales before computing distances. Another pitfall is using Euclidean distance on categorical data that has been one-hot encoded. A one-hot vector has all but one dimension equal to zero, which creates artificial clustering behavior. Categorical features should either be embedded into a lower-dimensional continuous space or handled with a different distance function like Hamming distance, which counts the number of positions at which two binary strings differ. Do not ignore the computational cost of your distance function either. Cosine similarity requires normalization and dot products, which are fine for moderate-sized datasets but can become expensive at scale. Manhattan distance is computationally cheaper per operation but may require more neighboring points to achieve the same recall. There is no free lunch here, and the right tradeoff depends on your constraints.

Summary

The distance between two points in your data is not an objective fact. It is a choice, and that choice shapes every downstream decision your model makes. Start by understanding what your features represent and what kind of neighborhood structure makes sense for your problem. Then pick a metric, normalize your data, and validate the results against a held-out set. If the nearest neighbors do not look reasonable to you visually, the metric is probably wrong, and no amount of tuning will fix that.