What Actually Happens When You Try to Use Machine Learning for Drug Discovery
Most people coming into this field have a very optimistic view of what it can do. They see headlines about AI designing new drugs and assume the pipeline is much simpler than it actually is. The reality is more like a series of broken assumptions and messy data cleanup sessions. I want to walk through how this actually works in practice, the specific tools people use, and where things tend to fall apart. I have spent years working on this, and the gap between what the papers claim and what happens in a real lab is substantial.
Data Science In Drug Discovery
This field sits at the intersection of computational chemistry, statistics, and biological data. The core problem is straightforward: given a disease target, find a molecule that interacts with it in the right way. The hard part is that the search space is roughly 10^60 possible small molecules, and you have maybe a week or two of experimental time per candidate. You need algorithms that can narrow that down without ignoring the molecules that actually work. The data you work with comes from a few sources. Public databases like ChEMBL and PubChem have millions of bioactivity measurements, but they are noisy and biased toward certain types of compounds. Screen results from your own lab are cleaner but far more expensive to generate. Then there are structural datasets from X-ray crystallography and cryo-EM, which are gold standards but require actual wet-lab work to produce.
The Pipeline and How People Build It
A typical workflow starts with target identification, then moves to hit finding, lead optimization, and finally preclinical evaluation. Each stage has different data requirements and different failure modes. For hit finding, the two main approaches are structure-based and ligand-based. Structure-based methods use the 3D shape of the target protein. You run molecular docking simulations to score how well each compound fits in the binding pocket. Ligand-based methods skip the structure entirely and rely on known active molecules to find similar ones. Similarity here usually means structural fingerprints, and the standard metric is the Tanimoto coefficient computed over ECFP4 circular fingerprints. Here is where things get tricky. Docking scores are not predictive of actual binding affinity. The gap between a good docking score and a measurable Kd or IC50 can be enormous, often off by several orders of magnitude. People who learn this the hard way are the ones who try to optimize a docking pipeline without ever validating it against experimental data from their own targets. A practical workaround is to dock a set of known actives and decoys from DUD-E and measure your enrichment factor. If your method cannot enrich actives in the top 1 percent of a screened library, it will not help you in production either.
Get the Full Details
For lead optimization, the game changes completely. You are no longer just looking for any binder. You need potency, selectivity, solubility, metabolic stability, and low toxicity. This is where machine learning models become more useful, particularly quantitative structure-activity relationship models, commonly called QSAR models. These models predict how structural changes to a molecule affect a specific property.
Tools and Implementation Details
The most common toolchain in this space involves RDKit for molecular representations, scikit-learn or XGBoost for modeling, and PyTorch or TensorFlow for deep learning approaches. For structure-based work, AutoDock Vina is the entry-level docking tool, and Glide or GOLD are the industry standards if you have the budget. RDKit can generate ECFP4 fingerprints, compute molecular descriptors, and handle SMILES string manipulation. It is extremely well-documented and free. The learning curve is reasonable if you already know Python. Most of the friction comes from understanding the chemistry, not the code. For QSAR modeling, a basic approach is to take a dataset of known active and inactive compounds, generate fingerprints for each molecule, and train a classification model. Random forests tend to work well here, though gradient boosting often gives a slight edge. The key step that most tutorials skip is the split strategy. If you split your data randomly by compound ID, your model will almost certainly overfit because similar compounds will end up in both training and test sets. You need to split by scaffold, meaning all molecules sharing the same core ring system go into one set or the other. This is a much harder and more realistic evaluation.
I ran into this exact problem early in my career when I built a model that showed 95 percent accuracy on my test set and then failed completely on a real screening set. The issue was that the test set contained analogs of the training compounds, not truly novel scaffolds. Switching to a Bemis-Murcko scaffold split dropped my apparent accuracy to about 72 percent, which was still useful but honestly reflected what the model could do.
Deep Learning Approaches and Their Limits
Graph neural networks have become popular for molecular property prediction. The idea is that molecules are naturally graph structures, so a model that treats atoms as nodes and bonds as edges should learn better representations than one using fixed fingerprints. Models like GCN, GAT, and the more recent Graphormer architectures have shown improved performance on several benchmark datasets. Generative models are another area of intense interest. Variational autoencoders and diffusion models can be trained to generate novel molecular structures with desired properties. The claim is that these models can explore chemical space in ways that traditional screening cannot. The reality is more nuanced. One specific issue with generative models in this context is that they frequently produce molecules that are chemically invalid or synthetically inaccessible. A model might generate a structure that looks perfect on paper but requires a three-step synthesis that does not exist. I encountered this when evaluating several generative model outputs for a kinase target. Out of roughly two hundred generated candidates, only about twelve had reasonable synthetic routes documented in the literature, and perhaps four of those were actually novel structures rather than known compounds. The rest were chemical fantasies.
The workaround I found useful was combining the generative model with a retrosynthetic accessibility score. RDKit has a simple implementation of the SA score, and more sophisticated tools like ASKCOS or RetroSim can predict actual synthetic routes. Filtering generated molecules through a synthetic accessibility gate before any wet-lab testing dramatically improves the hit rate.
ADMET Prediction and Why It Matters More Than You Think
ADMET stands for absorption, distribution, metabolism, excretion, and toxicity. A compound that binds tightly to your target but is rapidly cleared from the body or causes liver toxicity is useless as a drug. This is why ADMET prediction is often the step that kills promising projects. Several public datasets exist for training ADMET models, including Tox21 for toxicity and a collection of pharmacokinetic datasets. The challenge is that these datasets are small by deep learning standards and have significant label noise. A model trained on public ADMET data will rarely perform as well on your proprietary data as you hope. The distributions shift, and your target biology introduces constraints that generic models do not capture. The practical advice here is to treat public ADMET models as first-pass filters, not final arbiters. Use them to rank candidates and eliminate obvious problems, but plan to validate any promising hits with actual assays. Computational models in this area typically achieve ROC AUC values in the 0.70 to 0.85 range, which is useful for triage but insufficient for making go-no-go decisions.
Common Pitfalls That Waste Months
The biggest waste of time I see is when people spend months building elaborate models on stale or inappropriate datasets. Public bioactivity data is heavily biased toward certain target classes and certain chemotypes. If you are working on a rare target with little public data, trying to fine-tune a model trained on GPCR data from ChEMBL will not transfer well. This is a domain shift problem, and there is no easy fix other than generating your own data or using transfer learning approaches that are still largely experimental. Another frequent mistake is optimizing for the wrong metric. In early-stage hit finding, enrichment factor at the top 1 percent is far more important than overall AUC. A model with a decent AUC but poor early enrichment is worthless for screening because you can only test a tiny fraction of your library. Make sure your evaluation metric matches your actual bottleneck, which is usually the number of compounds you can physically synthesize and test per month. There is also the issue of data leakage that is easy to miss. If your dataset contains multiple measurements for the same compound under slightly different conditions, and you do not deduplicate properly, your model will learn patterns from redundant entries rather than generalizable structure-activity relationships. Always check for duplicate SMILES strings and collapse them to a single representative value, typically the geometric mean of repeated measurements.
Getting Started Practically
If you want to build something, start small. Download the ChEMBL database, filter for human protein targets, and pick one target with at least a thousand bioactivity records. Use RDKit to convert the SMILES strings to fingerprints and build a simple random forest classifier. Evaluate with a scaffold split. Compare your results against a baseline of similarity search using Tanimoto similarity alone. This exercise will teach you more about the field than any tutorial that skips the evaluation step. RDKit is available through pip, and the installation is straightforward on most systems. The documentation includes many examples relevant to this work. For docking, AutoDock Vina is free and has good documentation. The process takes about five minutes to install and an hour or two to learn the basics of preparing a protein structure and setting up a grid box.
Data Science In Drug Discovery: What It Actually Gets You
The honest assessment is that data science in this field is a force multiplier, not a replacement for domain expertise. It helps you focus resources on the most promising candidates, but it does not eliminate the fundamental difficulty of the problem. The best results come from people who understand both the computational methods and the underlying chemistry and biology well enough to know when the model is lying to them. The field moves fast, and new architectures appear regularly. What remains constant is that every successful project I have seen shared one trait: the team spent more time on data curation and validation than on model architecture. That is the part nobody writes about in the papers, but it is the part that determines whether your project succeeds or stalls for six months.
