Preclinical MLOps Audit for Biotech Teams

Preclinical AI work keeps moving faster than the controls around it. That is what usually frustrates senior engineering and R&D teams: the model is not the hard part for long. The hard part is proving, week after week, that the model still means what everyone thinks it means after the assay changed, the chemistry shifted, the labels arrived late, or another site quietly used a different protocol .

In practice, the failure is rarely dramatic. A predictor does not explode. It degrades. It keeps returning numbers, but the numbers stop matching reality in ways that are easy to miss until someone makes a bad decision with them. That is how model rot turns useful signals toxic.

What changed in the last week

The pressure in biotech MLOps still concentrates around the same weak points. Model versions drift away from the data they were trained on. Retraining gets delayed until the prediction starts lying. And every discovery group keeps its own copy of the pipeline, so the same work gets rebuilt with slightly different assumptions.

That is the audit target. Not just whether the model runs. Whether the model still survives contact with fresh assay data, new chemistry, changed protocol settings, or a new batch effect from another site. Once that meaning slips, teams usually do not notice through a clean failure. They notice when downstream people stop trusting the output, or worse, keep trusting it for too long.

The layer stack that matters

The useful MLOps stack for preclinical AI is simple enough to describe and hard enough to keep clean in reality. You need a versioned data layer, a model build layer, a validation layer, a deployment layer, and a monitoring layer that can decide when to freeze or retrain .

The data layer is where ADMET predictors live or die. If assay panels shift, if compound libraries change, if labels come from a different vendor, the feature space moves. Version every source, every transformation, every feature definition. If a descriptor changes, that is a new model world. If you cannot reconstruct the exact inputs, you do not have an audit trail. You have a story.

The build layer should lock code, data, and environment together. Same package versions. Same container image. Same random seeds where possible. Same feature recipe. Same training window. If one discovery team trains in a notebook and another retrains from a copied script, drift starts at the keyboard.

The validation layer needs more than scorecards. For ADMET, you want calibration, not just ranking. You want assay level checks, scaffold splits, and holdouts that reflect the chemistry you will actually see next week, not just the easiest chemistry from last month. If the model only looks good on the old distribution, it is already starting to rot.

The deployment layer in preclinical settings is usually lighter than clinical, but discipline still matters. The serving artifact needs a version. The feature contract needs a version. The prediction output needs a timestamp and lineage. Without that, nobody can trace which model said what about which compound and why.

The monitoring layer is the guardrail. It watches input shift, output shift, calibration decay, and label delay. In preclinical work, label delay is common, so you often cannot wait for final assay truth. That means you also track proxy signals. If the model confidence changes fast while the chemistry mix stays stable, that is a warning. If the chemistry mix changes fast and confidence stays weirdly steady, that is another warning.

Versioning drift detection and retraining triggers for ADMET predictors

ADMET models are especially fragile because their targets are noisy, sparse, and often stitched together from different experiments. That makes versioning non negotiable.

At minimum, version the assay source, the raw compound set, the featurization code, the descriptor dictionary, the training split rule, the model artifact, and the evaluation set. If any one of those changes, you do not compare old performance to new performance as if nothing happened.

Drift detection should focus on three things.

First, input drift. Are the molecular properties, fingerprints, or assay covariates changing shape? If yes, the model may be seeing a new chemical universe.

Second, output drift. Are predicted permeability, clearance, solubility, or toxicity scores shifting in distribution? If yes, the model may be losing its grip.

Third, performance drift. When delayed ground truth arrives, is the error rising, is calibration worsening, or is rank order failing on the newer series?

Retraining should trigger only when drift crosses a threshold that was agreed in advance . Not when someone gets nervous. Not when a project lead wants a new run because the first answer was inconvenient. The trigger should be tied to measurable decay, such as a sustained drop in calibration, a jump in residual error on fresh chemistry, or a confirmed shift in assay behavior after a protocol change.

For ADMET, retraining should also respect data maturity. If the new labels are too thin or too biased, a retrain can make things worse. Sometimes the right move is to freeze the model, widen monitoring, and wait for enough signal. A rushed retrain often just teaches the model a new version of the same mistake.

Why federation across discovery teams breaks

This is where the pain gets real. Discovery teams want speed. Governance wants consistency. The two often collide in the model registry.

When every team builds its own predictors, the same endpoints get redefined in ten slightly different ways. One team uses one permeability assay. Another team uses a cleaned merge of three sources. A third team drops compounds with missing values in a different way. Then everyone calls it a shared ADMET model.

It is not shared. It is fragmented.

Federating models across teams fails when there is no common feature store, no common label standard, and no common release gate . Teams end up copying code instead of sharing artifacts. The cost is invisible at first. Then the same bug, the same leak, the same feature mismatch shows up in three places. The platform gets bigger, but the trust gets smaller.

The fix is boring and necessary. Shared definitions. Shared lineage. Shared validation rules. Shared registry entries. Shared monitoring. Teams can still own their models, but they should not each invent the basics of versioning and drift from scratch.

GPU provisioning and budget pain

Preclinical AI teams keep running into the same infrastructure trap. They need GPUs for training, screening, and sometimes inference, but they do not have the budget to leave hardware idle .

The pain starts with provisioning. If GPU environments take days to spin up, teams work around them with local machines, borrowed notebooks, and hidden cloud spend. Then the bill lands later and nobody can explain it.

The clean way is to keep compute elastic and disposable. Use short lived GPU workers. Keep images small. Cache only what saves real time. Stop long running jobs from hogging expensive accelerators. Move smaller experiments to cheaper hardware when possible. Not every model needs the biggest card in the room.

Budget control also depends on queue discipline. If every team can launch GPU jobs whenever they want, waste follows. If access is too tight, science slows down and people go rogue. The middle ground is a shared scheduler with clear quotas, visible usage, and automatic shutdowns on idle resources.

The real cost killer is not just GPU price. It is repeated training because the environment was not reproducible, repeated runs because the dataset was not versioned, and repeated troubleshooting because no one knows which container built which model.

The failure mode is model rot

Model rot is the quiet failure that matters most. The predictor keeps returning numbers, but those numbers get less and less safe to trust.

In preclinical AI, rot shows up as stale assay behavior, bad calibration, changed label quality, drifted chemistry space, and retraining that lags behind reality. Once that happens, predictions turn toxic because teams keep acting on them as if they still carry the same meaning.

That is why governance has to scale with the models. Not after the fact. Not as a review stamp. It has to be part of the build. If you cannot trace a model from data source to feature version to artifact to prediction, then you cannot tell whether the output is still alive.

The hard truth is simple. In preclinical biotech, fast models are easy. Trusted models are hard. The teams that do well are the ones that keep the stack boring enough to audit and strict enough to survive contact with real data.

If this matches what you are seeing in your own stack, it is usually worth comparing notes with peers who have already hit the same wall. The patterns are more common than people admit, and the useful fixes are often less glamorous than the failure modes.