Every enterprise technology conversation now arrives, sooner or later, at the same fork: keep paying a frontier lab per token, or train something smaller and owned — a “sovereign” model, tuned to the company’s own data and workflows, cheaper to run and immune to a vendor’s roadmap changes. The economics are obviously attractive once volume is high enough. What’s less obvious is what has to be true before that bet pays off.
Brian’s team’s actual day-to-day problem was not “we need a model.” It was that the same inventory item, the same compliance exception, the same customer request got described five different ways by five different teams — buying, sales, operations, procurement — each with their own vocabulary and their own ad hoc spreadsheet. Before anyone could build an intelligence layer that flagged a mispriced item or a customs conflict, someone had to first agree on what a “correctly described item” even looked like across departments.
That ordering matters more than it sounds. A predictive system, or a future in-house model, trained on inconsistent, undifferentiated process data will just learn the inconsistency. It will get very good at predicting chaos. The fix isn’t a smarter model — it’s a legible workflow underneath it.
The useful frame here is “crystallization”: taking a workflow that currently lives in people’s heads, in Slack threads, in a dozen slightly different spreadsheets, and turning it into a standardized, observable process with consistent inputs and outputs. Only once a workflow is crystallized does it become a candidate for automation at all — and only once it’s automated at scale does it generate the kind of clean, task-specific data that makes distilling a smaller model worthwhile.
This is a familiar idea from software engineering, where teams have used similar clustering techniques to find the repeatable shape inside messy debugging workflows before automating them. Supply-chain operations are not so different: underneath the apparent chaos of exceptions and edge cases, there’s usually a smaller number of recurring patterns than anyone believes, once someone bothers to name them consistently.
The order of operations, then, looks like this:
Skip straight to step four and you get an expensive way to automate your own confusion.
It gets skipped because standardization is organizationally hard and technically boring, while “let’s fine-tune a model” is exciting and fundable. Crystallizing a workflow means sitting with procurement and sales until they agree on shared vocabulary — unglamorous, slow, political. Buying a model or a platform feels like progress in a way that a shared glossary doesn’t, even when the glossary is the thing actually blocking value.
There’s also a sequencing trap specific to compliance-heavy domains like cross-border trade: the intelligence an enterprise most wants — flagging a banned material before it ships, catching a pricing error before it’s booked — only works if the underlying data about “what this item is” and “where it’s going” is already consistent. Compliance and predictive-analytics layers are downstream of standardization, not a substitute for it, no matter how good the model underneath them is.
If you’re an operating team evaluating whether to build an in-house model this year, the honest first question isn’t which model or which vendor. It’s whether the workflow you want to automate is described the same way by everyone who touches it. If it isn’t, the crystallization work — tedious, cross-functional, unglamorous — is the actual project. The model comes after, almost as a formality: a way of capturing, cheaply and repeatably, a pattern that by then is already understood.
Brian’s team is still early in this — the standardization work is still in progress, the analytics layer still being scoped, the question of an owned model still a “later” conversation rather than a “now” one. That’s the right order. I’d be skeptical of anyone telling a similar story in reverse.
]]>The objective was never to ship a product. It was to find out whether an explainable, honest foundation could be built at all: a system that shows its evidence, states its uncertainty, and refuses to answer when it has no basis to. That turned out to be a harder and more interesting problem than adding a model.
A generator fault is usually discovered in one of two ways: an alarm trips, or the machine stops. Both are late. By the time coolant temperature crosses a shutdown threshold, whatever caused it has been developing for days or weeks.
The obvious fix — set the threshold lower — does not work, and the reason is worth sitting with. A fixed threshold assumes a reading means the same thing in every condition, and it doesn’t. Ninety-two degrees of coolant temperature at ninety percent load on a hot afternoon is unremarkable. The same ninety-two degrees at twenty percent load on a cool morning means something is wrong with the cooling system. Lower the threshold enough to catch the second case and you generate nuisance alarms across every hot afternoon until someone disables it. That is how condition monitoring dies in practice: not from being wrong, but from being ignored.
There is a second problem, quieter but just as limiting. Engineers working on these machines need answers from product documentation — service intervals, sensor locations, alarm meanings — and they need those answers to be traceable to a source. An assistant that produces a confident, plausible, unsourced number is worse than no assistant, because someone will act on it.
Both problems have the same shape. The useful system is not the one that produces the most output. It is the one whose output you can check.
The prototype is a pipeline. Each stage produces something the next stage can use, and each stage can be inspected on its own.
Engineering knowledge base
A component model of the machine — 14 systems, roughly 260 sub-components, the failure physics and how faults propagate between subsystems. Every statement carries a provenance tag recording where it came from.
Cross-linked registers
164 telemetry signals, 116 alarms, 65 failure modes scored for risk, and 46 maintenance tasks. They reference each other, and the build fails if a reference does not resolve.
Telemetry quality checks
Nine validation checks — gaps, stuck sensors, out-of-range values — run before anything else. A model fed bad data still produces confident numbers, which is the failure mode worth guarding.
Physics residuals and normalisation
Each signal is compared against what it should be at the current load, ambient temperature and — for oil pressure — oil viscosity. The residual, not the raw value, is what gets modelled.
Anomaly detection and fault typing
An isolation forest scores the whole residual vector and reports which signals drove the score. A classifier attempts to name which fault mode it resembles.
Risk, diagnosis and trend indication
Attribution maps to subsystems, subsystems map to the failure-mode register, and time-to-threshold is estimated only where a genuine monotonic trend exists.
Everything is written in the Python standard library with no external dependencies — no scikit-learn, no charting library. That sounds like a strange constraint until you see the deployment target: a gateway box sitting next to a generator inside a plant network, where installing a package is a procurement conversation rather than a command.
I want to be careful here, because this is the part of the project most easily overstated.
What exists is a provenance discipline. Every factual statement in the knowledge base carries a tag recording its basis:
This is not decoration. It is the difference between a knowledge base and a plausible-sounding summary, and it is what makes the rest of the work auditable — anyone can ask “where did this number come from?” and get an answer rather than a shrug.
What does not exist is a document-retrieval layer. There is no search over manufacturer documents, no citation-answering assistant, no semantic index. I wrote the ingestion script for it. It has never run, for a simple reason: no service manual was ever supplied to the project.
That gap propagated everywhere. Alarm setpoints, maintenance intervals, controller register addresses — all of them are currently engineering estimates rather than manufacturer values, and every one of them is tagged as such. It has been the project’s gating dependency since the first week.
An assistant that admits “the source does not contain this information” is safer than one that invents an answer. The same applies to the engineer building it — the honest move was to tag 164 signals as unverified rather than fill them in from plausible-looking numbers.
I could have built the retrieval layer against public documents and called it done. I decided that a citation system pointed at sources that don’t actually specify this machine would be worse than no citation system, because it would look authoritative. That call is still the one I’d defend most strongly.
The core idea is small enough to state in a sentence: compare what the machine is doing against what it should be doing at this operating point, and model the difference.
Take coolant temperature again. Instead of asking “is this above 98 degrees?”, the pipeline asks “given the current load and the ambient temperature, what should this reading be — and how far off is it?” A residual near zero means the machine is behaving as its conditions predict. A residual drifting upward over days means something is degrading, even while the absolute value stays comfortably inside the alarm band.
A few consequences follow from building it this way.
The evidence is interpretable. When the system flags something, it can say which residual moved and in which direction, which maps to a subsystem, which maps to a documented failure mode. That chain is inspectable end to end. An alert that says “anomaly score 0.83” and nothing else gets ignored after its second false alarm.
Slow degradation shows up in the trend, not the reading. Rolling averages and slopes over hours and days catch things a single sample cannot.
Time-to-threshold is only estimated where it means something. Remaining-life estimation requires a monotonic degradation path, a defined threshold, and an observable signal. Most components on a generator fail at least one of those tests, so the system does not attempt an estimate for them. Of the failure modes in the register, 44 of 65 look tractable for machine learning at all — and considerably fewer than that qualify for a remaining-life number.
Normalisation has to be fitted on clean data only. The expected-value models are fitted on a verified-healthy baseline window. Fit them across the whole dataset and the faults get absorbed into the definition of “expected”, the residuals go flat, and the pipeline detects nothing while appearing to work perfectly. That failure mode is silent, which is what makes it dangerous.
Everything below was measured on a synthetic dataset: 30 days at one-minute resolution, 43,200 rows, with eight fault scenarios injected at known times. These results demonstrate that the software pipeline and its logic work. They do not demonstrate performance on a real generator.
There is a circularity that has to be stated plainly: the synthetic signals were generated from the same documented physics the feature engineering assumes. A model that scores well here has learned what I encoded into the data. It has not discovered anything about a real machine.
With that said, the comparisons are still informative — and several of them contradicted what I expected going in.
A control chart beat both machine-learning detectors. I compared an isolation forest and an autoencoder against plain four-sigma control limits on the same normalised residuals.
| Detector | Episodes found | Mean detection lag | False-alarm runs |
|---|---|---|---|
| Isolation forest | 2 / 3 | 2.84 d | 0 |
| Autoencoder | 3 / 3 | 0.67 d | 6 |
| Four-sigma control limits | 3 / 3 | 0.50 d | 2 |
The reason is instructive rather than disappointing. Once the residuals are properly conditioned on load, ambient and viscosity, a single-signal limit works very well — because the normalisation has already removed the confounding that a multivariate model exists to handle. The feature engineering was doing the work and the model layer was collecting the credit. I kept the isolation forest in the pipeline, but for attribution and its lower false-alarm rate, not for lead time.
A three-coefficient physics model beat the statistical estimators on remaining life. For the one consumable that genuinely qualifies for a remaining-life estimate, I compared linear extrapolation, an ARIMA model, and a parametric fit that encodes a known physical shape — filter restriction accelerates as the clean area shrinks, so the loading curve is convex.
| Estimator | Mean absolute error | No estimate produced |
|---|---|---|
| Linear extrapolation | 95.1 h | 23 / 72 |
| ARIMA | 89.5 h | 16 / 72 |
| Physics-informed fit | 13.7 h | 0 / 72 |
All three estimators are biased in the same direction — they predict more remaining life than there is, which schedules maintenance late. On a curve that accelerates, any locally-linear extrapolation has to do that.
The confidence interval failed. The main argument for using ARIMA was that it produces a confidence range, which is what turns a trend into a schedulable date. Measured against the known answer, its nominal ninety-percent interval covered none of the true values. The cause is structural rather than a tuning problem: the band represents random variance, while the error here is a systematic bias in one direction. So the prototype returns the estimate and withholds the upper bound rather than publishing a range it cannot support.
The classifier failed a test I built specifically to catch it. The synthetic data contains a deliberate trap: two faults that both pull oil pressure down. One is genuine bearing wear. The other is the pressure sensor drifting, with nothing physically wrong. The only discriminator is temperature dependence — real wear worsens as oil thins; a sensor offset does not care.
The classifier assigned 83.2% of the pure sensor fault to genuine bearing wear. Acted on, that means opening an engine because a transducer drifted. The asymmetry is telling: real wear was never mistaken for sensor drift, so the model had learned a one-way rule rather than the physics.
I did not treat that as a number to improve later. The prediction endpoint for that target now refuses to answer, returning the measurement as its reason and “obtain oil-analysis labels” as the remedy. It seemed better for the system to decline than to be confidently wrong once and distrusted afterwards.
Some smaller findings. A random forest outperformed gradient boosting on fault typing (0.843 against 0.764 macro-F1). Two features that only ever increase — accumulated running hours and thermal life — turned out to be a clock in disguise: they consumed 44.7% of the model’s decision capacity and made it measurably worse, so they were removed. And a 23-test suite caught two bugs that would never have thrown an error, only produced quietly wrong numbers.
None of the following are hedges added after the fact. Each one shaped a design decision.
Ordered roughly by how much each one unblocks.
Only the last of these is a modelling task. That ordering is deliberate.
The part of this project I found most valuable was not the machine learning. Measured honestly, the machine learning came third — behind the normalisation that made the residuals meaningful, and behind a physics model with three coefficients that beat every statistical estimator I put against it.
What I think was actually worth building was a system that can show its evidence. Every number traces to a source or is tagged as an assumption. Every model result carries the protocol that produced it. Three prediction targets refuse to answer at all, and each says why and what would fix it. When a measurement contradicted something I had written down earlier, the write-up says so rather than quietly dropping the earlier claim.
None of that makes the prototype useful on a real machine yet — it plainly is not, and the limitations section above is longer than the results section for good reason. But I would rather hand over a system that is clear about its own boundaries than one that produces confident numbers nobody can check. In this domain, the second kind gets switched off within a month, and the person who built it never finds out why.
]]>