Building an Explainable Predictive-Maintenance Prototype for Industrial Generators

Over the past few months I built a predictive-maintenance prototype for industrial diesel generator sets. This post is a record of what it does, what I actually measured, and — the part I think matters more — everything it still cannot claim.

The objective was never to ship a product. It was to find out whether an explainable, honest foundation could be built at all: a system that shows its evidence, states its uncertainty, and refuses to answer when it has no basis to. That turned out to be a harder and more interesting problem than adding a model.

The problem

A generator fault is usually discovered in one of two ways: an alarm trips, or the machine stops. Both are late. By the time coolant temperature crosses a shutdown threshold, whatever caused it has been developing for days or weeks.

The obvious fix — set the threshold lower — does not work, and the reason is worth sitting with. A fixed threshold assumes a reading means the same thing in every condition, and it doesn’t. Ninety-two degrees of coolant temperature at ninety percent load on a hot afternoon is unremarkable. The same ninety-two degrees at twenty percent load on a cool morning means something is wrong with the cooling system. Lower the threshold enough to catch the second case and you generate nuisance alarms across every hot afternoon until someone disables it. That is how condition monitoring dies in practice: not from being wrong, but from being ignored.

There is a second problem, quieter but just as limiting. Engineers working on these machines need answers from product documentation — service intervals, sensor locations, alarm meanings — and they need those answers to be traceable to a source. An assistant that produces a confident, plausible, unsourced number is worse than no assistant, because someone will act on it.

Both problems have the same shape. The useful system is not the one that produces the most output. It is the one whose output you can check.

What the prototype includes

The prototype is a pipeline. Each stage produces something the next stage can use, and each stage can be inspected on its own.

Everything is written in the Python standard library with no external dependencies — no scikit-learn, no charting library. That sounds like a strange constraint until you see the deployment target: a gateway box sitting next to a generator inside a plant network, where installing a package is a procurement conversation rather than a command.

Citations, and the layer I did not build

I want to be careful here, because this is the part of the project most easily overstated.

What exists is a provenance discipline. Every factual statement in the knowledge base carries a tag recording its basis:

This is not decoration. It is the difference between a knowledge base and a plausible-sounding summary, and it is what makes the rest of the work auditable — anyone can ask “where did this number come from?” and get an answer rather than a shrug.

What does not exist is a document-retrieval layer. There is no search over manufacturer documents, no citation-answering assistant, no semantic index. I wrote the ingestion script for it. It has never run, for a simple reason: no service manual was ever supplied to the project.

That gap propagated everywhere. Alarm setpoints, maintenance intervals, controller register addresses — all of them are currently engineering estimates rather than manufacturer values, and every one of them is tagged as such. It has been the project’s gating dependency since the first week.

I could have built the retrieval layer against public documents and called it done. I decided that a citation system pointed at sources that don’t actually specify this machine would be worse than no citation system, because it would look authoritative. That call is still the one I’d defend most strongly.

Expected versus actual

The core idea is small enough to state in a sentence: compare what the machine is doing against what it should be doing at this operating point, and model the difference.

Take coolant temperature again. Instead of asking “is this above 98 degrees?”, the pipeline asks “given the current load and the ambient temperature, what should this reading be — and how far off is it?” A residual near zero means the machine is behaving as its conditions predict. A residual drifting upward over days means something is degrading, even while the absolute value stays comfortably inside the alarm band.

A few consequences follow from building it this way.

The evidence is interpretable. When the system flags something, it can say which residual moved and in which direction, which maps to a subsystem, which maps to a documented failure mode. That chain is inspectable end to end. An alert that says “anomaly score 0.83” and nothing else gets ignored after its second false alarm.

Slow degradation shows up in the trend, not the reading. Rolling averages and slopes over hours and days catch things a single sample cannot.

Time-to-threshold is only estimated where it means something. Remaining-life estimation requires a monotonic degradation path, a defined threshold, and an observable signal. Most components on a generator fail at least one of those tests, so the system does not attempt an estimate for them. Of the failure modes in the register, 44 of 65 look tractable for machine learning at all — and considerably fewer than that qualify for a remaining-life number.

Normalisation has to be fitted on clean data only. The expected-value models are fitted on a verified-healthy baseline window. Fit them across the whole dataset and the faults get absorbed into the definition of “expected”, the residuals go flat, and the pipeline detects nothing while appearing to work perfectly. That failure mode is silent, which is what makes it dangerous.

What I tested

Synthetic prototype evaluation — not field validation

Everything below was measured on a synthetic dataset: 30 days at one-minute resolution, 43,200 rows, with eight fault scenarios injected at known times. These results demonstrate that the software pipeline and its logic work. They do not demonstrate performance on a real generator.

There is a circularity that has to be stated plainly: the synthetic signals were generated from the same documented physics the feature engineering assumes. A model that scores well here has learned what I encoded into the data. It has not discovered anything about a real machine.

With that said, the comparisons are still informative — and several of them contradicted what I expected going in.

A control chart beat both machine-learning detectors. I compared an isolation forest and an autoencoder against plain four-sigma control limits on the same normalised residuals.

The reason is instructive rather than disappointing. Once the residuals are properly conditioned on load, ambient and viscosity, a single-signal limit works very well — because the normalisation has already removed the confounding that a multivariate model exists to handle. The feature engineering was doing the work and the model layer was collecting the credit. I kept the isolation forest in the pipeline, but for attribution and its lower false-alarm rate, not for lead time.

A three-coefficient physics model beat the statistical estimators on remaining life. For the one consumable that genuinely qualifies for a remaining-life estimate, I compared linear extrapolation, an ARIMA model, and a parametric fit that encodes a known physical shape — filter restriction accelerates as the clean area shrinks, so the loading curve is convex.

All three estimators are biased in the same direction — they predict more remaining life than there is, which schedules maintenance late. On a curve that accelerates, any locally-linear extrapolation has to do that.

The confidence interval failed. The main argument for using ARIMA was that it produces a confidence range, which is what turns a trend into a schedulable date. Measured against the known answer, its nominal ninety-percent interval covered none of the true values. The cause is structural rather than a tuning problem: the band represents random variance, while the error here is a systematic bias in one direction. So the prototype returns the estimate and withholds the upper bound rather than publishing a range it cannot support.

The classifier failed a test I built specifically to catch it. The synthetic data contains a deliberate trap: two faults that both pull oil pressure down. One is genuine bearing wear. The other is the pressure sensor drifting, with nothing physically wrong. The only discriminator is temperature dependence — real wear worsens as oil thins; a sensor offset does not care.

The classifier assigned 83.2% of the pure sensor fault to genuine bearing wear. Acted on, that means opening an engine because a transducer drifted. The asymmetry is telling: real wear was never mistaken for sensor drift, so the model had learned a one-way rule rather than the physics.

I did not treat that as a number to improve later. The prediction endpoint for that target now refuses to answer, returning the measurement as its reason and “obtain oil-analysis labels” as the remedy. It seemed better for the system to decline than to be confidently wrong once and distrusted afterwards.

Some smaller findings. A random forest outperformed gradient boosting on fault typing (0.843 against 0.764 macro-F1). Two features that only ever increase — accumulated running hours and thermal life — turned out to be a clock in disguise: they consumed 44.7% of the model’s decision capacity and made it measurably worse, so they were removed. And a 23-test suite caught two bugs that would never have thrown an error, only produced quietly wrong numbers.

Limitations I designed around

What I would do next

Ordered roughly by how much each one unblocks.

• Obtain the Operation and Maintenance manual, and replace every estimated interval and setpoint with a cited value.
• Obtain the controller register map, and establish which of the 164 signals are actually readable.
• Connect one instrumented generator and begin collecting data, even before any modelling.
• Collect a verified-healthy baseline period, and refit the normalisation models on real behaviour rather than assumed physics.
• Review the fault signatures with a qualified service engineer — the assumptions most likely to be wrong are the ones I have least ability to check myself.
• Gather alarm and service history before attempting any supervised learning; without labelled real failures, supervised models have nothing to learn from.
• Add a calibrated confidence interval to the remaining-life estimator, since the current one demonstrably does not hold.

Only the last of these is a modelling task. That ordering is deliberate.

Closing reflection

The part of this project I found most valuable was not the machine learning. Measured honestly, the machine learning came third — behind the normalisation that made the residuals meaningful, and behind a physics model with three coefficients that beat every statistical estimator I put against it.

What I think was actually worth building was a system that can show its evidence. Every number traces to a source or is tagged as an assumption. Every model result carries the protocol that produced it. Three prediction targets refuse to answer at all, and each says why and what would fix it. When a measurement contradicted something I had written down earlier, the write-up says so rather than quietly dropping the earlier claim.

None of that makes the prototype useful on a real machine yet — it plainly is not, and the limitations section above is longer than the results section for good reason. But I would rather hand over a system that is clear about its own boundaries than one that produces confident numbers nobody can check. In this domain, the second kind gets switched off within a month, and the person who built it never finds out why.