Making models robust for industrial use

Article

Frank Westad, Idletechs AS, June 26^th, 2023

How to make models robust for industrial use

One challenge in modelling and predicting industrial processes is to make the models robust with respect to changes in the environment, process, and initial conditions. Non-robust models can lead to poor prediction accuracy and process monitoring in given situations, and it becomes hard to know if the process is under control.

In this era of deep learning and AI, it is easy to calculate hundreds of models for a given data set by comparing various machine learning methods and tuning the model hyperparameters. Depending on the choice of method(s), these will have various degrees of interpretability and performance.

We advocate for using methods that allow a view into both the sample and variable space for finding sample groups and outliers, making it possible to interpret the partial correlation of the variables/features and confirm findings with domain-specific knowledge, theory and causality.

Nevertheless, there is always a danger of being too optimistic in the training phase, even if one sets aside forty percent of the samples as an independent test set. This becomes especially crucial when repeated thousands of times and one selects the “best” model. Optimistic overfitting with unrealistic prediction results does no one any favors when it fails to deliver in real production conditions.

Illustration: Canva pro

In any process, several sources of variation, both known and unknown, will affect the product quality. Examples of known sources of variation are raw material quality, physical sampling, time (which often has an underlying cause), accuracy and precision of the sensors, and the operator’s tuning of the process based on experience (and gut feeling!).

It is paramount to add all qualitative information, such as batch identifiers, to the numerical data as meta-information. In fact, everything one observes in the process should be jotted down as textural information and later put into relevant categories (“too cold in the production hall”, “more noise in the machine than usual”). This is especially important for unwanted events as it provides a better basis for removing samples in the modelling and prediction phase.

One should never remove samples based on some statistical outlier criteria only because then “the model gets better”.

Illustration: Canva pro

So how can one make the models robust towards future sources of variation or when the model parameters are tweaked?

The answer is merely common sense: validate the model given the different sources of variation. As scientists, we are concerned with repeatability and reproducibility in our reference measurements in the laboratory. However, in most cases, the main reason for a non-robust model/process is all other sources of variation, as exemplified above. Therefore, the model should be validated by stratifying the samples according to qualitative meta-information. In this way, the most important sources of variation can be assessed.

For all real processes, there will be one or more reasons for grouping samples; this is why the thousand times split into training and test set might be too optimistic at the expense of reduced prediction accuracy.

Specific considerations during model validation must be taken for processes with known time dependencies. One example is the prediction of the output effect of wind turbines in three wind farms three days ahead. Assuming that data for ten years have been collected, one may validate at various levels. The options one obviously should NOT do is to random sample and define training and test sets repeatedly.

Some ways to stratify the samples are:

1. Make a model on two wind farms by setting aside farm three as a test set. The two other farms should be validated by keeping one out and predicting the other (segmented cross-validation)
2. Cross-validate the first eight years and predict the two other years.
3. Validate across month
4. Validate across daytime/nighttime.

Note that in all these cases, the results might be too optimistic if the training set is not validated given the actual stratification (farm, year, month, day), although a test set has been defined. 

One challenge in selecting a “sub-optimal” model is that only when applying the model for future scenarios can it be documented that a conservative model is more robust. In discussion with colleagues, it is better that we internally point out these challenges than if our customers question our product quality. As the English would say: “the proof of the model lies in the prediction”.

The use of Design of Experiments (DoE) may give insight into the essential sources of variation and understanding of causality in the process. However, applying DoE in full-scale production by tweaking the process settings may lead to out-of-spec products, which comes at a cost that may hinder the practical use of DoE.

Another approach is to make use of digital twins and simulate the process for future conditions. However, this is a topic for another article.

REFERENCES

[1] Westad, F. and Marini f., 2015. Validation of Chemometric Models – A Tutorial. Anal. Chim. Acta 893 p. 14–24. doi: 10.1016/j.aca.2015.06.056.

[2] Stone, M, 1974.  Cross-Validatory Choice and Assessment of Statistical Predictions. J. R. Stat. Soc. Ser. B Methodol. 36 (2), 111–133. doi:10.1111/j.2517-6161.1974.tb00994.x