A hybrid science-guided machine learning approach for modeling chemical processes: A review

This study presents a broad perspective of hybrid process modeling combining the scientific knowledge and data analytics in bioprocessing and chemical engineering with a science-guided machine learning (SGML) approach. We divide the approach into two major categories: ML complements science, and science complements ML. We review the literature relating to the hybrid SGML approach, and propose a systematic classification of hybrid SGML models. For applying ML to improve science-based models, we present expositions of direct serial and parallel hybrid modeling and their combinations, inverse modeling, reduced-order modeling, quantifying uncertainty in the process and even discovering governing equations of the process model. For applying scientific principles to improve ML models, we discuss the science-guided design, learning and refinement. For each subcategory, we identify its requirements, strengths, and limitations, together with their published and potential applications. We also present several examples to illustrate different hybrid SGML methodologies for modeling chemical processes.     

化工过程的智能混合建模方法及应用

随着人工智能技术和配套数据系统的快速发展,化工过程建模技术达到了新的高度,将多个机理模型和数据驱动模型以合理的结构加以组合的智能混合建模方法,可以综合利用化工过程的第一性原理及过程数据,结合人工智能算法以串联、并联或者混联的形式解决化工过程中的模拟、监测、优化和预测等问题,建模目的明确,过程灵活,形成的混合模型有着更好的整体性能,是近年来过程建模技术的重要发展趋势。本文围绕近年来针对化工过程的智能混合建模工作进行了总结,包括应用的机器学习算法、混合结构设计、结构选择等关键问题,重点论述了混合模型在不同任务场景下的应用。指出混合建模的关键在于问题和模型结构的匹配,而提高机理子模型性能,获取高质量宽范围的数据,深化对过程机理的理解,形成更有效率的混合建模范式,这些都是现阶段提高混合建模性能的研究方向。     

Integration of machine learning and first principles models

Model building and parameter estimation are traditional concepts widely used in chemical, biological, metallurgical, and manufacturing industries. Early modeling methodologies focused on mathematically capturing the process knowledge and domain expertise of the modeler. The models thus developed are termed first principles models (or white-box models). Over time, computational power became cheaper, and massive amounts of data became available for modeling. This led to the development of cutting edge machine learning models (black-box models) and artificial intelligence (AI) techniques. Hybrid models (gray-box models) are a combination of first principles and machine learning models. The development of hybrid models has captured the attention of researchers as this combines the best of both modeling paradigms. Recent attention to this field stems from the interest in explainable AI (XAI), a critical requirement as AI systems become more pervasive. This work aims at identifying and categorizing various hybrid models available in the literature that integrate machine-learning models with different forms of domain knowledge. Benefits such as enhanced predictive power, extrapolation capabilities, and other advantages of combining the two approaches are summarized. The goal of this article is to consolidate the published corpus in the area of hybrid modeling and develop a comprehensive framework to understand the various techniques presented. This framework can further be used as the foundation to explore rational associations between several models.     

发酵过程生物量软测量技术的研究进展

生物量是发酵过程中的关键过程参数之一,它直接影响着发酵过程的优化和控制。综述了近年来发酵过程生物量软测量技术的研究现状,讨论了基于过程机理分析、回归分析、状态估计和神经网络等的软测量建模方法,对基于神经网络和改进的神经网络建模方法进行了分析。指出基于多尺度建立软测量混合模型,是实现发酵过程生物量在线测量的有效方法,并给出了建立混合模型需要解决的关键问题。     

Hybrid semi-parametric modeling in process systems engineering: Past,present and future

Hybrid semi-parametric models consist of model structures that combine parametric and nonparametric submodels based on different knowledge sources. The development of a hybrid semi-parametric model can offer several advantages over traditional mechanistic or data-driven modeling, as reviewed in this paper. These advantages, such as broader knowledge base, transparency of the modeling approach and cost-effective model development, have been widely recognized, not only in academia but also in the industry.In this paper, the most common hybrid semi-parametric modeling and parameter identification techniques are revisited. Applications in the areas of (bio)chemical engineering for process monitoring, control, optimization, scale-up and model-reduction are reviewed. It is outlined that the application of hybrid semi-parametric techniques does not automatically lead into better results but that rational knowledge integration has potential to significantly improve model-based process operation and design.     

A Modular Process Modeling Tool for the Analysis of Energy Use and Cost in the Pulp and Paper Industry

A novel process modeling tool to facilitate the study of how various process changes in the paper dryer section affect the mill-wide energy system has been developed. A model library with steady-state block models describing paper dryers, heat recovery equipment, and auxiliary systems on component level have been developed using the software Extend. Process models are then created from these block models using graphical programming. In this article, the characteristics of the developed tool are described and a case study where the tool is shown to be useful for analyzing the energy performance of a hybrid dryer section is presented.     

A Modular Process Modeling Tool for the Analysis of Energy Use and Cost in the Pulp and Paper Industry

A novel process modeling tool to facilitate the study of how various process changes in the paper dryer section affect the mill-wide energy system has been developed. A model library with steady-state block models describing paper dryers, heat recovery equipment, and auxiliary systems on component level have been developed using the software Extend. Process models are then created from these block models using graphical programming. In this article, the characteristics of the developed tool are described and a case study where the tool is shown to be useful for analyzing the energy performance of a hybrid dryer section is presented.     

The Potential of Hybrid Mechanistic/Data-Driven Approaches for Reduced Dynamic Modeling: Application to Distillation Columns

Extensive literature has considered reduced, but still highly accurate, nonlinear dynamic process models, particularly for distillation columns. Nevertheless, there is a need for continuing research in this field. Herein, opportunities from the integration of machine learning into existing reduction approaches are discussed. First, key concepts for dynamic model reduction and their limitations are briefly reviewed. Afterwards, promising model structures for reduced hybrid mechanistic/data-driven models are outlined. Finally, crucial future challenges as well as promising research perspectives are presented.     

基于数据属性划分的递阶ELM研究及化工应用

针对极限学习机（ELM）不能有效处理化工过程中强耦合、带噪声的高维数据建模问题,提出了一种基于数据属性划分的递阶ELM神经网络DHELM。该神经网络采用数据属性划分（DAD）方法对高维输入进行聚类、建立自联想子网,并将自联想子网所提取的特征分量作为极限学习机的输入进行建模。同时,利用UCI标准数据集进行了测试,通过工业应用实例进行了验证,并进行了模型对比。结果表明,DHELM网络在处理复杂高维数据时具有收敛速度快、建模精度高、网络稳定性强的特点,为神经网络发展及其化工应用提供了新思路。     

Predicting nepheline precipitation in waste glasses using ternary submixture model and machine learning

Nepheline precipitation in nuclear waste glasses during vitrification can be detrimental due to the negative effect on chemical durability often associated with its formation. Developing models to accurately predict nepheline precipitation from compositions is important for increasing waste loading since existing models can be overly conservative. In this study, an expanded dataset of 955 glasses, including 352 high-level waste glasses, was compiled from literature data. Previously developed submixture models were refitted using the new dataset, where a misclassification rate of 7.8% was achieved. In addition, nine machine learning (ML) algorithms (k-nearest neighbor, Gaussian process regression, artificial neural network, support vector machine, decision tree, etc.) were applied to evaluate their ability to predict nepheline precipitation from glass compositions. Model accuracy, precision, recall/sensitivity, and F1 scores were systemically compared between different ML algorithms and modeling protocols. Model prediction with an accuracy of ~0.9 (misclassification rate of ~10%) was observed for different algorithms under certain protocols. This study evaluated various ML models to predict nepheline precipitation in waste glasses, highlighting the importance of data preparation and modeling protocol, and their effect on model stability and reproducibility. The results provide insights into applying ML to predict glass properties and suggest areas for future research on modeling nepheline precipitation.     

基于数据特征提取的AANN-ELM研究及化工应用

针对极限学习机不能有效解决化工过程中高维数据建模的问题,本文将其与自联想神经网络结合,通过自联想神经网络过滤输入数据中存在的冗余信息、提取特征分量,并对所提取的特征分量采用极限学习机进行训练,由此形成了一种基于数据特征提取的AANN-ELM(auto-associative neural network-extreme learning machine)神经网络。同时,以UCI标准数据集进行测试,以精对苯二甲酸(PTA)溶剂系统进行验证,结果表明,AANN-ELM在处理高维数据时具有学习速度快、网络稳定性强、建模精度高的特点,为神经网络在复杂化工生产中的应用提供了新思路。     

Machine Learning Based Suggestions of Separation Units for Process Synthesis in Process Simulation

As part of Industry 4.0, workflows in the process industry are becoming increasingly digitalized. In this context, artificial intelligence (AI) methods are also finding their way into the process development. In this communication, machine learning (ML) algorithms are used to suggest suitable separation units based on simulated process streams. Simulations that have been performed earlier are used as training data and the information is learned by machine learning models implemented in Python. The trained models show good, reliable results and are connected to a process simulator using a .NET framework. For further optimization, a concept for the implementation of user feedback will be assigned. The results will provide the fundamental basis for future AI-based recommendation systems.     

基于分阶段的LSSVM发酵过程建模

发酵过程建模是研究微生物发酵的重要课题,基于模型可实现被测参量的软测量、系统的优化控制。鉴于引入混合核函数的最小二乘支持向量机在过程建模中具有优良表现,采用基于混合核函数的最小二乘支持向量机建模。但由于发酵过程周期较长,最小二乘支持向量机的全局模型预测精度难以保证,算法复杂度很高,因此提出一种分阶段建模方法。首先,选择表征阶段特性的辅助变量,利用模糊C均值聚类算法对样本数据聚类,将发酵过程分成不同的阶段,然后为各个阶段分别建立最优混合核最小二乘支持向量机局部模型,最后将局部模型合成构成过程的完整模型。将此方法应用于青霉素发酵过程和重组大肠杆菌发酵过程中,验证了该方法的有效性。     

A trust region filter method for glass box/black box optimization

Modern nonlinear programming solvers can be utilized to solve very large scale problems in chemical engineering. However, these methods require fully open models with accurate derivatives. In this article, we address the hybrid glass box/black box optimization problem, in which part of a system is modeled with open, equation based models and part is black box. When equation based reduced models are used in place of the black box, NLP solvers may be applied directly but an accurate solution is not guaranteed. In this work, a trust region filter algorithm for glass box/black box optimization is presented. By combining concepts from trust region filter methods and derivative free optimization, the method guarantees convergence to first‐order critical points of the original glass box/black box problem. The algorithm is demonstrated on three comprehensive examples in chemical process optimization. © 2016 American Institute of Chemical Engineers AIChE J, 62: 3124–3136, 2016     

Advanced trust region optimization strategies for glass box/black box models

We present an improved trust region filter (TRF) method for optimization of combined glass box/black box systems. Glass box systems refer to models that are easily expressed in an algebraic modeling language, providing cheap and accurate derivative information. By contrast, black box systems may be computationally expensive and derivatives are unavailable. The TRF method, as first introduced in our previous work (Eason and Biegler, AIChE J. 2016; 62:3124–3136), is able to handle hybrid systems containing both glass and black box components, which can frequently arise in chemical engineering, for example, when a multiphase reactor model is included in a flow sheet optimization problem. We discuss several recent modifications in the algorithm such as the sampling region, which maintains the algorithm's global convergence properties without requiring the trust region to shrink to zero in the limit. To benchmark the development of this optimization method, a test set of problems is generated based on modified problems from the CUTEr and COPS sets. The modified algorithm demonstrates improved performance using the test problem set. Finally, the algorithm is implemented within the Pyomo environment and demonstrated on a rigorous process optimization case study for carbon capture. © 2018 American Institute of Chemical Engineers AIChE J, 64: 3934–3943, 2018     

Development of a Dynamic Gray-Box Model of a Fermentation Process for Spore Production

Fermentation processes are difficult to describe using purely mechanistic relations as the underlying biochemical phenomena are complex and often not fully understood. In order to cope with this challenge, we developed an approach to augment standard dynamic model equations by data-based components that are fitted to data using machine learning techniques, which results in dynamic gray-box models. This methodology is applied here to the batch fermentation process of the sporulating bacterium Bacillus subtilis, using experimental data from a lab-scale fermenter. The key step in developing the model is the estimation of a training set for the machine learning submodels. The quality of the resulting model is analyzed, and the predictions are compared with real data.     

Nonlinear Optimization Strategies for Process Separations and Process Intensification

Advanced nonlinear programming (NLP) strategies based on equation-oriented (EO) process models are leading to significant improvements in computer-aided process engineering. The EO paradigm allows the development of large, integrated optimization platforms that expand the scope of continuous optimization tasks in process engineering. In particular, these platforms deploy significantly faster NLP strategies than in commercial simulation tools. Moreover, they exploit exact derivatives and system structure in order to consider much larger and more challenging systems. Finally, they allow the incorporation of much more general models, such as multi-level optimization and complementarity constraints. For process optimization this allows the treatment of extended models for complex phase equilibrium and process separations. These advances facilitate the optimization of novel integrated systems that arise in process intensification. Several separation case studies are presented that illustrate these optimization concepts and demonstrate their effectiveness for hybrid membrane/distillation separations and reactive distillation systems that typify novel systems in process intensification.     

生物反应过程的在线支持向量机建模优化

针对生物反应过程具有较强的非线性、时变性,建立准确的机理模型较为困难,并且复杂的机理模型也无法用于在线控制和优化。将在线支持向量机和机理模型结合,提出串并联在线自校正混合建模方法。通过对典型生化过程谷氨酸的生产过程分析,找到影响谷氨酸浓度的关键参数;从现场历史数据中选取样本,建立基于在线向量机的软测量模型。实验结果表明该模型对谷氨酸浓度预测效果较好。     

基于深度集成支持向量机的工业过程软测量方法

基于支持向量机（SVM）的软测量建模方法已经在工业过程控制领域得到广泛应用,然而传统支持向量机直接针对原始测量变量建立模型,未能充分挖掘数据的内在特征信息以提高预测精度。针对该问题,本文提出一种基于深度集成支持向量机（DESVM）的软测量建模方法。该方法首先利用深度置信网络（DBN）来对数据进行深层次的信息挖掘,提取出数据的内在特征,然后引入基于Bagging算法的集成学习策略,构建基于深度数据特征的集成支持向量机模型,以提升软测量预测模型的泛化能力。最后通过数值系统和真实工业数据对方法进行应用分析,结果表明本文提出的方法能够有效提升支持向量机软测量模型的预测精度,能够更好地预测过程质量指标的变化。