Ontology construction

The purpose of ontology model construction is to conceptualize domain knowledge explicitly. Prior to building a knowledge graph for a specific domain, the knowledge concepts and the relationships between concepts need to be defined in the ontology model, providing specifications for the extraction of subsequent entities and relationships. At present, the seven-step approach proposed by Stanford University is widely used to construct ontology, and it is more mature and effective to construct ontology with the protege tool [19,20]. In this study, we simplified the seven-step approach into four steps. The first step was to build terminology related to CRDM. The second step was to determine the classes and hierarchies of terms. The third step was the construction of the properties and relationships of the classes, the properties included object and data properties. The last step was to determine the individuals by class.

In general, the major task of constructing ontology for fault diagnosis was to define the triples, which is composed of three elements, namely the subject (S), relation (R), and object (O). The triples constitute a fault event, which is mainly derived from the CRDM fault knowledge corpus. Formally, e is used to represent a fault event, and the triples can be represented by formula 1. $e=[S]-[r:R]\to [O]$ (1)

Here, S and O represent the subject and objects in the description text of a fault event, respectively. S and O constitute the entities of the CRDM fault diagnosis knowledge graph. R is the relationship between S and O, which corresponds to the event trigger word in the fault event text.

The ontology of the CRDM fault diagnosis knowledge graph can be divided into four conceptual classes from top to bottom, including equipment, components, fault_alarm, and fault_symptom. Among them, the fault_alarm represents the fault alarm codes when a fault occurs, and the fault_symptom represents the symptom of a fault that appears on a sensor. In the fault knowledge corpus of CRDM, each fault event is represented by unstructured text and the fault events corresponding to conceptual classes need to be decomposed based on the Subject-Relation-Object principle. The decomposed individuals will become the entities stored in the ontology model. Then, the relationships between each conceptual class need to be defined.

This study defines three types of relationships between conceptual classes, including consist_of, has_fault, fault_cause, and has_symptom. Further, consist_of represents the relationship between equipment and component, has_fault represents the relationship between equipment and fault_alarm, has_symptom represents the relationship between component and fault_symptom, while fault_cause represents the relationship between fault_symptom and fault_alarm. Some entity individuals have data properties, e.g., the property in each individual from the class of fault_alarm has three categories, including description, solution, and alarm_type. The description property mainly represents the content of the alarm signals, the solution property represents the measures to deal with the alarm signal, and the alarm_type property mainly gives a warning to the occurred fault.

After defining conceptual classes and relationships, the fault diagnosis ontology model is constructed by the protégé tool. Figure 2 shows the structure of the fault diagnosis ontology model. The display boxes of dot labeling in the fault diagnosis ontology model represent the conceptual classes, while those of diamond marks indicate individual present in each class. The relationships between class and individuals are represented by solid lines with arrows. Furthermore, the relationships between individuals are represented by dotted lines with arrows, and different relationships are distinguished in various colors. In Fig. 2, the No. 3 control rod has a fault alarm code, namely 10JDEGT202XM25. The fault_cause of this fault alarm code is the R6H20, which is the fault_symptom of the sensor and is named as resolver_6. The resolver_6 is the component of the No. 3 control rod. Thus, the ontology model has clear classifications and affiliation relationships. The specific configuration of the CRDM fault diagnosis ontology model is shown in Table 1.

Fig. 2Full-screen View

Structure of the CRDM fault diagnosis knowledge graph ontology model.

Graph storage

The above-mentioned fault diagnosis ontology model produced by the protégé development platform needs to be stored in a specific physical structure. Neo4j is one of the most popular graph database software, which consists of three elements: node, relationship, and attributes [12]. These elements correspond to the configuration of the constructed CRDM fault diagnosis ontology model. Thus, this study used the Neo4j graph database to store the CRDM fault diagnosis ontology model, which can represent the ontology graph structure clearly. The nodes represent the conceptual classes, and the relationships in Neo4j correspond to the relationships in the ontology model. The data properties of the ontology model will be stored in the attributes from the Neo4j database. Notably, the ontology model cannot be stored in the Neo4j graph database directly. However, the ontology model in the protégé development platform only supports the web ontology language (OWL) format, which conforms to the resource description framework (RDF) syntax. Therefore, the OWL file needs to be converted into an RDF file using the rdf2rdf jar package, which can be imported into the Neo4j graph database. Figure 3 shows parts of the knowledge graph stored in the Neo4j database.

Fig. 3Full-screen View

(Color online) Display of the CRDM fault diagnosis knowledge graph in Neo4j database (partial).

Figure 3 shows the nodes corresponding to different entities that are displayed in various colors, and the relationship connects the head entity and tail entity. In this figure, the No. 3 control rod used a certain fault event for nuclear fuel compensation, with a fault alarm code named 10JDEGT203XM240, and the fault cause was R6L20, which is related to the No. 6 resolver. The No. 3 control rod consists of No. 6 resolver. Moreover, compared to the conventional SDG model, the fault knowledge graph can directly store more fault information, which covers the prior probability of fault symptom, fault description, and solution. With the help of the Neo4j graph database, detailed information related to the fault alarm can be queried in the Neo4j database using attribute elements handily.

Entity extraction

The purpose of entity extraction is to identify the subject of the query statement entered by the operator. The subject (i.e., entity) is the major component of executing the Neo4j graph database query paradigm, which has been stored in the fault knowledge graph. In this study, the Jieba tool with the precise mode was adopted to segment the fault query sentence for extracting the entity. It is one of the most popular Chinese word segmentation modules developed in Python. The fault diagnosis of CRDM is mainly relevant at the equipment level, which leads to a finite number of terms. Therefore, a domain terms dictionary corresponding to CRDM was built to ensure that terminology can be segmented accurately. Then, each segmented word was queried individually in the dictionary. If matching words with the same meaning could be retrieved in the dictionary, the word was extracted as the recognized entity from the fault query statement. In this way, the entities related to the terms of CRDM can be accurately identified and extracted from the fault query statement. Figure 4 reveals the detailed algorithm process of entity extraction.

Fig. 4Full-screen View

Detailed algorithm process of entity extraction.

Intention recognition

Similarly, the query intent is associated with the relationship between entities. The aim of intent recognition in this study was to classify the query intent from the operator, who uses the CRDM fault diagnosis system to query the information about the fault event and make decisions. For example: the operator wants to find out the possible fault cause of a fault alarm code 10JDEGT203XM240. The input sentence contains the entity named 10JDEGT203XM240, and the fault cause is the intent of the query. First, the fault diagnosis system needs to recognize this intent and extract the entity. Then, the fault diagnosis system uses the extracted entity and the classified intent to search for the answers in the constructed knowledge graph database. During this process, natural language understanding is critical to the performance of the CRDM fault diagnosis system. In Chinese, there are different expressions of query intention, but they all have the same semantics, e.g., What are the causes of the fault? or What is the problem causing the fault? These two queries are related to fault cause. Hence, inspired by previous research [21], a method integrating a three-layer robustly optimized Bidirectional Encoder Representation from Transformers (RBT3) pretraining approach and text convolutional neural network (TextCNN) was introduced to address intent recognition. Figure 5 illustrates a high-level view of the proposed RBT3-TextCNN model.

Fig. 5Full-screen View

(Color online) Overall structure of the RBT3-TextCNN model.

The model architecture of BERT is a multi-layer bidirectional Transformer encoder based on the original Transformer mode and released in the tensor2tensor library [22]. The BERT model provides a powerful context-dependent sentence representation and can be used for intent classification, which is a various target task [21]. Particularly, RBT3 is a small and effective pre-trained model derived from BERT [24]. Additionally, the TextCNN is a variation model of the CNN, which has a better performance in terms of Chinese text classification [23]. In this study, the RBT3-TextCNN model was adopted to achieve high classification accuracy and low computational cost. The RBT3-TextCNN model consists of a word embedding layer, convolutional layer, global max-pooling layer, and fully connected layer, as shown in Fig. 5.

For a single sentence classification task based on the BERT model, a special classification embedding $([CLS])$ is inserted as the first token and a special token $([SEP])$ is added as the last token. Hence, the input sentence with $n$ words is constructed as $X=([CLS],x_1,x_2,\cdots ,x_n,[SEP])$. A word $x_i$ can be transformed into a word vector $v_i$, the formula is as follows: $v_i=\text{RBT3}(x_i),i=1,2,\cdots ,n$ (2) $V=[v_1,v_2,\cdots ,v_n]$ (3)

Consequently, the RBT3 model generates a word vector matrix $V\in R^{n\times d}$, where $n$ is the input length and $d$ is the word vector dimension.

Subsequently, the feature map is generated by the convolutional calculation in the TextCNN layer. The convolution kernel needs to be set as $w\in R^{h\times d}$, where $h$ is the height and $d$ is the width. In this study, three types of convolution kernels were set, including $w_1:(h_1=3,d)$, $w_2:(h_2=4,d)$, and $w_3:(h_3=5,d)$. Among these, the input and output channels of the convolution kernel were 1 and 256, respectively. The feature $c_{ij}$ generated by the word vector in the window $v_{i:i+h-1},h=[h_1,h_2,h_3]$ is given by the following formulas: $c_{ij}={\displaystyle \sum _{j=1}^3\text{ReLU}(w_j\cdot v_{i:i+h_j-1}+b),i=1,2,\cdots ,n}$ (4) Here $b\in R$ is a bias term, and ReLU is a non-linear activation function. The feature $c_j\in R^{n-h_j+1}$ is generated by the convolution operation of the convolution kernel $w$ and the word vector $V$, as shown in the following formula: $c_j=[c_{1j},c_{2j},\cdots ,c_{(n-h_j+1)j}],j=1,2,3$ (5)

Then, the most important feature information is compressed and retained through the global max-pooling operation, which only takes the maximum value of each feature. The new feature $z$ is obtained as follows: $c_j^{\text{'}}=\text{GMP}\left\{c_j\right\}=[c_{1j}^{\text{'}},c_{2j}^{\text{'}},\cdots ,c_{kj}^{\text{'}}],j=1,2,3,k=n-h_j+1$ (6) $z=\text{concat}(c_1^{\text{'}},c_2^{\text{'}},c_3^{\text{'}})(\text{axis}=-1)$ (7) where concat is the splicing operation and $k$ is the total number of kernels. The final output classification result $y$ of the fully connected layer is as follows: $y=\text{softmax}(w_{\text{dense}}\cdot (z\odot r)+b_{\text{dense}})$ (8) where $w_{\text{dense}}$ and $b_{\text{dense}}$ are weights and bias terms of the fully connected layer, respectively; $r\in R^h$ is the mask vector used to randomly drop elements out in $z$; the symbol represents the element-level multiplication calculation; and softmax denotes the activation function, which can efficiently tackle the problem of multi-categories classification. Meanwhile, the sparse categorical cross-entropy is adopted as the loss function in this model, as shown in the following formula: $\text{Loss}=-{\displaystyle \sum _{i=1}^T{y}_i\log p_i}$ (9)

Among them, $T$ goes over all the output classes, $p_i$ indicates the probability of the class as predicted by the model, and $y_i$ represents the real probability of the class as provided by the input data.

BNs

BNs are well-established graphical formalisms for encoding conditional probabilistic relationships between uncertain variables [31]. The structure of BNs is a directed acyclic graph (DAG) in which the nodes represent variables, arcs signify the existence of direct causal relationships between the linked variables, and strengths of these relationships are expressed by conditional probabilities. BNs aim to model conditional dependence, and therefore causation, by representing conditional dependence by edges in a DAG. BNs are ideal for taking an event that occurred and predicting the likelihood that any one of several possible known causes was the contributing factor. Bayes’ theorem is the foundation for uncertainty inference based on the BNs.

Bayes’ theorem is stated mathematically as follows: $P(A|B)=\frac{P(B|A)P(A)}{P(B)}$ (10) where $A$ and $B$ are events and $P(B)\ne 0$.

(1) $P(A)$ refers to a prior probability without any given conditions.

(2) $P(B)$ refers to the probability of observing $B$, known as marginal probability.

(3) $P(A|B)$ refers to the probability of event $A$ occurring given that $B$ is true, also called the posterior probability of $A$ given $B$.

(4) $P(B|A)$ refers to the probability of event $B$ occurring given that $A$ is true.

In the Bayesian context, $P(A|B)$ and $P(B|A)$ are all conditional probabilities. For each node $B$ in BN, $P(A|B)$ is completely described in the CPT. Hence, the prerequisite in building the BN model for decision support and risk assessment is to define the necessary conditional probability tables (CPTs), which rely on the judgment of domain experts. For a node in BN, along with n parents, which are Boolean, the size of the CPTs grows exponentially with $n$. This is known to be practically complex and intractable. To tackle the parameter explosion problem existing in CPTs, the Leaky Noisy-OR function has proven to be useful in simplifying the elicitation of complex CPTs in BNs involving Boolean variables.

Leaky Noisy-OR function

The Leaky Noisy-OR function is a good approximation of a wide range of BN model fragments and requires only n+1 parameters for the full CPT specification [31]. Formally, the definition of the Leaky Noisy-OR function is as follows:

Let $X=\{X_1,X_2,\cdots ,X_n\}$ be $n$ Boolean variables, i.e., $X_i=\{T,F\},i=1,2,\cdots ,n$. Let $Y$ be a Boolean variable with parent nodes {X₁, X₂, ..., Xn}{X₁, X₂, ..., Xn}. Then, the Leaky Noisy-OR function would be defined as follows [32]: $P\left(Y=T|X_p\right)=\{\begin{array}{cc}1-(1-q_l){\displaystyle \prod _{X_i\in X_T}{q}_i}& X_T\ne \varnothing \\ q_l& X_T=\varnothing \end{array}$ (11) where $X_p$ represents a set of states for all parent nodes, $X_T$ represents the set of all the parent nodes of T in $X_p$, $q_l$ represents the leak probability, and $q_i$ represents the failure probability. The corresponding concepts are defined as:

(1) leak probability $q_l$: Due to insufficient analysis, the cause of the fault event is not detected, while the corresponding fault symptom is detected. The probability of such happening is called leak probability.

(2) failure probability $q_i$: In contrast, the failure probability represents the probability that the cause is detected but the fault symptom is not detected because of monitoring means, measurement error, and low frequency of fault. In most cases, there are not enough fault cases, especially in NPP. Estimating the failure probability by expert experience and failure mechanisms is recommended.

Variable elimination

Bayesian inference is a method of statistical inference in which the Bayesian theorem is used to update the posterior probability of a hypothesis as more evidence or information becomes available. More specifically, inference in BNs can diagnose the causes given the fault occurrence under causality. Several exact approaches for computing posterior probabilities in BN have been proposed and implemented, including variable elimination (VE), junction tree propagation, and approximate inference. [33]. Among these, the VE algorithm, adopted in this study for the sake of the efficiency-complexity trade-off, is fundamental and easily understood. It tackles changes to the knowledge base more easily than other approaches. The detailed process of the VE algorithm is shown in Table 2:

Assuming that the BN contains n random variables to be eliminated and the type of each variable is Boolean, each step eliminates a random variable, and there are at most k probability distribution functions associated with each random variable. Then, the time complexity of the VE algorithm is $O(n*2^k)$.

Dataset

This study collected 753 fault query description texts from the CRDM in TMSR-LF1, and the corpus of the constructed knowledge graph is mainly derived from the design specification, operation and maintenance manual, and alarm list. The short sentences for training the RBT3-TextCNN model were categorized into four classes, namely fault cause, fault phenomenon, fault solution, and equipment composition relationship. The average length of each sentence was 15.7 words and the number of total characters was approximately 11800. Each sentence in the corpus was manually labeled. The dataset was divided into the training set and the test set at a ratio of 7:3. The information about fault query text distribution is shown in Table 3.

Training details

The configuration parameters of the RBT3 model were quoted from the open-source website¹. RBT3 has three hidden layers and was derived from the BERT-Base model, which has 768 hidden states. Therefore, the input word vector-matrix dimension was set to 768. The optimizer and the maximum length (MaxLen, 50 tokens) were cited from the literature [21]. The Text-CNN filter size, activation function, and pooling method were extracted from the literature [23]. The total number of TextCNN was equal to the input word vector-matrix dimension. Besides, another three hyper-parameters, including batch size, epochs, and learning rate, were generated via fine-tuning based on the comparison experiment. As shown in Fig. 9, the RBT3-TextCNN model training stopped before 20 epochs. For conservative consideration, the initial training epoch was set to 25. The batch size was set to 8 for the sake of validation loss and validation accuracy. The learning rate was automatically adjusted according to the training accuracy. Adam was used for optimization with the initial learning rate value of 5e-5 [21]. The training details are shown in Table 4. The operating environment was set as an 8-core Intel Core i7-7740K CPU@4.30 GHz and the GPU was Geforce GTX 1080Ti. Moreover, the major dependency environments were tensorflow-gpu-1.14.0 and keras-2.3.1.

Fig. 9Full-screen View

Performance of batch size with different values. (a)Validation Loss. (b) Training accuracy. (c) Validation accuracy.

Results

To test the validity of the model with respect to the intention recognition using the above-mentioned dataset, we selected the TextCNN and RBT3 as the baseline models to compare with RBT3-TextCNN. The experiment involved the use of two indicators to evaluate the model performance. Among them, the macro average composite index (MaF1) is the weighted average of macro accuracy (MaP) and macro average recall rate (MaR). Considering the uneven distribution of various samples under multiple classification issues, the accuracy of different models was evaluated based on the MaF1 indicator, which is widely adopted to evaluate the performance of classification models in machine learning. Assuming the dataset can be divided as $(C_1,C_2,\cdots ,C_n)$, the MaF1 formula is as follows: $\text{MaF1=}\frac{\text{2}\times \text{MaP}\times \text{MaR}}{\text{MaP+MaR}},$ (12) $\text{MaP}=\frac{1}{n}{\displaystyle \sum _{i=1}^n{P}_i},$ (13) $\text{MaR}=\frac{1}{n}{\displaystyle \sum _{i=1}^n{R}_i},$ (14) ${\text{P}}_i=\frac{\text{TP}}{\text{TP+FP}},$ (15) ${\text{R}}_i=\frac{\text{TP}}{\text{TP+FN}},$ (16) where n is the number of categories in the dataset, TP represents the number of samples correctly predicted in C_i, TP+FP represents the number of samples predicted in C_i, and TP+FN represents the number of real samples in C_i. Hence, P_i (i.e. Precision) reflects the proportion of true positive samples among the positive samples by the classifier, and R_i (i.e. Recall) reflects the proportion of correctly predicted positive samples in the total positive samples. MaF1 combines the effects of P_i and R_i. In addition, the total training time of different models was calculated to evaluate the efficiency.

RBT3 is a powerful model for extracting the characteristics of input statements, especially under the circumstance of limited training datasets. RBT3 has many parameters, and the training time complexity is higher than that of the other models. However, TextCNN is proposed as a relatively fast-training model, but its performance is subject to limited training datasets. In this study, TextCNN and RBT3 were chosen as the baseline models, respectively. The small dataset utilized in this study was derived from the CRDM fault corpus of the TMSR-LF1. The results are shown in Table 5. The TextCNN model took more training time to achieve convergence than the RBT3 model based on the same batch size. The RBT3 model showed a better performance in recognizing the CRDM-related fault query intent than the TextCNN model. Compared with RBT3 and TextCNN, the RBT3-TextCNN had a 1.1% and 2.36% advantage in intent recognition performance, respectively, but the total training time gap among these three models was not obvious.