Materials and dataset preparation

Our dataset was obtained primarily from two sources. One is a widely used open-source dataset published by Anderson [33], which is derived from nickel-based superalloys used in Canada Deuterium Uranium reactors [7], specifically the neutron-irradiated ex-service Inconel X-750—a precipitation-hardened variant of Alloy-600—irradiated to approximately 55 dpa and 18000 appm helium. This dataset has been used as a benchmark in numerous experiments.

The other source was obtained from our previous dataset on the ion irradiation of additively manufactured and conventionally rolled 304L stainless steel [38]. This section describes the simulation of various stages of neutron irradiation using ion irradiation. The samples were irradiated with a 350 keV He⁺ ion beam at a density of 1 × 10¹⁷ ions/cm², thus resulting in the formation of helium bubbles within the material with a damage level of 2.35 dpa and a helium concentration of 5.5 atomic percent. The irradiated samples were cut to 50 nm and examined using TEM (Tecnai G2 F20 S-TWIN; FEI Company, USA). The Mask R-CNN training dataset is publicly available [33]. It contains 224 images as the training data, and the validation set comprises 19 images. For the test set, we selected eight bright-field TEM images from this dataset, as well as four bright-field TEM images from our in-house dataset for testing.

We selected underfocused images from both datasets to be included in our study.

User-guided interactive machine learning

In this section, we present a novel methodology that integrates human expert knowledge with multimodel fusion for the analysis of helium bubbles of various sizes and characteristics. Our approach leverages user-guided interactive machine-learning techniques to combine the advantages of human expertise and advanced computational models. This integration enables a comprehensive and accurate assessment of helium-bubble formation and behavior across different scales.

To facilitate the adoption of our method, we provide an example code that includes the necessary libraries and demonstrate the training of the generative convergence model (GCM) using a few helium-bubble annotations. Users can easily follow this example to install the required libraries in a Python environment and replace the provided images and annotations with their own data to apply the model to similar images. This integration enables a comprehensive and accurate assessment of helium-bubble formation and behavior across different scales.

2.2.1

Morphological features of helium bubbles

TEM is an effective tool for examining the microstructure of materials and can reveal the distinct characteristics of helium bubbles based on their size and shape. As illustrated in Fig. 1, which presents both simulated and actual TEM images of helium bubbles, the diameter of Fresnel rings serves as a crucial indicator of the bubble size. Bubbles exceeding 2 nm in diameter exhibited clearly discernible first bright and dark Fresnel rings. This phenomenon is due to the interaction between the electron beam and helium-filled cavity, which results in a phase contrast that manifests as characteristic rings. As the bubble size decreased, an interesting optical effect occurred: the relative size of these diffraction rings increased in proportion to the bubble diameter. This inverse relationship provides valuable information regarding bubble dimensions. However, when the bubble size decreased to below 1 nm, the imaging characteristics changed significantly. For these nanoscale bubbles, the first bright ring overlapped with the bubble, thus effectively obscuring its shape. Simultaneously, the dark ring became indistinct and blended into the background, thus rendering its differentiation from the surrounding matrix challenging. This transition in imaging features presents a unique challenge in the detection and analysis of miniscule helium bubbles.

Fig. 1Full-screen View

(Color online) Fresnel-diffraction characteristics of helium bubbles in simulation and actual imaging. Columns from left to right show bubbles >2 nm, 1-2 nm, and <1 nm in diameter. Bubbles >2 nm exhibit a faint first bright Fresnel ring. The 1-2 nm bubbles show the most distinct boundaries between dark and bright Fresnel rings. For bubbles <1 nm, boundary definition is challenging as the first bright Fresnel ring exceeds the bubble’s imaging range

To address these challenges and achieve a comprehensive detection and quantitative analysis of helium bubbles across all size ranges, this study proposes a novel model-fusion approach. This method allows each model to focus on the size range in which it excels. Mask-RCNN—a deep-learning model—was employed to detect helium bubbles with clear Fresnel rings, typically those larger than 1 nm. Mask-RCNN is particularly effective in identifying and segmenting objects with well-defined boundaries, thus rendering it ideal for bubbles with distinct ring structures. To complement this, an improved GCM was utilized to detect bubbles smaller than 1 nm with blurred boundaries. The GCM was adapted to address the challenges posed by the overlapping bright rings and indistinct dark rings, which are characteristic of these miniscule bubbles. By combining the results from these two models, we aim to comprehensively detect helium bubbles across the entire size spectrum. This fusion approach leverages the advantages of each model to overcome the limitations inherent in analyzing bubbles of varying sizes.

To enhance the accuracy and reliability of our analysis, we incorporated user-guided interactive machine learning. This novel approach eliminates biases that may be introduced during model training, whether from manual annotation or automated processes. The interactive component allows the model to learn from expert inputs, thus continually improving its performance and adapting to the nuances in different TEM images. This comprehensive methodology not only enables the detection of helium bubbles across all size ranges but also ensures high accuracy and reliability in quantitative analysis. By combining sophisticated deep-learning techniques with expert human guidance, we aim to advance e material characterization using TEM, thereby providing invaluable insights into the behavior of helium bubbles in various materials and conditions.

2.2.2

Machine-learning models

Among the various methods for segmenting helium bubbles, the Mask-RCNN model demonstrated effectiveness in helium-bubble detection, as reported in previous studies [34]. However, this approach relies on pre-annotated training data, which can introduce inherent limitations. The model may inherit biases from human annotators, thus potentially restricting its ability to recognize helium bubbles with characteristics different from those in the training set. This limitation is particularly evident when examining helium bubbles smaller than 1 nm, which do no present clear Fresnel diffraction rings and exhibit highly variable shapes. Consequently, the Mask-RCNN model may struggle to identify and segment these smaller structures accurately.

Hence, the GCM is introduced to computer vision-based machine learning, particularly for detecting minute nanosized helium bubbles. The innovation of the GCM is its unsupervised learning framework, which differs from conventional methods using predefined shape features. Specifically, the GCM utilizes a series of trainable filters (or convolution kernels) that are thoughtfully designed and stacked together.

Unlike conventional CNNs, which typically process various color and grayscale images using randomly initialized convolution kernels, the GCM is specifically optimized for grayscale electron microscopy images. This is achieved by significantly simplifying the model structure and replacing random convolution kernels with filters of predefined distributions and functions. These filters include Gaussian, adaptive histogram equalization, and adaptive binarization filters. The key stages of the workflow are as follows:

Preprocessing with Gaussian and Uniform Filters: In the initial stage, the model preprocesses grayscale electron microscopy images using Gaussian and adaptive histogram equalization filters. These filters suppress noise and enhance the clarity of helium-bubble structures by emphasizing relevant features. This preprocessing step generates a substantial amount of training data, thus establishing a foundation for the subsequent stages.

Feature Extraction via Adaptive Binarization: Following preprocessing, the GCM applies adaptive binarization to extract helium bubbles by analyzing the contrast between the bubbles and their surrounding environment. This step isolates the key features of the helium bubbles while maintaining structural integrity, thus enabling effective feature extraction. Additionally, the model outputs the helium-bubble count.

Parameter Refinement via Iterative Training: Based on a specific loss function’s evaluation of the reasonableness of the helium-bubble count, the model iteratively refines the parameters of the filters, including their size, intensity, and distribution. These refinements are guided by results from previous iterations, thus allowing the GCM to progressively optimize the filter parameters and improve detection accuracy.

The filters utilized in the GCM can be efficiently implemented using libraries such as Python-OpenCV, thus rendering the approach computationally efficient and accessible for practical applications.

Owing to this unique design, fully random kernels need not be trained; thus, learning parameters such as filter size and intensity as well as other characteristics can be prioritized. Although this approach limits the applicability of the GCM to grayscale images, it provides significant advantages in terms of interpretability and reduces the number of annotations required for training.

The initial processing stages of the GCM involve Gaussian filtering as a preprocessing step, which can be represented by the following Eq. (1):(1)where G(x,y) is the weight of the filter and, σ is the standard deviation that determines the degree of blurriness, and x and y are the distances from the center point.

The application of Gaussian filtering as a preprocessing step in the GCM serves a dual purpose in enhancing the detection of helium bubbles. Primarily, it serves as a noise-reduction mechanism that smooths out random fluctuations in an image that may potentially interfere with accurate bubble detection. Simultaneously, this filtering process enhances the distinguishing features of the helium bubbles. Because helium bubbles feature indistinct or fuzzy boundaries, the blurring effect of a Gaussian filter is advantageous for the detection process. By applying a controlled amount of blur, the filter homogenizes the appearance of these diffuse boundaries, thus rendering them more uniform and thus more easily recognizable in subsequent processing steps. This homogenization effect is particularly beneficial for creating a more consistent representation of helium bubbles across the image, thus allowing the GCM’s trainable filters to learn and detect these structures more effectively, even when they vary slightly in size or shape. Essentially, the Gaussian filter prepares the image data such that they align well with the GCM’s unsupervised-learning approach, thus providing a basis for more robust and accurate helium-bubble detections.

However, the GCM may not exhibit instability when they are trained without manual annotations. Hence, we have enhanced the model by integrating user-guided interactive machine learning. This approach eliminates the necessity to focus on the specific shapes of the nanosized helium bubbles, which is impractical owing to their variable morphology. Instead, users provide point annotations at the central positions of the bubbles. Subsequently, the GCM learns to infer the potential bubble shapes by analyzing the features surrounding these labeled points. This interactive method enables an interpretable training of the GCM, thus resulting in more reliable helium-bubble segmentations.

Figure 2 illustrates our ensemble-model approach for segmenting helium bubbles of various sizes by incorporating user annotations. When the user-guided point annotations are provided, the ensemble model trains the GCM to segment the corresponding helium bubbles based on these annotations. Otherwise, the model employs the GCM with default parameters alongside the Mask R-CNN for helium-bubble segmentation and analysis. This dual approach enables the merging of results, thus ensuring a comprehensive segmentation of helium bubbles across all size ranges. This figure shows the manner by which this adaptive method addresses different input scenarios to maximize the accuracy and completeness of bubble detection and characterization.

Fig. 2Full-screen View

(Color online) Example workflow of helium-bubble detection model. Input comprises the original image and a user-annotated image. Output includes identified helium bubbles (indicated by red regions or red circles) and their size distribution. Rows 1 and 2 show the model’s operation with user-guided point annotations, which results in the precise segmentation of all helium bubbles with accurate shape and size information. Row 3 illustrates the model’s performance without user annotations, where it effectively detects larger helium bubbles, which are denoted by circular indicators. This comparison highlights the effect of user input on the model’s segmentation capabilities across various bubble sizes

Examples of our workflow demonstrate the flexibility and efficiency of the ensemble model. In scenarios with user-guided point annotations (rows 1 and 2), the model precisely captured the requirements of the annotator by employing a user-guided approach to segment helium bubbles corresponding to the annotated sizes. This underscores the significance of user-guided point annotations in enhancing the model performance, particularly in the detection of microscopic bubbles. Conversely, the scenario without user annotations (row 3) demonstrates the autonomous capabilities of the model, although it primarily focused on the detection of larger bubbles. This comparison highlights the adaptability of the model to various input conditions. Notably, the output encompasses not only the location and morphology of the bubbles but also the size-distribution analysis, which is crucial for material-science research. The model integrates the unsupervised-learning advantages of the GCM with the object-detection capabilities of the Mask R-CNN. In the absence of user annotations, the results of these two methodologies are intelligently merged to achieve the most comprehensive bubble-detection outcomes. This adaptive approach not only enhances detection accuracy and completeness but also provides researchers with the flexibility to select optimal analysis methods under diverse experimental conditions.

Evaluation metrics

Standard metrics such as precision and the F1-score are widely used to quantify the overlap between a segmentation map and its ground truth.(2)(3)(4)

We evaluated our model’s predictions using point annotations as the ground truth, as shown in Fig. 3. True positive (TP) instances were defined as cases in which the model accurately identified a positive sample, with a predicted bubble region encompassing a ground-truth point annotation. False positive (FP) cases represent misclassifications in which the model predicts a bubble in an area without a corresponding ground-truth annotation. False negative (FN) instances indicate situations in which the model fails to detect a bubble at a location denoted by a ground-truth point annotation. Because of the challenge in precisely defining helium-bubble edges in TEM images, we utilized point annotations placed at the core of each bubble as our ground-truth reference. This approach allowed us to calculate the TP, FP, and FN while accommodating the inherent difficulties of exact boundary delineation at the nanoscale. Figure 3a shows an original image with point annotations serving as the ground truth, whereas Fig. 3b presents the model’s predictions overlaid on the image, which shows the manner by which the TP, FP, and FN are defined in the context of our evaluation framework.

Fig. 3Full-screen View

(Color online) Evaluation of model predictions using point annotations as ground truth. (a) Original image with point annotations serving as ground truth. (b) Model prediction overlaid with labeled examples showing the manner by which true positive (TP), false positive (FP), true negative (TN), and false negative (FN) are defined

Model performance

In this study, we compared the performance of our ensemble model with that of a previously used Mask R-CNN for analyzing larger helium bubbles. This comparison is particularly relevant because helium-bubble sizes typically vary with irradiation temperature. The benchmark micrographs used in this analysis showed helium-bubble formation in the helium-dense regions of 304L stainless steel subjected to high-temperature irradiation at 750 °C. This approach allowed us to evaluate the effectiveness of our ensemble model in accurately detecting and characterizing larger helium bubbles under specific irradiation conditions.

Figure 4 presents a comparative analysis of helium-bubble segmentation in 304L stainless-steel samples after 100 h of irradiation annealing. Individual helium bubbles are indicated by red circles in all the images. Our model, which integrates the Mask R-CNN and GCM, demonstrated significantly improved segmentation accuracy compared with previous state-of-the-art methods, particularly for smaller bubbles and those located near image edges, as highlighted by the dashed yellow circles. These areas were particularly challenging to detect and segment accurately using previous methods.

Fig. 4Full-screen View

(Color online) Comparison of helium-bubble segmentation methods. (a) Original transmission electron microscopy image. (b) Expert-annotated ground truth. (c) Results from our proposed ensemble model. (d) Results from previous state-of-the-art method (Mask R-CNN)

The ground truth shown in Fig. 4b was annotated by domain experts. Compared with the results from the previous state-of-the-art method shown in Fig. 4d, our ensemble model presented in Fig. 4c demonstrated superior performance in comprehensively detecting and delineating helium bubbles of various sizes and locations within the image. This advancement is attributable to the synergistic combination of the Mask R-CNN and GCM, where their respective strengths were leveraged. However, owing to its tendency to detect all bubbles, our model may split overlapping helium bubbles into multiple segments. This behavior contrasts with the primary error observed in the previous Mask R-CNN method, which merged overlapping bubbles into one and struggled to detect smaller bubbles.

Statistically, this resulted in an error of approximately 5% in the detected bubble count and a slight reduction in the average bubble size (tested on the 304L stainless-steel dataset). This effect may be more pronounced in images with specific characteristics. Therefore, we recommend that users compute the helium-bubble count and average size across multiple images to obtain more reliable results.

Notably, our model achieved improved and more comprehensive segmentations without relying on handcrafted annotations, which represents a significant achievement in the automated helium-bubble detection and analysis of irradiated steel samples. The ability to accurately segment smaller bubbles and those near edges, which were previously overlooked or inaccurately segmented, enables more precise quantifications and characterizations of the microstructural features. Consequently, this provides deeper insights into the microstructural evolution of irradiated materials and ultimately contributes to the development of radiation-resistant materials for nuclear energy, space exploration, and other critical applications.

Figure 5 shows two examples in which our ensemble model excels in accurately detecting and segmenting smaller and densely packed helium bubbles—a challenge that previous methods struggled to address effectively. Conventional approaches are typically inadequate in managing complex scenarios with numerous small and closely arranged bubbles. By contrast, our model demonstrated exceptional ability in precisely delineating individual helium bubbles, even in intricate situations. This capability is particularly significant because the accurate detection and analysis of smaller bubbles are vital for the comprehensive characterization of irradiated materials.

Fig. 5Full-screen View

(Color online) Comparison of segmentation results for smaller helium bubbles. (a) Original image; (b) output from our ensemble model; (c) output from Mask-RCNN

As shown in Table 1, our ensemble model achieved significant improvements in terms of the overall segmentation performance. In particular, it obtained an F1-score of 0.85, which surpassed those achieved by previous state-of-the-art methods, including those of Wu et al. [37] (0.799), Anderson et al. [33] (0.78), Chen et al. [35] (0.77), and Jacobs et al. [34] (0.82). Notably, our model achieved a well-balanced performance between precision (0.81) and recall (0.89), thereby outperforming competing methods in both metrics.

Chen et al. [35] and Jacobs et al. [34] employed Mask R-CNN models. Specifically, Chen et al. [35] achieved a higher recall (0.88) through scaling techniques but at the cost of reduced precision (0.75). Meanwhile, Jacobs et al. [34] reported a high F1-score of 0.82 but did not provide detailed precision and recall metrics. In comparison, our model not only achieved the highest F1-score but also maintained an optimal balance between precision and recall, thus demonstrating superior segmentation performance in general.

Notably, the improvement in the evaluation metrics shown in Table 1—while significant—is not particularly pronounced in terms of numerical increase. This is partly because our test dataset excludes results from low-temperature irradiation, which contains fewer nanometer-scale helium bubbles. Meanwhile, previous methods, such as the Mask R-CNN, almost failed completely (e.g., the results shown in Fig. 5(c)). Additionally, most of our test data were derived from a publicly available dataset, which exhibited a much higher similarity to the training data used by the competing methods. Consequently, the performance gap between our method and the other methods may appear to be numerically smaller owing to these factors. Nevertheless, our proposed approach significantly reduced the annotation burden and enabled the analysis of nanoscale helium bubbles, which was previously challenging or infeasible.

User-guided point annotations and effect on Segmentation

We conducted a comprehensive evaluation to investigate the effect of the number of user-provided point annotations on the segmentation performance of the model and then compared the model with existing neural-network-based interactive learning methods [40]. The assessment began with a single-point annotation, which was then incrementally increased across multiple iterations. At each step, the model was retrained using the point annotations provided as input data, and the segmentation outputs were analyzed in detail.

Specifically, we selected a diverse set of images and manually annotated one to five points on each image to simulate user-guided point annotations of different densities. Subsequently, these annotated images were used to independently train separate instances of the segmentation model. Systematically varying the number of user-guided point annotations—–from sparsely annotated cases with a single point to more densely annotated examples with five points——enabled us to analyze the correlation between user-input quantity and model performance, thus ultimately enabling us to determine the optimal number of annotations required for effective segmentation.

We benchmarked our approach against two established methods: the pretrained Mask R-CNN model and current neural network-based interactive segmentation techniques [40]. To provide a more comprehensive evaluation, we expanded our analysis by incorporating helium-bubble contours as the background input, thus offering additional context for segmentation-performance assessment.

Figure 6 illustrates the effect of increasing the number of user-provided point annotations on segmentation performance. The figure shows the model input—comprising the original image with point annotations, the resulting segmentation map, and an overlay of the segmentation prediction on the original image—progressing from a single annotation to five annotations.

Fig. 6Full-screen View

(Color online) Performance comparison between our approach and neural network-based interactive methods under increasing point annotations. (a–e) illustrate the model predictions corresponding to one to five point annotations, respectively. Each column includes the original image with point annotations, the segmentation map, and the overlay of model predictions. For comparisons with neural network-based interactive methods (human-in-the-loop), each column presents the output from the original input, the modified input with additional background annotations, and the final segmentation results

As the number of point annotations increased, the model progressively learned the approximate shapes and distinctive features of helium bubbles, thus yielding more accurate and stable segmentation outcomes. Because of only one or two annotations (Fig. 6a and b), the model struggled with inconsistency and yielded erratic segmentations that failed to capture the actual boundaries of the bubbles. However, by introducing three or four annotations (Fig. 6c and d), the model began to effectively recognize the essential characteristics of helium bubbles and successfully segmented most of them, albeit with occasional omissions. Further increasing the number of annotations to five (Fig. 6e) significantly enhanced the performance, thus enabling the model to segment the helium bubbles almost completely. At this stage, the model not only captured most of the bubbles but also began to approximate their shapes more precisely by replacing irregular boundaries with smoother contours. Moreover, it became less sensitive to variations in contrast and noise, thereby achieving robust segmentation across all helium bubbles. This progression highlights the ability of the model to generalize better as more annotations are provided; specifically, its segmentation capabilities are refined by leveraging the increased information regarding the bubble morphology.

This observation suggests that the model effectively leverages the limited user input, learns the visual features of helium bubbles from a few point annotations, and extrapolates this information to achieve high-quality segmentations. A quantitative analysis of this trend is presented in Table 2, which presents the F1 scores for various numbers of annotations across multiple images. In general, the F1 score exhibited an increasing or stabilizing trend with as the number of user-provided points increased. For instance, in Image 1, the F1 score began at 0.623 with a single annotation and then improved marginally to 0.634 with two annotations. A notable increase was observed with three annotations, where an F1 score of 0.712 was achieved. Four annotations further enhanced the score to 0.723, and five annotations yielded an F1 score of 0.801, thus demonstrating the exceptional performance of the model. The results for the other images corroborated this trend, thereby highlighting the ability of the model to efficiently learn the features of the target object from a limited number of interactive annotations, which consequently yielded accurate and comprehensive segmentations. This capability is advantageous for the rapid analysis of cells, particles, defects, and other entities in various applications.

The training time of our method was affected by the image size and the number of helium bubbles. In the test experiment, under an image size of 768×768 pixels, the model converged in approximately 300 iterations, which required approximately 10 min. By contrast, the human-in-the-loop [40] method converged within 2000 iterations per training cycle on average, with a runtime of approximately 5 min. The human-in-the-loop interactive segmentation approach demonstrated significant limitations when trained solely on helium-bubble labels. It incorrectly classified substantial portions of the ambiguous background and other defects, such as helium bubbles (column 4 in Fig. 6). Even with additional background annotations (column 5 in Fig. 6), the training remained unstable, as evidenced by noisy outputs and persistent defect misclassifications. This limitation stems from the inability of the method to effectively extract helium-bubble features from the annotations provided, which requires comprehensive background labeling of all defects for accurate performance.

This challenge is particularly significant as helium bubbles are smaller than 1 nm, which renders precise contour delineation extremely difficult and time intensive. Our method addresses this limitation by requiring only central point annotations for helium bubbles, thereby reducing the labeling time by approximately 90%. This efficiency is achieved by implementing Gaussian filtering in the GCM, which transformed point annotations into probability distributions, i.e., high at the center and gradually decreasing outward—, thus creating more informative spatial labels compared with simple binary annotations.

By contrast, the conventional Mask R-CNN approach required extensive training data: 389 images containing 33,517 annotations, with an average of 86 helium bubble labels per image. Although such a comprehensive annotation is valuable for diverse, large-scale datasets, it is impractical for typical applications or smaller datasets. Our method achieved comparable results with only five point annotations per image, i.e., approximately 6% of the original labeling effort, thus rendering it significantly more efficient for practical applications.