Statistical Method

Statistical methods require offline data storage and utilize Gaussian fitting and the center-of-gravity method to locate and fit trajectories [30]. The center-of-gravity method involves summing all the pixel values in each column, thereby compressing one frame into one row, creating a one-dimensional distribution of the frame’s values. Subsequently, N-order Gaussian fitting is applied to this one-dimensional distribution to obtain μ and σ for each trajectory in every frame. For instance, single-frame data with four trajectories require a sum of four one-dimensional Gaussian distributions. The region of each track was manually determined based on the μ and σ of the Gaussian distribution. Then, the center-of-gravity of each row is calculated within the region of the trajectory, and the slope (k) and intercept (b) are extracted by fitting each row’s center of gravity. Statistical methods do not rely on image features to analyze data and can directly identify the positions of trajectories from raw data. However, the fitting process in this method requires manual interaction. Furthermore, this method generates substantial offline data, which makes it inadequate for our requirements of online automated positioning.

Computer Vision Methods

Prior to the advent of artificial intelligence, the detection of track-like objects in an image relied on leveraging visual information. The core approach involved extracting visual cues through various image processing techniques, such as utilizing the HSI color model [31] and edge extraction algorithms [32, 33], where the color and shape of the track boundary are the keys to the process [34, 35]. Furthermore, post-processing methods such as Markov random fields and conditional random fields, were employed to enhance the detection accuracy [36, 36]. With advancements in machine learning, several methodologies have been proposed that utilize algorithms, such as template matching and support vector machines [38, 39]. For instance, SCNN [40] introduced a specialized convolution operation in the segmentation model to utilize features more efficiently. Other studies have focused on developing lightweight methods suitable for real-time applications [41-43]. However, these methods rely on image quality and encounter difficulties when processing images with strong background noise.

Lane Detection Methods

The reconstructed beam tracks closely resemble lane lines. Hence, lane detection methods are good candidates for trajectory location after adjusting part of the network structure according to the beam track features. The main algorithm design concepts are as follows: Beam trajectory detection in the segmentation problem can be abstracted by classifying N × M pixels in the image. However, this approach is extremely slow and requires an additional segmentation head to the backend of the network. [44] converted this problem into a row-wise detection problem in which detecting the presence of beam trajectory features in each row is equivalent to classifying N rows with M dimensions, allowing the network to explicitly establish relationships between different rows. This approach bridges the original semantic gap caused by low-level pixel-level modeling and high-level trajectory long-line structures and substantially simplifying the N × M classification problem. Another idea, based on the work of high-level semantic information in [45] to guide low-level semantic information. The network can realize detection more accurately by fusing beam trajectories, such as long bars, color highlighting after visualization, slope uniformity, and other features of different levels. Traditional lane detection methods usually define two lane lines with a set of prior points and then recognize lane lines with similar shapes and locations. In addition, these methods typically have difficulty in processing multiple trajectories simultaneously.

Object Detection

The task of object detection aims to identify all targets (objects) of interest in an image and determine their categories and locations. Object detection involves three main methods: proposal-based, anchor-based, and anchor-free methods. The proposal-based method searches all generated regions to determine whether any region includes an object [46]. This method can achieve high accuracy but requires significant time. An anchor-based method was used to match a predefined set of anchors [47]. Due to the different sizes of the anchors, the network can recognize objects of different sizes. The anchor-based method is faster but requires a non-maximum suppression process for post-processing. The anchor-free method is similar to a segmentation method. It outputs feature maps of different sizes to determine whether an object is included in the feature map [48]. However, this method still requires post-processing and has lower accuracy compared to the proposal-based method. High robustness is required for the object detection method model to achieve high speed and accuracy.

Comparision and discussion

• The trajectories generally exhibit elongated shapes with the width being due to charge diffusion during the drift process. In addition, they are susceptible to irregular background noise, which appears as discrete energy points at considerable distances with similar intensity to the trajectories. Therefore, directly using the computer vision method to determine the continuity conditions makes it challenging to locate and return the overall beam trajectory.

• The number of trajectories in a single frame is uncertain; each frame may contain ten or more trajectories. Lane detection methods, influenced by pre-defined prior lane, cannot simultaneously locate all trajectories.

• The beam trajectory detection task requires high speed and positioning accuracy with online processing. Therefore, we cannot use the statistical method that requires manual interaction, the anchor-free method with low accuracy, or the proposal-based method with slow inference speed. We need to design a new network with an efficient structure that can be accelerated on different platforms to achieve the speed and accuracy requirements.

• FML [49] is previous work from our group designed for the previous Hi’Beam-SEE [19], from which the real trajectory feature is that a single frame contains one or a few (typically no more than 5) large-angle trajectories. Because of the track features with large inclination angles, we designed a pre-defined slot structure for FML based on the lane detection method to improve its processing speed. In the upgraded Hi’Beam-SEE in this work, the data feature is that a single frame contains multiple tracks (up to more than ten) with small angles. We conducted experiments and analyzed the results, which revealed that it is difficult for FML training to converge when the number of tracks increases, especially when multiple adjacent tracks can be easily misclassified as a single track, resulting in a high miss detection rate. In addition, the slot-structure design of FML grids the frame to improve the processing speed (a grid contains multiple pixels, and the network first determines whether the grid includes a track and then determines the pixels in each grid that may belong to a track). However, because the coordinates of the tracks in the column direction are relatively close, it is easy to match local tracks from different rows on the same column. Therefore, an upgraded Hi’Beam-SEE requires a new method to process multiple small-angle tracks simultaneously.

Hence, the Online Multi-particle Locating (OML) algorithm for extracting the positions of particles is designed based on anchor-based object detection method.

The Backbone

The input image size is 128×512×3 (128 and 512 are the rows and columns of the image, respectively, and 3 is the RGB color channel of the image) and is computed in the first set of blocks in each stage by undergoing a set of convolutions with a stride of two. By observing network structures, including ResNet152 [50], Swin Transformer [51], and ConvNeXT [52], we found that the addition of convolution operations in the third and fourth stages can effectively improve the network’s ability to generalize the data. This is because the expression of high-dimensional features in images is more critical than the expression of low-dimensional features. Therefore, we applied this relevant experience and repeated the convolution operation for 4, 6, and 16 times in stages 2,3,4, respectively. After each stage, the size of the feature map is halved, and the dimensions of the feature map are doubled. Table 1 demonstrates the architectural specification of the backbone.

The upper right corner of Fig. 2 shows the backbone structure. The structures of the training, inference, and merge-and-infer phases are shown from left to right. The module structure in the training phase has three branches: the middle branch (main branch) uses a convolution kernel of size 3×3, batch normalization (BN), and a rectified linear unit (ReLu) function, and the first 3×3 convolution layers of each module use a step size of 2. The bypass branch uses a convolution kernel of size 1×1 and BN. The other bypass branch used a separate BN operation. The main branch sums the results of the two bypass branches in dimensions after the second convolution and BN operations (only the channel is changed, not the dimensions).

Fig. 2Full-screen View

The structure of the anchor-box-based object detection approach is as follows: The backbone part uses a parameterizable VGG-style sub-module. The neck part obtains information from different backbone stages and combines it. In the final stage, high-level features are superimposed on low-level features to improve the performance. In the head part, the feature map containing real beam trajectories is first cropped, and after strip convolution, it is combined with the resized scene. Finally, the coordinates and shapes of the particle trajectories are regressed, and the loss is calculated

The BN layer accelerates the convergence speed of the model by normalizing the data and reducing overfitting, gradient explosion, and gradient disappearance. The principle is illustrated in Eq. (1). μi, σi are the mean and variance of the channel i, respectively. ϵ is a minimum parameter set to prevent σi from being zero, and γi, βi are used to adjust the variance and mean value of the data distribution in each batch. $y_i=\frac{x_i-{\mu }_i}{\sqrt{{\sigma }_i^2+ϵ}}\cdot {\gamma }_i+{\beta }_i$ (1) When the training phase is completed, μ, σ, γ, β become constant. The BN layer can then be integrated into the weight update process of the convolution kernel, and the final weight formula after fusion is $W_{i\mathrm{,}:\mathrm{,}:\mathrm{,}:}^{\text{'}}=\frac{{\gamma }_i}{{\sigma }_i}{W}_{i\mathrm{,}:\mathrm{,}:\mathrm{,}:}\mathrm{,}{b}_i^{\text{'}}={\beta }_i-\frac{{\mu }_i{\gamma }_i}{{\sigma }_i}\mathrm{,}$ (2) where W and B are the convolution kernel weights and bias after fusion, respectively. Therefore, the calculation formula after integrating the convolution, BN, and ReLu layers was as follows: $y_i=\text{Max}\left(W_{i\mathrm{,}:\mathrm{,}:\mathrm{,}:}^{\text{'}}{x}_i+b_i^{\text{'}}\mathrm{,0}\right)\mathrm{.}$ (3) The Conv1×1+BN branch can be regarded as a convolution BN fusion layer with a core size of 3×3 (only the center point has a 1×1 convolution kernel weight, and the eight surrounding points are 0). The BN branch can be regarded as a 3×3 convolution BN fusion layer with a weight of one. The process in Eq. (3) and Fig 3 is called reparameterization.

Fig. 3Full-screen View

(Color online) After unifying the network size of the bypass branch to be consistent with the main branch, add their weights. This operation can remove the bypass branches and connection layers and unify the scale of the convolution layer while retaining the data characteristics learned by multiple bypass branches

After reparameterization, we obtain the module structure in the inference phase, which improves the speed of inference by removing the bypass branches and maintaining the backbone. Subsequently, we integrate the different layer weights obtained from their training into the fusion layer. Therefore, the final fusion layer formula is $y_i=\text{Max}\left({W^{″}}_{i\mathrm{,}:\mathrm{,}:\mathrm{,}:}{x}_i+{b^{″}}_i\mathrm{,0}\right)\mathrm{,}$ (4) ${W^{″}}_{i\mathrm{,}:\mathrm{,}:\mathrm{,}:}={W^{\prime }}_{i\mathrm{,}:\mathrm{,}:\mathrm{,}:}+W_{i\mathrm{,}:\mathrm{,}:\mathrm{,}:}^{\text{Conv}+\text{BN}}+W_{i\mathrm{,}:\mathrm{,}:\mathrm{,}:}^{\text{BN}}\mathrm{,}$ (5) ${b^{″}}_i={b^{\prime }}_i+b_i^{\text{Conv}+\text{BN}}+b_i^{\text{BN}}\mathrm{,}$ (6) Through reparameterization and layer fusion, we have integrated the nine-layer module structure into three layers, resulting in an accelerated structure for the merge-and-infer phase. This structure can significantly speed up network operations and effectively reduce hardware overhead.

The Neck

The Feature Pyramid Network (FPN) [53] has become the standard neck model because it integrates seamlessly with backbones such as ResNet, and can fuse different levels of features, making it applicable to various application areas such as target detection and segmentation. Utilizing small- or medium-sized backbone networks can reduce computational costs and improve efficiency. However, shallow backbones typically provide fewer rich semantic features for object detection. Most SOTA (state-of-the-art) methods use deep CNN structures to improve the accuracy of irregular object detection but this approach is not suitable for Hi’Beam-SEE. Because the last-stage’s feature map size of our beam data is 16×64, the trajectories in the images are compressed into features with only a few pixels in the last layer of the feature pyramid, which is insufficient for accurate trajectory recognition. Considering that the features of particle trajectories in Hi’Beam-SEE are different from those of ordinary target detection objects, it is vital for the neck to extract the accurate features of the particle beam trajectories with long shapes and a high-noise background, allowing the head to efficiently use the features. Additionally, the detection of irregular objects requires high-level semantics to distinguish objects in the background from low- or medium-level features for accurate object localization. For this reason, we designed a new neck structure called Feature Fusion Component (FFC) to create a feature map with less feature loss when transforming between different scales, and to retain more detailed and integrated features. As shown in Fig. 4, FFC obtains original information from different stages in the backbone. The features of the middle stage are extracted as the main features of the DW-Conv module. The larger the size of the convolution kernel, the larger the receptive field, especially on the particle beam trajectory dataset. High- and low-level features were aligned in size using bilinear difference and regular convolution, respectively. The features are superimposed and can contain information of different scales after a 1×1 convolutional fusion layer.

Fig. 4Full-screen View

FFC detailed structure

Head Part

The head requires features extracted from the backbone and neck for different detection and segmentation tasks. As the particle trajectories have various shapes, colors, locations, and degrees of association with the background, compared with the traditional model, the loss function must be redesigned to improve the object localization accuracy. Referring to the concepts of conditional convolution and prior LaneNet works, we use position prior knowledge to aid in detecting trajectories and propose a composite loss function consisting of multiple parts, which can further improve the detection accuracy of the network.

3.3.1

ROI Extractor

The trajectories respond more strongly to high-level features, whereas low-level features relate more to the interaction information with background noise and edge information. Therefore, it is important to allow the network to recognize trajectories as a whole by simultaneously using different stage features. In addition, it is difficult to use only a detection head and a single-stage feature for trajectory location. In addition, more comprehensive and finer features can help improve the localization accuracy. Therefore, our aim is to utilize Conv’s gradually increasing receptive field and gradually decreasing feature hierarchy, which has a semantic hierarchy from low to high, and build a fused feature pyramid network with different levels of semantic information.

By clustering the particle trajectory data in the dataset, we can assign trajectory priors to each feature map and make the network more interested in the regions where trajectories are likely to exist (ROI). However, additional contextual information regarding these features is required. In corner cases, the particle trajectories may be incomplete. Therefore, the network lacks local semantic information to determine whether a trajectory exists and whether the background noise and trajectory are misclassified. Hence, determining whether a pixel belongs to a trajectory must be aided by pixels surrounding the trajectory.

This was also demonstrated in nonlocal experiments, showing that the adequate utilization of long-range dependencies can improve performance. Therefore, we incorporated more background noise to interact with the trajectories, enabling the network to distinguish them better. Through convolution operations on the ROI, further connections can be made between the pixels of the trajectory and possibly reconstruct incomplete areas. The enhanced ROI were then subjected to an attention operation with the original feature map to establish a mapping relationship between the ROI and the background. This approach further exploits background noise, resulting in a more robust model.

Figure 5 presents the detailed structure of the ROI Extractor. To address the difference between the slender shapes of the trajectories and the rectangular detection box in classical object detection methods, we used bar-shaped convolution for feature extraction. We found that a convolution kernel with a kernel size of 13 works best, and we deployed these convolution kernels in both row and column directions within the ROI Extractor.

Fig. 5Full-screen View

ROI extractor detailed structure

3.3.2

Loss Function

Conventional loss function designs typically involve classification and regression components. However, our findings reveal that this approach does not enable the network to describe trajectories precisely through multiple backpropagation iterations. This limitation arises from the use of the conventional L1 loss function, which calculates the regression loss by dispersing the points during computation instead of considering them as a cohesive unit. Consequently, the regression accuracy failed to satisfy the experimental requirements of Hi’Beam-SEE, particularly in scenarios with high-noise backgrounds. To address this issue, we introduce a composite loss function consisting of three components: IoU (Intersection over Union) loss, cls(classification) loss, and xykb(start position with (x,y), slope, and intercept) loss.

IoU loss is the most important aspect for predicting and regressing a trajectory as a whole. As shown in Fig. 6, the blue line is the set of real trajectory line coordinates, and the red line is the set of network-predicted trajectory line coordinates. First, we consider a horizontal coordinate xi, where the corresponding real coordinate is $x_i^{\text{G}}$ the predicted coordinate is $x_i^{\text{P}}$, and its horizontal deviation is di. If a small range of perturbations is added to $x_i^{\text{G}}$ and $x_i^{\text{P}}$ as e, the IoU of point xi is calculated as $\begin{array}{c}{\text{IoU}}_{x_i}=\frac{d_i^{\text{Offset}}}{d_i^{\text{Horizontal}}}\\ =\frac{\min \left(x_i^{\text{P}}\mathrm{,}{x}_i^{\text{G}}\right)-\max \left(x_i^{\text{P}}\mathrm{,}{x}_i^{\text{G}}\right)}{\max \left(x_i^{\text{P}}+e\mathrm{,}{x}_i^{\text{G}}+e\right)-\min \left(x_i^{\text{P}}-e\mathrm{,}{x}_i^{\text{G}}-e\right)}\end{array}$ (7) If the trajectory line is discretized into a set of N points, the IoU loss is ${\text{IoU}}_{\text{Line}}=1-{\displaystyle \sum _{i=1}^N{\text{IoU}}_{x_i}}=1-\frac{{\displaystyle {\sum }_{i=1}^N{d}_i^{\text{Offset}}}}{{\displaystyle {\sum }_{i=1}^N{d}_i^{\text{Horizontal}}}}$ (8) To better detect the trajectory using prior knowledge, we also need xykb loss to predict the coordinates, slope, and intercept of the starting point of each trajectory. In addition, we added focal loss for the classification of different types of particle trajectories. The total loss formula is as follows: ${\text{Loss}}_{\text{Total}}={\text{Loss}}_{\text{IoU}}+{\text{Loss}}_{\text{xykb}}+{\text{Loss}}_{\text{focal}}$ (9)

Fig. 6Full-screen View

(Color online) IoU loss calculation method

Track Fitting

The detected trajectory bounding boxes are represented in the form of vertex coordinates. Subsequently, the center of gravity of each row was calculated along the trajectory within the bounding box. Least-squares fitting was then performed on all the rows’ centers of gravity, yielding the corresponding trajectory slope and intercept. The fitting accuracy of each row was assessed by calculating the standard deviation Δ P between the center of gravity of each row and the fitted point in each row. Then, the positioning accuracy of each trajectory was calculated using Eqs. (11), where PA is the positioning accuracy of the trajectory, $\overline{\text{Δ}P}$ is the mean of the standard deviation of all the rows in the bounding box. The visualization results are presented in Fig. 7. $y_{\text{exp}}=k\times x_m+b$ (10) $\text{PA}=\sqrt{\frac{1}{n\left(n-1\right)}{\displaystyle \sum _{i=1}^n\left(\text{Δ}{P}_i-\overline{\text{Δ}P}\right)}}$ (11)

Fig. 7Full-screen View

(Color online) The visualization of the track fitting process. The red line represents the ground truth for each row and the blue line represents the fitted trajectory

Dataset

The raw data contained either the heavy-ion trajectories collected at HIRFL or the laser trajectories collected in the labs at low density. These two trajectory types exhibit different image features, necessitating two separate categories for labeling. The OML must effectively handle various particle flux densities to ensure that the Hi’Beam-SEE fits different test terminals. Therefore, images covering various cases were generated based on the raw data to build the dataset.

The dataset image generation involves the following steps. First, background noise reduction is performed on the raw data using a 9×9 convolutional kernel. Next, particle tracks extracted from pre-processed raw data were merged to generate dataset images covering trajectories with varying types, numbers, and spacing. Also, part of the pre-processed images containing a single trajectory was directly included in the dataset. Finally, Gaussian noise was randomly injected into the image to simulate the noise that may have arisen from different experimental terminals. The final dataset contained more than 3000 images. Figure 8 shows example images of the 12 cases covered by the final dataset: images with low-, medium-, or high-density heavy-ion trajectories or laser trajectories; images with mixed-type trajectories at different densities. The dataset was then split into training and test parts in a ratio of 8:2. We tested the experimental performance of our model using the test set.

Fig. 8Full-screen View

Visualization the example cases. (a) Image with single laser trajectory. (b) Image with single heavy-ion trajectory. (c) Image with medium-density laser trajectories with different locations. (d) Image with medium-density heavy-ion trajectories with different locations. (e). Image with high-density laser trajectories at different locations. (f) Image with high-density heavy-ion trajectories with different locations. (g) Image with low-density mixed heavy-ion and laser trajectories. (h) Image with medium-density mixed heavy-ion and laser trajectories. (i). Image with high-density mixed heavy-ion and laser trajectories. (j). Image with high-density mixed heavy-ion and laser trajectories

Comparison Experiments and Ablation Study

This subsection compares the performance of CLRNet, yolov7-base, the center-of-gravity method, and OML. Additionally, ablation experiments for each component of OML were conducted separately to verify the effectiveness of the design. The performance metrics included the miss detection rate, false detection rate, speed, and positioning accuracy. The miss detection rate is defined as the proportion of undetected trajectories to the total number of trajectories; the false detection rate is the proportion of incorrectly detected trajectories to the total number of detected trajectories; the speed is the number of frames containing trajectories per second that OML can process; and the positioning accuracy is the average value of all trajectories in the test dataset.

According to the results presented in Table 2, our OML demonstrates good performance in terms of miss detection and false detection rates, maintaining high processing speed and positioning accuracy. Notably, the yolov7-base model struggles, particularly in nonvertical trajectory scenarios, primarily because of its object detection principle’s inability to accurately detect bounding boxes with rotation angles, resulting in increased IoU loss. However, CLRNet satisfactorily fulfills the detection requirements in scenarios with small tilt angles and provides a fast processing speed. However, CLRNet faces difficulties in dealing with multiple trajectories, which leads to a high missing detection rate. The center-of-gravity method cannot locate trajectories online, although it has good sounding performance.

In the ablation study, yolov7-base was utilized as the baseline model, gradually replacing its components with the corresponding parts of OML. Table 3 lists the results of this study. The results indicate that after reparameterization, the backbone component exhibits significant improvements in positioning accuracy while reducing the parameter size, thereby achieving a higher model inference speed of 188 fps. However, the obtained accuracy still fails to meet the demands of real-time detection. Substituting the FFC neck, which enables the fusion of different stage features, and the ROI head, which focuses on the entire trajectory, further improves accuracy. Finally, the overall model was constructed by optimizing the loss function, striking a balance between detection accuracy and speed that aligned with our expectations.

Figure 9 shows the visualized final detection results obtained using the OML, corresponding to the example images in Fig. 7. The results demonstrate the capability of OML to accurately detect both heavy ion and laser trajectories, even in the presence of high background noise. The model exhibited exceptional robustness when faced with trajectories of different angles and densities. However, it is worth noting that OML still encounters certain challenges, as illustrated by the heavy-ion trajectories on the right side of Fig. 9f. The excessive intensity of background noise poses difficulties during data preprocessing, making it challenging for OML to locate the center of the trajectory precisely, resulting in low positioning accuracy. However, such corner cases occur with a relatively low probability. Overall, OML satisfies the detection requirements of the Hi’Beam-SEE.

Fig. 9Full-screen View

Visualization of OML test results