Launching point of developing the new method: Limitations of the current idea

Improving the accuracy of SR-CT reconstruction results is the premise for further research given that the detailed information in the reconstruction results typically contains important structures. A detailed analysis of the CT imaging principles is necessary to develop a new Ultra-sparse angle SR-CT optical measurement method. A schematic of projection acquisition conducted using SR-CT is shown in Fig. 2(a). The mathematical model for generating a projected sinogram is represented by , where RL and f(x, y) represent the projected integral intensity along the X-ray and target to be detected, respectively. The sinogram was obtained by integrating the tomogram, and therefore, some detailed signals in the tomogram f(x, y) were buried in the sinogram. The curve shown in Fig. 2(b) can be obtained by integrating the tomogram f(x, y) along the X-ray direction. Regions of interest (ROI) are marked by red arrows in Fig. 2(a). The difference between the red and blue curves in Fig. 2(b) is whether there are small particle in the ROIs in the tomogram. The difference between red and blue curves was of 1.507%, which is indeed minimal. Tiny structural information in the tomogram can easily be buried in the projected sinogram.

Fig. 2Full-screen View

(Color online) (a) Tomogram f(x, y). The red arrow marks small particles of interest. (b) Integral curve of Fig. 2(a) along the X-ray direction. (c) Gradient of tomogram. (d) Integral curve of Fig. 2(c) along the X-ray direction. (e) Schematic of the process for extracting high-frequency information

However, accurate representations of detailed information have not been considered by current deep learning ideas. The sinogram is integral to the tomogram along the X-ray direction, and therefore, high-frequency information of internal details can be easily lost. Considering the idea of a DDC-Net as an example, the lost details cannot be recovered in subsequent optimization in the tomogram domain if the high-frequency information cannot be observed in the sinogram domain. This will lead to a distorted reconstruction result, limiting its application potential in accurate SR-CT characterization. Therefore, adding high-frequency information constraints on deep neural networks is necessary to improve the representation of the detailed information.

The gradient transformation of the tomogram can highlight the expression of detailed information in its sinogram. The integral curve in Fig. 2(d) reflect this characteristic. The contribution of detailed information to the integral value is indicated by the difference between the peak values of labeled points that refer to the integral values with and without the small particle in the ROI region. Compared with 1.507% in Fig. 2(b), the difference ratio of the green marker points in Fig. 2(d) is as high as 14.23%. Therefore, we add “high-frequency information” constraints in the neural network to learn the detailed information of the tomogram.

In response to this problem, “high-frequency information” constraints are added to drive the learning direction of the neural network, as shown in Fig. 2(e).

The “high-frequency information” constraint is implemented in two steps: perform gradient transformation on tomogram f(x, y) to obtain G(x, y), and perform radon transformation on the gradient image G(x, y).

In recent years, the continuous development of deep learning has provided new ideas to solve serious ill-posed problems such as ultra-sparse angle SR-CT reconstruction. The dual domain learning idea is used considering the physical process of CT reconstruction. Given this context, emphasizing the expression of high-frequency information in a neural network can help improve the accuracy of the reconstruction results. Therefore, designing a neural network considering the high-frequency detail information is a primary task for alleviating the problem of ultra-sparse angle SR-CT reconstruction.

Finally, a novel method suitable for accurate SR-CT characterization under ultra-sparse angle conditions is proposed. This method is referred to as HFIC-Net.

Deployment of high-frequency information constrained neural network

The HFIC-Net framework is illustrated in Fig. 3. The HFIC-Net comprises two deep neural networks G₁ and G₂. These networks drive learning in the sinogram and tomogram domains, respectively. In the sinogram domain, G₁ restores the sparse angle sinogram to a high-quality sinogram. Assuming that as an ultra-sparse angle sinogram input for HFIC-Net, it is converted to a fully sampled sinogram by mapping G₁. Subsequently, is converted to the tomogram domain through the FBP algorithm to obtain the reconstructed result . Next, high-frequency information on is extracted to obtain . Then, mapping G₂ performs super-resolution reconstruction on the tomogram, yielding . Finally, the loss of the HFIC-Net comprises sinogram content loss L₁, high-frequency information loss L₂, and tomogram content loss L₃. G₁ and G₂ are updated in reverse through gradient descent.

Fig. 3Full-screen View

(Color online) (a) Architecture of the proposed HFIC-Net. (b) Network structures of G₁ and G₂ of the HFIC-Net. G₁ and G₂ are five-layer deep neural networks with the same input size and output size as 256 × 256 × 3

Sinogram content loss L₁: In the sinogram domain, the mean square error (MSE) loss between the high quality sinogram and real label is used as the sinogram content loss L₁. The richness of the sampled projection information directly determines the quality of the tomogram. The quality of reconstructed tomogram can be effectively improved if the degradation degree of the projected sinogram can be reduced. Mathematically, the sinogram content loss of HFIC-Net can be expressed as(1)where xi, , G₁, N, and θ represent the input sparse angle projection, real full angle projected sinogram, mapping in the sinogram domain, number of training data pairs, and training parameters in the entire network, respectively.

High-frequency information loss L₂: The high-frequency information loss is added in the network considering the importance of internal detail information. The MSE loss between the high- frequency information feature map and real label is used as the high-frequency information loss. Mathematically, the high-frequency information loss of HFIC-Net can be expressed as(2)where xi, , G₁, , N, and θ represent the input sparse angle projection, real label, mapping in the sinogram domain, high-frequency information extraction operation, number of training data pairs, and training parameters in the entire network.

Tomogram content loss L₃: In the tomogram domain, the mean square error between the high-quality tomogram y generated by G₂ and real label is used as the tomogram content loss. Some small errors in the sinogram domain are considerably magnified after FBP reconstruction. Therefore, it is necessary to achieve further improvements in the tomogram domain. Mathematically, the tomogram loss of HFIC-Net can be expressed as(3)where xi, f_bp, , G₁, N, and θ represent the input sparse angle projection, FBP reconstruction, real tomogram, mapping in the tomogram domain, number of training data pairs, and training parameters in the entire network, respectively.

The final objective of the proposed HFIC-Net is defined by combining these three losses as(4)Network structure of the HFIC-Net: As shown in Fig. 3(b), G₁ and G₂ are composed of an encoding–decoding neural network. The coding module is used to extract feature information from the input image. The encoder comprises five convolutional layers; the size of convolution kernel is 4 × 4, the step size is 2, and the number of channels in each layer is 64, 128, 256, 512, and 512, respectively. The activation functions of these five convolutional layers are all Relu. Batch normalization is performed after the activation of each layer. The decoder comprises five layers of deconvolution. The decoding module recombines the acquired feature information into an image. The first four layers of the convolution kernels are 4 × 4 in size, with a step size of 2, and the number of channels are 512, 256, 128, and 64. The fifth-layer network convolution kernel is 4 × 4, the step size is 2, and the number of channels is 3. The activation functions of these five convolutional layers were all Relu. The output size of HFIC-Net is the same as the input; both are 256 × 256 × 3.

Training configuration and performance evaluation

The HFIC-Net was verified using simulation and real experimental data. All training work was conducted on Inter(R) Core(TM) i7-8700 3.20 GHz CPU and NVIDIA RTX 2070 GPU. All experiments were run in the environment of Python3.7, with CUDA10.0 and CUDNN-v7.4 for acceleration, and the TensorFlow deep learning framework to implement the proposed method. We applied the Adam optimizer for the HFIC-Net. The learning rate was fixed at 0.0002, and the exponential decay rates for the moment estimates in the Adam optimizer were β₁ = 0.5 and β₂=0.999. The weight balance parameters of different losses were set to 1.

If the HFIC-Net method was used in in situ CT analysis in a training environment based on Inter(R) Core(TM) i7-8700 3.20 GHz CPU and NVIDIA RTX 2070 GPU, it would incur the following computation cost: (1) Preprocessing time for forming the dataset: This part of the time referred to the time required to generate low-quality projected sinograms, high-frequency information constraints, and sinogram labels. This part of the time was approximately 354 s. (2) HFIC-Net training time: The training time for the HFIC-Net was approximately 33 h. The actual computational cost of deploying HFIC-Net would be approximately 33 h more than that when not using this method.

Quantitative parameters were adopted to evaluate the HFIC-Net. The parameters were used for evaluating the difference between the reconstructed and original images, including (1) structural similarity index (SSIM), (2) normalized mean square criterion D, and (3) normalized average absolute distance criterion R [39, 40]. These parameters were respectively calculated as(5)(6)(7)where represents the pixel value of the original image; represents the pixel value of the reconstructed image; and represent the average of all pixel values in an image; N represents the total number of pixels in the image; σu and σv represent the standard deviations; and σu v represents the covariance. Constants C₁ and C₂ are set as in [40]. Parameter SSIM is a criterion for structural similarity between the reconstructed and original images, and the value range is [0,1]. The larger the value, the higher is the reconstruction accuracy. Parameter D and R are used to evaluate the relative errors of reconstruction. Parameter D indicates a large deviation in few points of the reconstructed image, while R indicates a small deviation in most points of reconstructed image. The smaller the parameter values of D and R, the higher is the reconstruction quality.

Reconstruction results of simulation data

3.2.1

Comparison with other methods based on simulation data

The effectiveness of the HFIC-Net was verified through numerical experiments with simulation data. The simulation data were a series of randomly generated particle images: 6400 randomly generated images were used as the training set, and another 100 images were generated as the testing set. A complete sinogram was sampled in the range of 180° to be used as a label for the sinogram domain. Then, an ultra-sparse angle sinogram of 8 projections was used as the input of HFIC-Net, and high-quality model images were used as the label for tomogram domain. Finally, the reconstruction tomogram was output.

The effectiveness of the new method was verified by comparing the simulation data test results with those of the most commonly used FBP method (Table 1). SSIM increased nearly 500% by the HFIC-Net compared to the result of the FBP with eight angles. In addition, the HFIC-Net can only use eight angle projections to achieve the reconstruction effect of the FBP method in 360 projections. Figure 4(c) and (d) appear to be almost consistent. A comprehensive quantitative evaluation revealed that the SSIM of the image structure similarity increased from 0.1951 of the eight-angle FBP to 0.9620. Further, in terms of image relative error, the HFIC-Net was superior or equivalent to that of the full-angle FBP.

Table 1Full-screen View

Quantitative evaluation (100 testing images)

	FBP (8 angles)	FBP (360 angles)	HFIC-Net (8 angles)
arg.SSIM	0.1951	0.5319	0.9620
avg.D	1.2968	0.2783	0.0978
avg.r	1.2890	0.2192	0.0492

Show more

Fig. 4Full-screen View

(Color online) (a) Original image. (b) FBP reconstruction results at eight angles. (c) FBP reconstruction results at 360 angles. (d) HFIC-Net reconstruction results at eight angles

The reconstruction quality of the HFIC-Net was compared with that of the FBP-Conv, DDC-Net, and SART-FDTV-ASD [41] methods. The results of one of the 100 testing model with eight angles by several methods are illustrated in Fig. 5. Under the eight angles sampling conditions, although the result of FBP-Conv, SART-FDTV-ASD, and DDC-Net methods was considerably better than that of FBP, the details of the reconstruction results were biased. The reconstruction quality of the HFIC-Net was improved compared to those of the other methods. Image details were preserved, and internal artifacts were significantly suppressed. Figure 5(g)-(although the result of FBP-Conv, k) shows an absolute difference between results of each method and the original image to demonstrate the effect of the new method more clearly. The difference between the results of the HFIC-Net and original image was minimal. Figure 5(l) and (m) show the profiles along the blue and red solid line in Fig. 5(a), respectively. Through visual inspection, the gray distribution value of the HFIC-Net method is the closest to the original image. The results of the other algorithms indicated by the red arrow indicate error information, which is far from the real label. The comparison results indicate that the proposed new method has advantages in detail characterization.

Fig. 5Full-screen View

(Color online) (a) Original image. (b)–(f) FBP, FBP-Conv, DDC-Net, SART-FDTV-ASD, and HFIC-Net reconstruction results at eight angles, respectively. (g)–(k) represent the absolute differences of (b)–(f) with respect to the original image, respectively. (l) Profiles along the blue solid line in Fig. 5(a). (m) Profiles along the red solid line in Fig. 5(a)

For a quantitative analysis, the average calculation results of 100 testing images are listed in Table 2. The reconstruction results of HFIC-Net are better than those of FBP-Conv, DDC-Net, and SART-FDTV-ASD. The parameter SSIM is increased from 0.1951, 0.7937, 0.9212, and 0.9309 of FBP, SART-FDTV-ASD, FBP-Conv, and DDC-Net to 0.9620, respectively. The parameter D was decreased from 1.2968, 0.2786, 0.1247, and 0.1448 of FBP, SART-FDTV-ASD, FBP-Conv, and DDC-Net to 0.0978, respectively. HFIC-Net achieved a good performance for parameter R that emphasized the small error of most points.

Table 2Full-screen View

Quantitative evaluation (100 testing images)

	FBP	FBP-Conv	DDC-Net	SART-FDTV-ASD	HFIC-Net
avg.SSIM	0.1951	0.9212	0.9309	0.7937	0.9620
avg.D	1.2968	0.1247	0.1448	0.2786	0.0978
avg.R	1.2890	0.0465	0.0851	0.1439	0.0492

Show more

This new method shows outstanding advantages in local detail representation. The ROI of red rectangle in Fig. 6(a) is enlarged in Fig. 6(b) to further demonstrate the performance of this new method. Figure 6(c)–(f) corresponded to the results of different algorithms for ROI, respectively. The major areas of visual difference are marked by red arrows. In Fig. 6(c), the reconstruction results of FBP had almost no effective information. Local detail structure information is submerged. In Fig. 6(d)–(f), the reconstruction results are wrong, i.e., original small particles disappear. In Fig. 6(g), the HFIC-Net reconstruction result demonstrated clear edges and accurate structures. The comparison results indicated that the proposed new method has advantages in detail characterization.

Fig. 6Full-screen View

(Color online) (a) Original image. (b) Local enlarged region of the original image. (c)–(g) Correspond to the results of FBP, FBP-Conv, DDC-Net, SART-FDTV-ASD, and HFIC-Net for ROI, respectively. Red arrows mark major areas of visual difference

Table 3 lists the quantitative evaluation indicators of local details. The new method has significant advantages in terms of SSIM. The parameter SSIM increases from 0.3679, 0.4000, 0.4144, and 0.4247 of FBP, DDC-Net, FBP-Conv, and SART-FDTV-ASD to 0.7318, respectively. The parameter D decreases from 1.6070, 1.4716, 1.2590, and 1.2445 of FBP, DDC-Net, SART-FDTV-ASD, and FBP-Conv to 0.7691, respectively. The parameter R decreases from 0.2214, 0.1693, 0.1437, and 0.1343 FBP, DDC-Net, SART-FDTV-ASD, and FBP-Conv to 0.0737, respectively.

Table 3Full-screen View

Quantitative evaluation of ROI

	FBP	FBP-Conv	DDC-Net	SART-FDTV-ASD	HFIC-Net
SSIM	0.3679	0.4144	0.4000	0.4247	0.7318
D	1.6070	1.2445	1.4716	1.2590	0.7691
R	0.2214	0.1343	0.1693	0.1437	0.0737

Show more

Validation of real experimental data

The HFIC-Net method was applied to the reconstruction of actual SR-CT experimental projection data to evaluate the effectiveness of the new method in practical applications. The experiment was conducted at the BL13W1 beamline of Shanghai Synchrotron Radiation Facility (SSRF). The real experimental data comprised a series of tomograms of particle samples. The training set consisted of 4300 tomograms, and another 50 were selected as the testing set to verify the training results. The number of sparse sampling angles was 8.

Compared to other methods, HFIC-Net also shows certain advantages. The results of one of the 50 testing model with eight angles obtained using several methods are shown in Fig. 7. Compared with the other algorithms, the new method improves the quality of reconstruction with a clear boundary and complete structure. The reconstruction results of FBP have serious truncation artifacts under the eight angles sampling conditions, and the detailed structural information is distorted. Compared with the FBP reconstruction results, the artifacts of FBP-conv, SART-FDTV-ASD, and DDC-Net method are significantly suppressed and the visual effect is improved. Unfortunately, in terms of the detailed structure, there exists considerable error information. The reconstruction quality of the HFIC-Net is improved when compared to the quality obtained using the other methods. Image details are also preserved, and the internal artifacts are significantly suppressed. Figures 7(g)–(k) show the absolute difference between the results of each method and the original image to demonstrate the effect of new method more clearly. Figure 7(l) and (m) indicated the profiles along the blue and red solid line in Fig. 7(a), respectively. Through visual inspection, the gray distribution value of the HFIC-Net method is the closest to the original image. The results of other algorithms indicated by the red arrow had error messages, which is far from the real label. The comparison results demonstrate that the proposed new method showed advantages in detail characterization.

Fig. 7Full-screen View

(Color online) (a) FBP reconstruction results at 180 angles. (b)–(f) FBP, FBP-Conv, DDC-Net, SART-FDTV-ASD, and HFIC-Net reconstruction results at eight angles, respectively. (g)–(k) are the absolute differences of (b)–(f) with respect to the original image, respectively. (l) Profiles along the blue solid line in Fig. 7(a). (m) Profiles along the red solid line in Fig. 7(a)

This new method also demonstrated advantages in local detail representation. The ROI indicated by the red rectangle in Fig. 8(a) was enlarged in Fig. 8(b) to further demonstrate the performance of this new method. Figure 8(c)–(g) corresponded to the results of different algorithms for ROI, respectively. The main differences between results of these methods and original images were marked by the red arrows. In Fig. 8(c), the detailed structure information of FBP reconstruction is almost completely lost. In Figs. 8(d)–(f), the reconstruction results are wrong: there is no particle gap. In Fig. 8(g), the HFIC-Net reconstruction result demonstrates clear edges and accurate structures. The comparison results demonstrate that the proposed method can realize accurate reconstruction.

Fig. 8Full-screen View

(Color online) (a) FBP reconstruction results at 180 angles. (b) Local enlarged region of Fig. 8(a). (c)–(g) Results of FBP, FBP-Conv, DDC-Net, SART-FDTV-ASD, and HFIC-Net at eight angles for ROI, respectively. Red arrows mark major areas of visual difference

With the rapid development of deep learning, some researchers have begun to construct deep neural network based on CT reconstruction process to overcome the bottleneck problem of in-situ SR-CT characterization of ultrafast evolution process. The deep neural networks are arranged in the sinogram and tomogram domains, and some progress had been achieved. However, a good visual effect does not indicate the accurate reconstruction of a tomogram. Therefore, we added a “high-frequency information constraint,” which can reflect the expression of real detail information based on DDC-Net to improve the high preservation of reconstructed results.

The proposed HFIC-Net achieved the best score on quantitative evaluations such as SSIM. The HFIC-Net recovers accurate structure and weak detail information under ultra-sparse angle conditions. Figure 8 shows that the HFIC-Net method accurately reconstructed the gaps between particles. This weak but important information was crucial for analyzing the mechanism of material evolution; however, other methods easily lost these details. In conclusion, the HFIC-Net method had advantages in image detail restoration, which is expected to aid in improving the accuracy of SR-CT characterization under ultra-sparse angle conditions.