Application of the asynchronous advantage actor-critic machine learning algorithm to real-time accelerator tuning

Application of the asynchronous advantage actor-critic machine learning algorithm to real-time accelerator tuning 1 2 3 first-author 1 2 3 4 1 2 3 1 2 3 corresp Corresponding author, chengch@mail.tsinghua.edu.cn Corresponding author, chengch@mail.tsinghua.edu.cn chengch@mail.tsinghua.edu.cn 4 1 2 3 Key Laboratory of Particle & Radiation Imaging (Tsinghua University), Ministry of Education, Beijing 100084, China Key Laboratory of Particle & Radiation Imaging (Tsinghua University), Ministry of Education, Beijing 100084, China Laboratory for Advanced Radiation Sources and Application, Tsinghua University, Beijing 100084, China Laboratory for Advanced Radiation Sources and Application, Tsinghua University, Beijing 100084, China Department of Engineering Physics, Tsinghua University, Beijing 100084, China Department of Engineering Physics, Tsinghua University, Beijing 100084, China State Key Laboratory of Intense Pulsed Radiation Simulation and Effect (Northwest Institute of Nuclear Technology), Xian 710000, China State Key Laboratory of Intense Pulsed Radiation Simulation and Effect (Northwest Institute of Nuclear Technology), Xian 710000, China This paper describes a real-time beam tuning method with an improved asynchronous advantage actor-critic (A3C) algorithm for accelerator systems. The operating parameters of devices are usually inconsistent with the predictions of physical designs because of errors in mechanical matching and installation. Therefore, parameter optimization methods such as pointwise scanning, evolutionary algorithms (EAs), and robust conjugate direction search (RCDS) are widely used in beam tuning to compensate for this inconsistency. However, it is difficult for them to deal with a large number of discrete local optima. The A3C algorithm, which has been applied in the automated control field, provides an approach for improving multi-dimensional optimization. The A3C algorithm is introduced and improved for the real-time beam tuning code for accelerators. Experiments in which optimization is achieved by using pointwise scanning, the genetic algorithm (one kind of EAs), and the A3C-algorithm are conducted and compared to optimize the currents of four steering magnets and two solenoids in the low-energy beam transport section (LEBT) of the Xi’an Proton Application Facility (XiPAF). Optimal currents are determined when the highest transmission of a radio frequency quadrupole (RFQ) accelerator downstream of the LEBT is achieved. The optimal work points of the tuned accelerator were obtained with currents of 0 A, 0 A, 0 A, and 0.1 A, for the four steering magnets, and 107 A and 96 A for the two solenoids. Furthermore, the highest transmission of the RFQ was 91.2%. Meanwhile, the lower time required for the optimization with the A3C algorithm was successfully verified. Optimization with the A3C algorithm consumed 42% and 78% less time than pointwise scanning with random initialization and pre-trained initialization of weights, respectively. Real-time beam tuning Parameter optimization Asynchronous advantage actor-critic algorithm Low-energy beam transport 1 Introduction 1 The low-energy beam transport (LEBT)section is commonly applied to proton and heavy ion accelerators to transport and match a beam to the entrance of a radio frequency quadrupole (RFQ) accelerator [ 1 1 , 2 2 ]. The magnetostatic focusing LEBT, which has been widely explored, has the following advantages: the space charge effect can be neutralized by the residual gas and emittance growth can be decreased [ 3 3 ]. The magnetostatic focusing LEBT usually consists of several steering magnets and solenoids. The beam envelope and trajectory can be adjusted by changing the currents of their power supplies [ 4 4 , 5 5 ]. However, the operational currents are different from the predictions of physical designs because of errors in mechanical matching and installation. Therefore, parameters should be optimized in beam tuning to compensate for the inconsistency. Methods of parameter optimization mainly comprise pointwise scanning and algorithm optimization. In pointwise scanning, after each point is scanned, a global optimal work point can be detected. However, the scanning process may take a long time if multi-dimensional parameters are involved and an even longer time in the case of re-tuning and re-scanning. Thus, algorithm optimization is proposed to reduce the cost. This optimization is achieved by evolutionary algorithms (EAs) [ 6 6 ], robust conjugate direction search (RCDS) algorithm[ 7 7 ], machine learning algorithm [ 8 8 ], etc. EAs are a set of population-based, fitness-oriented, and variation-driven algorithms that are inspired by Darwin’s natural selection [ 9 9 ]. In particular, EAs start with a group of solutions and evolve with variation and other policies until the fittest solution is obtained. Two typical EAs include the genetic algorithm (GA) and particle swarm optimization (PSO). The GA encodes parameters into binary codes that will crossover and mutate like a gene does to generate the next generation. The fittest codes in the new generation will be reserved and will evolve again. PSO starts with multiple solutions; then, the fittest solution is searched in the optimization direction of local historically best and global best solutions [ 6 6 ]. RCDS is an improved conjugate direction search algorithm that combines Powell’s method with a robust line optimizer and can realize optimization for noisy data [ 10 10 ]. In reinforcement learning (RL), a machine learning algorithm, a neuron network, is adopted as an agent that reacts to the real world, and a series of actions are chosen to obtain the highest reward [ 8 8 ]. The above algorithms have been widely applied in the parameter optimization of accelerators. Bartolini has optimized the beam dynamics in free electron lasers through the GA [ 11 11 ], and the set point of the LANSCE linac has been found by Pang through PSO [ 12 12 ]. RCDS have been adopted by Huang for the online optimization of accelerators in SLAC [ 10 10 ]. The machine learning algorithm has been applied in the low-level radio frequency tuning for FNAL by Steimel [ 13 13 ] and radio frequency design by Shin [ 14 14 ]. However, owing to time constraints and instabilities in the above algorithms, the global optimum is difficult to obtain within a short period of real-time beam tuning. To realize fast real-time tuning for accelerators, the asynchronous advantage actor-critic(A3C) algorithm is introduced and improved in this paper. The A3C algorithm is an RL algorithm that uses the asynchronous parallel actor-critic approach to speed up the training process and obtain a convergence solution [ 15 15 ]. It has been adopted in the automated control field [ 16 16 ]. To verify the improved A3C algorithm, we conducted three beam tuning experiments at the Xi’an Proton Application Facility (XiPAF) with parameter optimization by pointwise scanning, the GA, and the improved A3C algorithm. This paper is organized as follows: Section 2 presents RL and the A3C algorithm. Section 3 describes the improvements made to the A3C algorithm and its application in LEBT beam tuning. Section 4 discusses the experiments and results. Finally, the conclusions are summarized in Sect. 5. 2 Reinforcement Learning and A3C algorithm 1 2.1 Reinforcement Learning 2 RL aims to create an optimal action policy that can be used to choose actions based on the current environment and to maximize the long-term sum of rewards, which are the evaluations of the environment [ 17 17 ]. Figure 1 Figure 1 represents the algorithm structure of RL. RL acquires the current state from the environment, which can be the beam position, beam current, etc. The algorithm also needs rewards, which are calculated as feedback from the environment. Then, the action policy is updated to maximize the reward and choose an action to change the environment. 2.2 A3C algorithm 2 The actor-critic method contains two networks, as shown in Fig.2 Fig.2 . One is the action policy network (actor), which chooses an action to change the environment, and the other is the value function network (critic), which predicts a reward when the environment is changed by the action chosen by the actor network. An improvement on the AC algorithm, the A3C algorithm is an asynchronous AC algorithm with an advantage function. Asynchronous means that there are multiple ACs in the algorithm as shown in Fig. 3 Fig. 3 . Furthermore, each AC, which downloads network parameters and runs an episode without interacting with other ACs, is independent. A new episode will not start for an AC until all current episodes of the AC are completed. Then, the headquarters (a database that stores the parameters of the AC network) merges all updates from each AC to calculate the total update, which it then applies to its network parameters. Multiple independent ACs that run independent episodes will reduce the correlation of the selection for actions so that A3C can avoid being trapped in local optima. The advantage function is used as a normalized value function, which reduces the bias from the prediction after each step and speeds up the convergence process. The pseudocode of the A3C algorithm is shown below: The update functions of the weights for the actor and critic can be described as follows [ 15 15 ]: <math display="block" id="M1"><mrow><mrow><mo>{</mo><mrow><mtable columnalign="left"><mtr columnalign="left"><mtd columnalign="left"><mrow><mi>Δ</mi><mi>θ</mi><mo>=</mo></mrow></mtd><mtd columnalign="left"><mrow><mo>−</mo><mi>α</mi><msub><mi>Δ</mi><mi>θ</mi></msub><mo stretchy="false">[</mo><mtext>log</mtext><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mn>,</mn><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>⋅</mo><msub><mi>q</mi><mi>ω</mi></msub><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mn>,</mn><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mtext>(actor)</mtext></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow><mi>Δ</mi><mi>ω</mi><mo>=</mo></mrow></mtd><mtd columnalign="left"><mrow><mi>β</mi><mo stretchy="false">[</mo><mi>R</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mn>,</mn><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mo>+</mo><mi>γ</mi><msub><mi>q</mi><mi>ω</mi></msub><mo stretchy="false">(</mo><msub><mi>s</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mn>,</mn><msub><mi>a</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo><mo>−</mo><msub><mi>q</mi><mi>ω</mi></msub><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mn>,</mn><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mo stretchy="false">]</mo></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow/></mtd><mtd columnalign="left"><mrow><mo>⋅</mo><msub><mi>Δ</mi><mi>ω</mi></msub><msub><mi>q</mi><mi>ω</mi></msub><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mn>,</mn><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mtext>(critic)</mtext></mrow></mtd></mtr></mtable></mrow></mrow></mrow></math> where s is the environment state, a is the action, ω represents the parameters of the critic network, θ represents the parameters of the actor network, t is the current step, q ω ( s t , a t ) is the current step’s reward from the critic and π θ ( s t , a t ) is the action probability from the actor when the action is a t and state is s t , α and β are learning rates, γ is the discount factor, R ( s t , a t ) is the real reward of the current step calculated from the environment when the state is s and action is a , and q ω ( s t +1 , a t +1 ) is the reward of the next step predicted from the critic when the action is a t +1 and state is s t +1 . The update function for the actor increases the probability of a "good" action, which is expected to acquire a high reward and decrease the probability of a "bad" action. The degree of the update depends on the reward and the probability of the action. If a low-probability action achieves a high reward, the actor drastically increases its probability. The critic decreases the difference between the real reward from the environment and the predicted reward from the critic [ 18 18 ]. 3 Application of A3C to the real-time beam tuning 1 3.1 XiPAF LEBT 2 The LEBT of XiPAF contains two pairs of steering magnets and two solenoids [ 19 19 ], as shown in Fig. 4 Fig. 4 . Figure 5 Figure 5 is an illustration of the LEBT of XiPAF. In this experiment, the current of the steering magnets ranges from -0.4 A to 0.4 A with an increment of 0.1 A. The current of the solenoids ranges from 90 A to 110 A with an increment of 1 A. Beam tuning achieves the highest RFQ transmission by adjusting the currents of steering magnets and solenoids. 3.2 Improvements made to the A3C algorithm 2 Real-time beam tuning with the A3C algorithm faces several challenges. First, existing local optima hinder the pursuit of the global optimum. Second, although the A3C algorithm can converge more rapidly than the AC algorithm [ 18 18 ], it is too time-consuming for real-time beam tuning. Therefore, we improved the A3C algorithm in the following three ways: A) To prevent the optimization from being trapped in local optima, an exploration item was added to Eq. 1: <math display="block" id="M2"><mrow><mtable columnalign="left"><mtr columnalign="left"><mtd columnalign="left"><mrow><mi>Δ</mi><mi>θ</mi><mo>=</mo></mrow></mtd><mtd columnalign="left"><mrow><mo>−</mo><mi>α</mi><msub><mi>Δ</mi><mi>θ</mi></msub><mo stretchy="false">[</mo><mtext>log</mtext><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mn>,</mn><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>⋅</mo><msub><mi>q</mi><mi>ω</mi></msub><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mn>,</mn><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow/></mtd><mtd columnalign="left"><mrow><mo>+</mo><mi>ϵ</mi><mo>⋅</mo><mi>p</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mn>,</mn><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mtext>log</mtext><mi>p</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mn>,</mn><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow></mtd></mtr></mtable></mrow></math> where ϵ is the discount factor, and p ( s t , a t ) is the probability of the current action a t when the current state is s t . The exploration item is the entropy [ 20 20 ] of action a t when the state is s t , which increases the probability of the unchosen actions. Furthermore, random start points are set in the algorithm, thereby allowing the algorithm to explore more points in the scanning range. Figure 6 Figure 6 shows that, without the exploration term, the A3C algorithm will converge when an RFQ transmission of around 60% is achieved when the currents of the four steering magnets are 0.2 A, 0.1 A, 0.1 A, and 0 A. This proves that, without the exploration term, the A3C will be stuck in a local optimum, and with the introduction of the exploration term, this situation will be avoided. In Figure 6 Figure 6 , after the convergence, the RFQ transmission fluctuates by approximately 20% owing to two reasons: 1) the instability of the accelerator system, including the beam current fluctuation of 10% from the ion source, and reading errors from the ACCT and Faraday cup, and 2) the A3C algorithm, which may explore work points with low RFQ transmissions to avoid being trapped in local optima. B) To speed up the convergence process, a factor was introduced into the learning rate β of the update rule in Eq. 1 as Eq. 3 shows. <math display="block" id="M3"><mrow><mtable columnalign="left"><mtr columnalign="left"><mtd columnalign="left"><mrow><mi>Δ</mi><mi>ω</mi><mo>=</mo></mrow></mtd><mtd columnalign="left"><mrow><mi>β</mi><mtext>log</mtext><mo stretchy="false">(</mo><msub><mi>N</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mo>⋅</mo><mo stretchy="false">[</mo><mi>R</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mn>,</mn><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mo>+</mo><mi>γ</mi><msub><mi>q</mi><mi>ω</mi></msub><mo stretchy="false">(</mo><msub><mi>s</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mn>,</mn><msub><mi>a</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo><mo>−</mo></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow/></mtd><mtd columnalign="left"><mrow><msub><mi>q</mi><mi>ω</mi></msub><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mn>,</mn><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>⋅</mo><msub><mi>Δ</mi><mi>ω</mi></msub><msub><mi>q</mi><mi>ω</mi></msub><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mn>,</mn><msub><mi>a</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow></mtd></mtr></mtable></mrow></math> where N t is the current number of steps in one episode. After the modification, the learning rate is low at the beginning of an episode and increases as the optimization proceeds. Then, the convergence process is shorter than that specified in the original update rule, which means that updates should be performed equally according to the original update policy for each scan step. However, the first scan steps are searched in random directions, which will cause fluctuation in the convergence if updates are made equally. Therefore, with the modified learning rate, the convergence process is shorter. Figure 7 Figure 7 shows the A3C algorithm without modification of the learning rate converges after 600 episodes when the currents of the steering magnets are 0 A, 0.1 A, 0 A, and 0 A, and the RFQ transmission is 87%. After modifying the learning rate, A3C can obtain the same optimal steering currents of 0 A, 0.1 A, 0 A, and 0 A, and an RFQ transmission of 87% in 200 episodes. C) To further speed up the convergence of the process, the network structure is altered. In pointwise scanning, as scanning multi-dimensional parameters is impractical, an alternating scan method that divides the process into several parts is used. In each part, only several parameters are scanned. The scanning process can alternate between parts because the parameters in different parts are less dependent on each other. For example, in the LEBT, the steering magnets and solenoids exhibit different effects on the beam, and their fields are not coupled. Thus, the currents of the steering magnets can be scanned first to determine the optimal currents. Then, using the same method, the currents of the solenoids and currents of the steering magnets can be scanned alternatively until the highest RFQ transmission is acquired. Learning from the alternating scanning, the A3C assumes the alternated network structure shown in Fig. 8 Fig. 8 . In the A3C network, the alternating structure freezes weights (red lines in Fig. 8 Fig. 8 ) associated with solenoid current parameters during the scanning of the steering magnets and locates the optima. Then, the weights associated with the currents of the steering magnets are frozen to obtain the optimal solenoid currents. In this way, the number of parameters trained in each scanning process are reduced to expedite the convergence. Figure 9 Figure 9 shows the A3C algorithm that uses the frozen structure converges quicker than that which does not use the frozen structure. 3.3 Design of the input state and output action 2 The input state, output action, and reward are specified in order to apply the A3C algorithm to real-time beam tuning of the accelerator. The A3C consists of four network layers, one input layer, one output layer, and two hidden layers. The hidden layers contain 50 neurons. In the LEBT, the input layer consists of steering magnet currents, solenoid currents, currents at the entrance and exit of the RFQ, and the total number of steps N s in one episode. The output layer of the actor increases and decreases the currents of the power supplies. The output layer of the critic only includes one neuron for the reward. 3.4 Reward design 2 Zero RFQ transmission has been observed for around 90% of the points in the current scanning of steering magnets and solenoids. The variation in RFQ transmission after an action is not continuous. The intent of our scan is to find an optimal point when an episode is over, indicating that the final step is the most important in the scanning process. For these reasons, the reward is expressed in the following form, as in Eq. 4: <math display="block" id="M4"><mrow><mi>R</mi><mo>=</mo><mrow><mo>{</mo><mrow><mtable columnalign="left"><mtr columnalign="left"><mtd columnalign="left"><mrow><mfrac><mrow><mi>Δ</mi><mi>E</mi></mrow><mrow><mo>|</mo><mi>Δ</mi><mi>E</mi><mo>|</mo></mrow></mfrac><mo>+</mo><mo stretchy="false">(</mo><mi>E</mi><mo>−</mo><mn>0.5</mn><mo stretchy="false">)</mo><mtext> </mtext></mrow></mtd><mtd columnalign="left"><mrow><mtext>(not</mtext><mtext> </mtext><mtext>final</mtext><mtext> </mtext><mtext>step)</mtext></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow><mn>10</mn><mo stretchy="false">(</mo><mfrac><mrow><mi>Δ</mi><mi>E</mi></mrow><mrow><mo>|</mo><mi>Δ</mi><mi>E</mi><mo>|</mo></mrow></mfrac><mo>+</mo><mo stretchy="false">(</mo><mi>E</mi><mo>−</mo><mn>0.5</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mtext> </mtext></mrow></mtd><mtd columnalign="left"><mrow><mtext>(final</mtext><mtext> </mtext><mtext>step)</mtext></mrow></mtd></mtr></mtable></mrow></mrow></mrow></math> where R is the reward, and E is the RFQ transmission. The first item of Eq. 4 includes two parts. Δ E /|Δ E | represents the trend of the derivative. If the transmission increases, the reward is 1. Otherwise, the reward is -1. (E-0.5) is the penalty of low RFQ transmission point, which means that if the RFQ transmission is lower than 50%, the item is negative. The reward in the final step is 10 times larger than those in the other steps, which was determined from the experiment conducted to obtain a fast convergence in our case. 4 Experiment and results 1 4.1 Experimental method 2 In pointwise scanning, the alternating scanning process mainly includes three steps: 1) scan the currents of the four steering magnets; 2) scan the currents of the two solenoids; and 3) re-scan the currents of the four steering magnets. In GA optimizing, we use a population of 100 sets of parameters and run the algorithm for 50 generations in order to acquire the optimized steering magnets currents. In A3C-algorithm optimizing, the improved A3C algorithm is used to search the optimal currents of the steering magnets and solenoids. Each AC algorithm used in the A3C algorithm for real-time beam tuning of the XiPAF LEBT is shown in Fig.10 Fig.10 . The scanning process is presented in Fig.11 Fig.11 . 4.2 Experimental results and discussions 2 4.2.1 Pointwise scanning 3 Figure 12 Figure 12 presents the RFQ transmission after pointwise scanning of the currents of the four steering magnets is performed for the first time (currents of the first and second solenoids are fixed as 110 A and 90 A, respectively). In each cube, the 3D scanning results of the first three steering magnets are projected on the three coordinate planes. On each coordinate plane, the point values are the maximum RFQ transmissions in the corresponding projection direction. The highest RFQ transmission is 87% when the currents of the steering magnets are 0 A, 0.1 A, 0 A, and 0 A. The first and third steering magnets are positively correlated, which means increases in the currents of two steering magnets have the same effect on the RFQ transmission. The currents of the second and fourth steering magnets are negatively correlated. The scanning result of the solenoids in Fig. 13 Fig. 13 reveals that 89% is the highest RFQ transmission that can be achieved, and the solenoid currents are 107 A and 96 A. Figure 14 Figure 14 shows the results of the re-scanning of the currents of the four steering magnets. The solenoid currents are fixed as 107 A and 96 A. The highest transmission is 90.6% when the steering magnet currents are 0 A, 0 A, 0 A, and 0.1 A. The currents of the second and fourth steering magnets are still negatively correlated but have an offset that was not observed after the first scan. The first and third steering magnets are no longer positively correlated. These results confirm that the solenoid currents have an influence on the scanning result of the optimal points of steering magnets. Therefore, alternating scanning is necessary in pointwise scanning. From these results, we can see there are many local optima in the beam tuning of the LEBT. Furthermore, the relationship among four steering magnets can be altered when the current in the solenoids change. 4.2.2 Optimization with the genetic algorithm 3 Figure 15 Figure 15 shows the results when optimization was conducted with the GA six times. Each time, the GA converged at different RFQ transmissions with different sets of steering magnets currents. The result reveals that the final solutions are trapped in local optima. Early-maturing and instability are the major weaknesses of the GA [ 21 21 ], which means its exploratory ability is limited, and its results are unstable. Therefore, the GA cannot converge at the global optimum owing to the presence of too many discrete local optima as shown in the results of pointwise scanning. This means the GA is not suitable for the beam tuning of the LEBT. 4.2.3 Optimization with the A3C algorithm 3 To verify the A3C algorithm in real-time beam tuning, two sets of experiments were performed. The first experiment involved optimization with random weights as the initialization, which verified that the A3C algorithm has a higher efficiency than pointwise scanning. The second one comprised optimization with the initialization of pre-trained weights, that is, weights from the previous scanning process. The learnability was verified for the second experiment. Figure 16 Figure 16 presents the RFQ transmission with episodes with random weights as the initialization. The scanning result reveals that the real-time beam tuning code converges at 260, 150, and 240 episodes for the steering magnet scanning, solenoid scanning, and steering magnet re-scanning, respectively. The corresponding numbers of scans are approximately 3800, 2000, and 3000. A maximum of 20 actions can be performed in each episode (one action refers to one current scan and each scan takes approximately 2 s considering the RF repetition rate of 1 Hz and an additional 1 s for the stabilization of the power supplies). Meanwhile, the average number of actions is approximately 10 in an episode because the A3C algorithm can stop early when the global optimum is obtained. The optimal currents are 0 A, 0 A, 0 A, and 0.1 A for the steering magnets, and 107 A and 96 A for solenoids. The optimal currents are the same as those with pointwise scanning. Furthermore, the performance of the tuned accelerator will be the same, with the highest transmission of the RFQ as 91.2%. The RFQ transmission fluctuates because of the exploration factor and the current instability (±5%) of the ion source. Optimization with the A3C algorithm in the first experiment consumes 42% less time than pointwise scanning. In the second experiment, the current scanning of the steering magnets and solenoid, and current re-scanning of the steering magnets are performed successively. The current re-scanning of the steering magnets is conducted with weights from the previous scanning of the steering magnets. Figure 17 Figure 17 presents the RFQ transmission with episodes presented as a red line. The RFQ transmission converges at 120 episodes (corresponding to 1500 current scans) for the red line. With the pre-trained initialization of the weights, optimization with the A3C algorithm results in faster convergence than that with the random initialization, and consumes 78% less time than pointwise scanning. The optimal currents are the same as those in the first experiment. The RFQ transmission achieved in the two experiments differs slightly but actually the steering magnets currents are the same when convergence occurs. The difference is caused by the fact that the accelerator status is not very stable, it includes a 10% fluctuation of the ion source and instability in reading the ACCT. 5 Conclusion 1 Pointwise multi-dimensional parameter scanning can be time consuming in the real-time beam tuning of an accelerator. For the current optimization of four steering magnets and two solenoids, the total scanning process lasts approximately 3.6 h. To speed up the multi-dimensional optimization, the A3C algorithm is introduced and improved for a real-time beam tuning code. To prevent the A3C from being trapped in a local optimum, we add an exploration item in the update functions of the weights. To speed up the convergence process, variations in the learning rate and the network structure with the alternating-scan method are proposed. The improved A3C algorithm was verified through two experiments in which it was applied to beam tuning in the LEBT of XiPAF. The currents of the four steering magnets and two solenoids in the LEBT were optimized to achieve the highest transmission of an RFQ accelerator downstream of the LEBT. By pointwise scanning with optimal currents of 0 A, 0 A, 0 A, and 0.1 A for the steering magnets, and 107 A and 96 A for solenoids, the highest transmission of the RFQ obtained was 91.2%. The results of RFQ transmission optimized by the GA vary from 66.1% to 82.6% and the results mean that the GA cannot find the global optimum. Optimization with the A3C algorithm results in faster convergence than that with the random initialization and consumes 78% less time than pointwise scanning. The optimized currents of the steering magnets and solenoids are the same as those of pointwise scanning. The results show that the improved A3C algorithm is time effective even when the scanning is performed with random weights. The time can be further optimized if weights from the previous scan are adopted for initialization. The improved A3C algorithm can provide an efficient approach for the real-time beam tuning of accelerators. Furthermore, it is expected to be applicable in other automated control systems with similar requirements. References C.A. Valerio-Lizarraga , J.B. Lallement , I. Leon-Monzon , et al ., Space charge compensation in the Linac4 low energy beam transport line with negative hydrogen ions . REV SCI INSTRUM . 85 ( 2 ), 02A505 ( 2014 ). doi: 10.1063/1.4847196 http://doi.org/10.1063/1.4847196 Space charge compensation in the Linac4 low energy beam transport line with negative hydrogen ions N. Chauvin , R. Duperrier , D. Uriot , et al ., Final design of the IFMIF-EVEDA low energy beam transport line . Paper presented at 23rd Particle accelerator Conference , Vancouver, Canada , 4-8 May, 2009 . Final design of the IFMIF-EVEDA low energy beam transport line P.Y. Beauvais , R. Ferdinand , R. Gobin , et al ., Emittance improvement of the electron cyclotron resonance high intensity light ion source proton beam by gas injection in the low energy beam transport . REV SCI INSTRUM . 71 , 1413 ( 2000 ). doi: 10.1063/1.1150448 http://doi.org/10.1063/1.1150448 Emittance improvement of the electron cyclotron resonance high intensity light ion source proton beam by gas injection in the low energy beam transport X.L. Lu , Y. Zhang , J.R. Wang , et al ., Transport characteristics of space charge-dominated multi-species deuterium beam in electrostatic low-energy beam line . NUCL SCI TECH . 29 , 4 - 51 ( 2018 ). doi: 10.1007/s41365-018-0384-2 http://doi.org/10.1007/s41365-018-0384-2 Transport characteristics of space charge-dominated multi-species deuterium beam in electrostatic low-energy beam line Y.K. Batygin , I.N. Dragnanic C.M. Fortgang , et al ., Design of low energy beam transport for new LANSCE H+ injector . NUCL INSTRUM METH A . 753 , 1 ( 2014 ). doi: 10.1016/j.nima.2014.03.041 http://doi.org/10.1016/j.nima.2014.03.041 Design of low energy beam transport for new LANSCE H+ injector X.J. Yu , M. Gen , in Introduction to evolutionary algorithms ( Springer , London Dordrecht Heidelberg New York , 2010 ) doi: 10.1007/978-1-84996-129-5 http://doi.org/10.1007/978-1-84996-129-5 Introduction to evolutionary algorithms H.F. Ji , Y. Jiao , M.Y. Huang , et al ., Optimization of the collimation system for CSNS/RCS with the robust conjugate direction search algorithm . CHINESE PHYS C . 40 , 097006 ( 2016 ). doi: 10.1088/1674-1137/40/9/097006 http://doi.org/10.1088/1674-1137/40/9/097006 Optimization of the collimation system for CSNS/RCS with the robust conjugate direction search algorithm I.H. Witten , E. Frank , M.A Hall , et al ., in Data Mining: Practical machine learning tools and techniques , vol. 1 , ed. by C. Kent ( Elsevier , Cambridge , 2017 ), p. 417 Data Mining: Practical machine learning tools and techniques C. Darwin , A.R. Wallace , Evolution by natural selection . Evolution by natural selection ( 1958 ). Evolution by natural selection X.B. Huang , J. Corbett , J. Sanfranek , et al ., An Algorithm for Online Optimization of Accelerators , NUCL INSTRUM METH A . 726 77 - 83 , ( 2013 ). doi: 10.1016/j.nima.2013.05.046 http://doi.org/10.1016/j.nima.2013.05.046 An Algorithm for Online Optimization of Accelerators R. Bartolini , M. Apollonio , I.P.S. Martin ., Multiobjective genetic algorithm optimization of the beam dynamics in linac drivers for free electron lasers . PHYS REV ACCEL BEAMS . 15 , 030701 ( 2012 ). doi: 10.1103/PhysRevSTAB.15.030701 http://doi.org/10.1103/PhysRevSTAB.15.030701 Multiobjective genetic algorithm optimization of the beam dynamics in linac drivers for free electron lasers X. Pang , L.J. Rybarcyk ., Multi-objective particle swarm and genetic algorithm for the optimization of the LANSCE linac operation . NUCL INSTRUM METH A . 741 , 124 ( 2014 ). doi: 10.1016/j.nima.2013.12.042 http://doi.org/10.1016/j.nima.2013.12.042 Multi-objective particle swarm and genetic algorithm for the optimization of the LANSCE linac operation J. Steimel , C. Baffes , P. Berrutti , et al ., Beam Commissioning Status and Results of the FNAL PIP2IT Linear Accelerator RFQ . Paper presented at 28th Linear Accelerator Conference , East Lansing, Milwaukee , 25-30 September 2016 . Beam Commissioning Status and Results of the FNAL PIP2IT Linear Accelerator RFQ S.W. Shin , M. Ghergherehchi , J.S. Chai , et al ., Using Deep Reinforcement Learning for Designing Sub-Relativistic Electron Linac . Paper presented at 9th International Particle Accelerator Conference , Vancouver, Canada , 29 April- 4. May, 2018 doi: 10.18429/JACoW-IPAC2018-THPML032 http://doi.org/10.18429/JACoW-IPAC2018-THPML032 Using Deep Reinforcement Learning for Designing Sub-Relativistic Electron Linac V. Mnih , A.P. Badia , M. Mirza , et al ., Asynchronous methods for deep reinforcement learning . Paper presented at 33rd International Conference on Machine Learning , New York, New York , 19-24 June, 2016 Asynchronous methods for deep reinforcement learning PA. van der Linden , Dissertation , University of Amsterdam , 2017 Dissertation B.J. Pandian , S.T. Kumar , M.M. Noel , et al ., Q-learning with policy function approximation for a benchmark ball and beam control problem . INT J INNOV COMPUT I . 13 , 5 - 1467 ( 2017 ). Q-learning with policy function approximation for a benchmark ball and beam control problem J. Peters , S. Schaal , Natural actor-critic . NEUROCOMPUTING . 71 , 7 - 1180 ( 2008 ). doi: 10.1016/j.neucom.2007.11.026 http://doi.org/10.1016/j.neucom.2007.11.026 Natural actor-critic T. Ruo , L. Du , T. Du , et al ., H- Beam Dynamics Study of a LEBT in XiPAF Project with the WARP PIC Code . ICFA Advanced Beam Dynamics Workshop on High-Intensity and High-Brightness Hadron Beams . 2 , 4 - 45 ( 2016 ). doi: 10.18429/JACoW-HB2016-WEPM5Y01 http://doi.org/10.18429/JACoW-HB2016-WEPM5Y01 H- Beam Dynamics Study of a LEBT in XiPAF Project with the WARP PIC Code J. Lin , Divergence measures based on the Shannon entropy , IEEE Transactions on Information Theory 37 , 145 - 151 ( 1991 ). doi: 10.1109/18.61115 http://doi.org/10.1109/18.61115 Divergence measures based on the Shannon entropy J. Li , J. Zhao , Combining Differential Evolution Algorithm with biogeography-based optimization algorithm for reconfiguration of distribution network . Paper presented at 2012 IEEE International Conference on Power System Technology , Auckland, New Zealand , 30. Oct- 2. Nov, 2012 doi: 10.1109/PowerCon.2012.6401351 http://doi.org/10.1109/PowerCon.2012.6401351 Combining Differential Evolution Algorithm with biogeography-based optimization algorithm for reconfiguration of distribution network 10.1007/s41365-019-0668-1 2019-03-23 2019-04-27 2019-05-26 2019-10-03 2019-10-01 2021-04-11 © China Science Publishing & Media Ltd. (Science Press), Shanghai Institute of Applied Physics, the Chinese Academy of Sciences, ChineseNuclear Society and Springer Nature Singapore Pte Ltd. 2019 This is a post-peer-review, pre-copyedit version of an article published in Nuclear Science and Techniques. The final authenticated version is available online at: http://dx.doi.org/10.1007/s41365-019-0668-1 http://dx.doi.org/10.1007/s41365-019-0668-1 . 2019 Nuclear Science and Techniques 10 30