ADC evaluation boards design and test framework for LCLS-II precision receiver

ADC evaluation boards design and test framework for LCLS-II precision receiver 1 2 first-author 1 corresp Corresponding author, ghuang@lbl.gov Corresponding author, ghuang@lbl.gov ghuang@lbl.gov 1 1 1 Lawrence Berkeley National Laboratory, Berkeley, CA 94720, USA Lawrence Berkeley National Laboratory, Berkeley, CA 94720, USA Department of Engineering Physics, Tsinghua University, Beijing 100084, China Department of Engineering Physics, Tsinghua University, Beijing 100084, China In the Low Level RF control field, ADC acquisition accuracy and noise sets the boundary of our control ability, making it important to develop low noise acquisition systems. From the design to test stage, all the noise terms should be understood and characterized . The specific need addressed here is the precision acquisition system for the second Linac Coherent Light Source (LCLS-II), led by SLAC National Accelerator Laboratory. Test circuit boards for the LTC2174 and AD9268 ADCs are designed and fabricated by LBNL. An ADC test framework based on FPGA evaluation board to assess performance has been developed. The ADC test framework includes both DSP (Digital Signal Processing) firmware and processing software. It is useful for Low Level RF control and other synchronization projects. Investigating the clock jitter between two channels give us an understanding of that noise source. Working with the test framework, the raw ADC data is transferred to a computer through a Gigabit Ethernet interface. Then short term error signal can be calculated based on a sine wave fit. By changing low pass filter bandwidth, relative long term performance can also be obtained. Amplitude jitter and differential phase jitter are the key issues for ADCs. This paper will report the test results for LTC2174 and AD9268 chips. The integral amplitude jitter is smaller than 0.003%, and the integral phase noise is smaller than 0.0015 (measured at 47 MHz RF, 100 MHz clock, bandwidth 1 Hz to 100 kHz) for both ADC chips. Precision acquisition LLRF LTC2174 AD9268 ADC Framework 1 Introduction 1 LCLS was the world’s first hard X-ray free electron laser [ 1 1 ]. The follow-on project known as LCLS-II plans to provide up to 1 million pulses per second, rather than the 120 pulses per second from the original LCLS project. The project upgrades the whole accelerator system with a 4 GeV CW superconducting linac. 35 TESLA-ILC 1.3 GHz style cryomodules will be installed. The relatively high loaded Q (4d7) leads to 16 Hz half-bandwidth [ 2 2 ]. This narrow bandwidth combined with microphonic detuning poses a difficult control task for cavity resonance control and field control. But at the same time, the integral noise in the system bandwidth will be very small. The field detection will be relatively easier. The Low Level RF control system is tasked with holding the cavity field stable to 0.01% and 0.01 rms. From the experience in Lawrence Berkeley National Laboratory (LBNL), detecting and controlling the field as specified is possible, but gets close to the hardware limitations [ 2 2 - 6 6 2-6 ]. LBNL used the LLRF4 board for many synchronization projects. Its control ability is limited mostly by the board’s ADCs. Based on the experience from the LLRF4 board, ADC evaluation boards are designed for LCLS-II precision receiver chassis. One board is for LTC2174 [ 7 7 ], the other is for AD9268 [ 8 8 ]. The purpose of using LTC2174 is try to use the high speed serial data acquisition. AD9268 has the relative high Signal-to-Noise Ration (SNR) 78.2 dBFS ( Fig. 1 Fig. 1 ). 2 Hardware design 1 Two separate ADC test boards are designed by LBNL. One board is for LTC2174, the other is for AD9268. The two boards are shown in Fig 2 Fig 2 . One board is using the 14-bit/4-channel/105 Msps LTC2174 chip. This chip claims 73.1 dB SNR and 88 dBc Spurious-Free Dynamic Range (SFDR). it is configured in 2-lane output mode. 14 bit data for each channel can be sent by the chip through 2 Low-Voltage Differential Signaling (LVDS) pairs. This high speed serialization feature reduces pin count, which is an important factor when designing a ADC mezzanine card and selecting an FPGA package. It takes 10 pairs (8 pairs for data output, 1 pair for FRAME, 1 pair for DCO) of LVDS pins for 4 channels of data acquisition, and 4 LVCMOS (Low Voltage Complementary Metal Oxide Semiconductor) pin for Serial Peripheral Interface (SPI) control. The ADC has good performance and saves data pins. But it takes effort to deal with the high speed serialized data output interface. We succeeded in using the Spartan-6 ISERDES [ 12 12 ] to receive the high speed serialized data. The AD9268 chip has higher SNR of 78.2 dB and almost the same SFDR 88 dBc. Using the AD9268 is for the higher precision measurement. It take 18 pairs of LVDS pins for acquiring 2 channels, and 3 LVCMOS pins for SPI control. On the driver side, the task is to take care of the SPI control part. The Double Data Rate (DDR) data acquisition logic is comparatively easier. The performances of the 2 ADCs are summarized in Table 1 Table 1 . 1 1 PART NO. 1 1 AD9268 1 1 LTC2174 1 1 Num of bits 1 1 16 1 1 14 1 1 Sampling rate (Msps) 1 1 105 1 1 105 1 1 ADC SNR@30MHz (dB) 1 1 78.2 1 1 73.0 1 1 p-p full scale (V) 1 1 2.00 1 1 2.00 1 1 FPGA pins per channel 1 1 17 1 1 5 1 1 Num of channels 1 1 2 1 1 4 In this design, the LTC6401 [ 9 9 ] ADC driver chip is used in the signal conditioning path. Its common-mode voltage output range is from 1.0 V to 1.5 V. But the common mode voltage output from the ADC should be around 0.9 V to 1.0 V. This is a small defect, but the system still works. The next time to use an ADC driver chip, the common-mode range should be carefully chosen. An ultra low noise 1.8 V power supply is made up of an LTC6655 precision voltage reference chip, an op-amp, and a transistor. The noise floor above 100 Hz is 5nV/ <math id="M1"><mrow><msqrt><mrow><mtext>Hz</mtext></mrow></msqrt></mrow></math> , much cleaner than commercial LT1763-1.8 (40nV/ <math id="M2"><mrow><msqrt><mrow><mtext>Hz</mtext></mrow></msqrt></mrow></math> ) [ 10 10 ]. In the 0.1 Hz to 10 Hz range, with careful design, 1/ f noise is measured as expected. There are two jumper-selectable 1.8 V analog power supplies on the AD9268 board, allowing us to characterize ADC performance with either the custom ultra-low noise supply or a simple LT1763-1.8. Not much difference is observed, although experiments are continuing. 3 Firmware and Software 1 The test system is based on the SP601 Spartan-6 Evaluation board [ 11 11 ]. Utilizing the general purpose Ethernet/IP/UDP (User Datagram Protocol) engine developed by LBNL, the boards can communicate at high speed to a commodity workstation. Figure 5 Figure 5 shows the simplified ADC test setup and the framework of the FPGA firmware. On the firmware side, it takes lots of effort to deal with the ISERDES clocking resources (especially bank restrictions) and bitslip. This is the most difficult part of our driver development. Figure 3 Figure 3 shows LTC2174 driver diagram. Figure 4 Figure 4 shows AD9268 driver diagram. The remaining DSP part is same for the two chips, and is based on the logic used in Low Level RF control (LLRF). The DSP part is composed of mixer, DDS (Direct Digital Synthesizer) LO (Local Oscillator) generator and CORDIC [ 13 13 ] module. After implementing DSP module, the amplitude/phase data can be sent to a CIC low pass filter. The averaging length of the low pass filter can be configured from 1 to 2 29 , so that the noise spectra can be characterized down to 0.01 Hz. Using a set of external triggerable circular buffers, the selected waveforms will be recorded simultaneously. Then the data will be transmitted over Ethernet as UDP packets read by PC side software. On the software side, python script is used for board configuration and data acquisition. For each test board, a class is created which is implemented with board operation and raw data conversion. The base class ’adc_board’ is developed for general purpose ADC data processing. The important functions in the class are listed in Table 2 Table 2 . 1 1 Function 1 1 Description 1 1 read_buf_data 1 1 read buffers data and raw data decoding; 1 1   1 1 utilize the buf_read() from the derived class 1 1 buf_skip 1 1 skip the buffer data 1 1 buf_wait 1 1 for different low pass filter averaging length, 1 1   1 1 wait for different time for buffer filling time 1 1 get_adc_data 1 1 plot or save raw adc data to file, 1 1   1 1 length=1024 1 1 get_amp_info 1 1 plot or save amplitude information, 1 1   1 1 length=1024 1 1 get_phase_info 1 1 plot or save phase information, length=1024 1 1 plot_buf_data 1 1 plot real time waveforms: 1 1   1 1 (1) ADC raw data; FFT spectrum; sine fit; 1 1   1 1 (2) amplitude information for two channels; 1 1   1 1 (3) phase information for two channels; 1 1 sine_fit 1 1 sine wave fit function 1 1 histogram 1 1 shorted input histogram and sine wave fit 1 1   1 1 error histogram 1 1 plot_hist 1 1 plot the histogram 1 1 fft_db 1 1 data spectrum analyze 1 1 plot_fft_avg 1 1 plot fft spectrum with averaging ability 1 1 phase_noise_density 1 1 plot phase noise density same as signal 1 1   1 1 source analyzer does; 1 1 amp_noise_density 1 1 plot amplitude noise density The general purpose ’adc_board’ class is mostly for data processing. The spectrum can tell if there is any abnormal noise. The phase noise and amplitude noise density can tell the phase and amplitude detection ability of the ADC board. To test the LTC2174 and AD9268, two classes are derived from the ’adc_board’ class. Most of the functions in the derived class are for board communication, board setting, and buffers operation as listed in Table 3 Table 3 . 1 1 Function 1 1 Description 1 1 reset 1 1 reset ADC board to default setting 1 1 registers 1 1 set registers in FPGA 1 1 buf_mode 1 1 select which data is sent to buffers 1 1 buf_trig 1 1 trigger buffers to record input waveforms 1 1 buf_read 1 1 read buffer data back 1 1 set_lpf_length 1 1 set averaging length for low pass filter 1 1 set_trig_internal 1 1 internal/external trigger selection The general setting and buffer operation details are in the derived classes. For a new ADC board, general setting and buffer operations need to be implemented in the derived class. The DSP module in FPGA firmware and the ’adc_board’ class can still be used without modification. The ’DSP’ module in FPGA firmware, and ’adc_class’ in python code are the cores of the ADC test framework. 4 DSP Algorithm 1 The digital signal processing chain is based on Low Level RF signal processing using non-IQ sampling [ 15 15 ]. The input signal is mixed with local oscillator signal, then goes through a cascaded-integrator-comb (CIC) filter to get the I/Q component. Then a CORDIC module converts I/Q to amplitude/phase. Then processing data gets sent to buffers directly or through a low pass filter as shown in Fig. 5 Fig. 5 . The ’local oscillator generator’ or DDS is composed of a phase accumulator and a CORDIC module. The structure of this DDS is shown in Fig. 6 Fig. 6 . The CORDIC module is supplied with accumulated phase and a fixed amplitude, and the subsequent sin and cos signals are generated by the CORDIC module. This kind of DDS is flexible. The LO frequency can be changed by simply adjusting the ’phase_step’ register. The mixer is just a multiplier. In this case the ADC signal inputs to the multiplier is 16 bits signed (for a 14 bit ADC, two zeros are appended after the Least Significant Bit, LSB). The LO signal to the multiplier is 18 bits signed. <math display="block" id="M3"><mrow><mfrac><mrow><msup><mrow><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><msup><mi>z</mi><mrow><mo>−</mo><mi>R</mi></mrow></msup><mo stretchy="false">)</mo></mrow><mi>N</mi></msup></mrow><mrow><msup><mrow><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><msup><mi>z</mi><mrow><mo>−</mo><mn>1</mn></mrow></msup><mo stretchy="false">)</mo></mrow><mi>N</mi></msup></mrow></mfrac><mo>=</mo><msup><mrow><mrow><mo>(</mo><mrow><mstyle displaystyle="true"><munderover><mo>∑</mo><mrow><mi>k</mi><mo>=</mo><mn>0</mn></mrow><mrow><mi>R</mi><mo>−</mo><mn>1</mn></mrow></munderover></mstyle><msup><mi>z</mi><mrow><mo>−</mo><mi>k</mi></mrow></msup></mrow><mo>)</mo></mrow></mrow><mi>N</mi></msup></mrow></math> In Eq. 1, R is sampling rate change factor and N is the number of cascaded integrator-comb stages. In the test, N =2 is set for I/Q CIC filter. N =1 is set for the amplitude and phase CIC filter. R for both CIC filters can be configured through the python script. In LLRF and synchronization systems, the dominant performance metrics are the amplitude and phase noise spectra. By changing the sampling rate factor R for the amplitude/phase CIC filter, and sending the filtered data to buffers, the noise in different frequency bands can be investigated. For amplitude noise, it can be derived from the Discrete Fourier Transformation of amplitude data directly [ 16 16 ]. For phase noise, the single side band phase jitter density spectra ϕ jitter_density also can be derived from the Discrete Fourier Transformation of phase data directly. Based on Eq. 2, the general phase noise density function L ( f ) can be calculated [ 17 17 ] in ’dBc/Hz’. In Eq. 2, the unit of ϕ jitter_density ( f ) is ’rad/ <math id="M4"><mrow><msqrt><mrow><mtext>Hz</mtext><mo>’</mo></mrow></msqrt></mrow></math> . The integral phase jitter ϕ jitter ( f ) can be derived from Eq. 2. The performance of these two ADC chips is shown in the next section. <math display="block" id="M5"><mrow><mtable columnalign="left"><mtr columnalign="left"><mtd columnalign="left"><mrow><mi>L</mi><mo stretchy="false">(</mo><mi>f</mi><mo stretchy="false">)</mo></mrow></mtd><mtd columnalign="left"><mrow><mo>=</mo><mn>20</mn><mi>log</mi><mo stretchy="false">(</mo><msub><mi>ϕ</mi><mrow><mtext>jitter_density</mtext></mrow></msub><mo stretchy="false">)</mo></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow><msub><mi>ϕ</mi><mrow><mtext>jitter</mtext></mrow></msub><mo stretchy="false">(</mo><mi>f</mi><mo stretchy="false">)</mo></mrow></mtd><mtd columnalign="left"><mrow><mo>=</mo><msqrt><mrow><mn>2</mn><mo>×</mo><mstyle displaystyle="true"><mrow><mo>∫</mo></mrow></mstyle><msup><mrow><mn>10</mn></mrow><mrow><mi>L</mi><mo stretchy="false">(</mo><mi>f</mi><mo stretchy="false">)</mo><mo>/</mo><mn>10</mn></mrow></msup><mo>⋅</mo><mtext>d</mtext><mi>f</mi></mrow></msqrt></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow/></mtd><mtd columnalign="left"><mrow/></mtd></mtr></mtable></mrow></math> 5 Test result 1 The test system setup is shown in Fig. 7 Fig. 7 . An Agilent 33600A Series Waveform Generator is used for the 100 MHz sampling clock generation. A Berkeley Nucleonics Model 845 RF signal Generator generated the 47 MHz RF signal fed to the ADCs. First, the ADC raw data and sine fit result are shown here. Figure 8 Figure 8 shows the raw ADC data and sine curve fit for LTC2174. Figure 9 Figure 9 is for AD9268 sine fit. The 14 bits of ADC data for LTC2174 is padded to 16 bits by adding two zeros at the end. The maximum ADC count is about 25000, or -2.3 dBFS. This is representative of real LLRF system use, where some control and operability margin needs to be included in the RF chain setup. Figure 10 Figure 10 shows the histogram of fitting errors @16 bits for LTC2174. The RMS fitting error is 3.4 LSB@16 bits or 1.4 LSB@14 bits for LTC2174 channel 1. Figure 11 Figure 11 shows the fitting error histogram for AD9268. The RMS fitting error is 3.2 LSB rms error for AD9269 channel 1. Figure 12 Figure 12 shows the LTC2174 channel 1 measured amplitude information. Figure 13 Figure 13 shows the AD9268 channel 1 measured amplitude information. The amplitude information are from ’amp1’ shown in Fig. 5 Fig. 5 directly. The only filter applied on the amplitude information is the I/Q CIC filter ( R =100, N =2). In this short term measurement (1 ms), the RMS amplitude jitter is smaller than 0.01%. Much more details are shown in the integral amplitude noise figures. Fig. 14 Fig. 14 shows single side band amplitude noise density for the two ADCs. The other side band noise power need to be taken into account for the integral noise calculation. It means that the noise power need to be doubled for noise integration. The noise measured by the ADCs is composed of RF signal generator noise, the drive amplifier noise, and the noise from the ADC itself. The effect of 2 stages CIC filter can be seen from 100 kHz to 500 kHz in this figure. Without CIC filter the noise floor should be flat. Computations show that the CIC filter will affect the test result by not more than 5% below 200 kHz. The result below 200 kHz should be trustworthy. Figure 15 Figure 15 shows the integral amplitude noise from 1 Hz to 500 kHz. It can be implied from this figure that how much noise will this kind of amplitude detector adds. And the control ability can be roughly predicted. For the LCLS-II Superconducting cavity, the cavity bandwidth is smaller than 100 Hz, so the amplitude detection RMS additive noise generated by ADC from 1 Hz to 100 Hz should be lower than 0.001%. The integral amplitude noise figure implies that the performance of LTC2174 and AD9268 are similar in the low frequency range (1 Hz to 1 kHz). This may result from the RF signal source itself. In the relative high frequency range (1 kHz to 500 kHz), performance of amplitude detection for AD9268 is better than for LTC2174. The conclusion is that even in 0.01% amplitude control system, a 14 bit ADC is still suitable. After showing the amplitude detection ability, here shows the phase detection ability. Figure 16 Figure 16 shows the differential phase noise density between two channels. Figure 17 Figure 17 shows the two channels differential phase of AD9268. Compared with real LLRF control board, in this experiment, the result is free from the additive noise from clock distribution chip, for example an AD9510. The result in this paper will show the limitation of the phase detection ability. This test framework is still applicable for the performance test between two chips. As is in amplitude result part, showing the noise density and integral noise will give the detail information for phase detection ability. Figure 18 Figure 18 shows the phase noise spectrum of the differential phase. Figure 19 Figure 19 shows the integral phase noise. The phase detection performance for AD9268 is better than LTC2174. From 1 Hz to 100 Hz, the integral phase jitter is smaller than 0.0003 which is much better than the requirement. The integral phase jitter from 1 Hz to 100 kHz is smaller than 0.0015. This result tells us, if the interested bandwidth is smaller than 100 kHz, it is possible to get sub 10 femtosecond control ability or even better. This feature is very important for phase reference distribution system. The Phase averaging reference line based on digital LLRF structure will be suitable for sub 10 femtosecond synchronization. The performance may be limited by the ADC drive chip. Designing a board without an active device may tell us the real detection ability. 6 Conclusion 1 The ADC test framework can measure the crucially important amplitude and phase spectrum of an ADC board. Using the framework as implemented here will give a fast evaluation of the ADC performance. These LTC2174/AD9268 boards show excellent amplitude and phase detection ability. These two ADCs both satisfy the core LCLS-II LLRF control requirements. The RMS amplitude noise is smaller than 0.003% (integral from 1 Hz to 100 kHz). And the 0.0015 phase noise (integral from 1 Hz to 100 kHz, 47 MHz RF, 100 MHz clock) really seems adequate for phase ultra low noise distribution, LLRF control and laser oscillator synchronization. References P. Emma , R. Akre , J. Arthur , et al . First lasing and operation of an angstrom-wavelength free-electron laser . Nature Photonics , 2010 , 4 : 641 - 647 . doi: 10.1038/nphoton.2010.176 http://doi.org/10.1038/nphoton.2010.176 First lasing and operation of an angstrom-wavelength free-electron laser L. Doolittle , G. Huang , A. Ratti , C. Serrano , Ramakrishna Bachimanchi , J. Curt Hovater , S. Babel , et al . The LCLS-II LLRF system . IPAC , Richmond, VA, USA , 2015 The LCLS-II LLRF system R. Wilcox , J. M. Byrd , L. Doolittle , et al . Stable transmission of radio frequency signals on fiber links using interferometric delay sensing . Optics Letters , 2009 , 34 : 3050 - 3052 . doi: 10.1364/OL.34.003050 http://doi.org/10.1364/OL.34.003050 Stable transmission of radio frequency signals on fiber links using interferometric delay sensing L. Doolittle , LLRF4 Board , http://recycle.lbl.gov/llrf4 http://recycle.lbl.gov/llrf4 LLRF4 Board G. Huang , R. B. Wilcox , L. R. Doolittle , and J. M. Byrd . Synchronize Lasers to LCLS e- Beam . Proc. of 2011 Particle Accelerator Conf , 2011 : 636 - 638 . Synchronize Lasers to LCLS e- Beam G. Huang , L. R. Doolittle , J. W. Staples , R. Wilcox , and J. M. Byrd . Signal processing for high precision phase measurements . BIW10 , Santa Fe, NM , TUPSMO82 ( 2010 ): 375 . Signal processing for high precision phase measurements Linear Technology . LTC2174-14 105MHz datasheet . Analog Devices . AD9268 105MHz datasheet . Linear Technology . LTC6401-14 datasheet . Linear Technology. LT1763 Series datasheet . Xilinx . Spartan-6 FPGA SP601 Evaluation Kit . Spartan-6 FPGA SP601 Evaluation Kit Xilinx . Spartan-6 FPGA SelectIO Resource User Guide . Spartan-6 FPGA SelectIO Resource User Guide CORDIC . https://en.wikipedia.org/wiki/CORDIC https://en.wikipedia.org/wiki/CORDIC CORDIC M. P. Donadio , CIC filter introduction. Iowegian , https://www.users.snip.net/donadio/cic.pdf https://www.users.snip.net/donadio/cic.pdf , 2000 . CIC filter introduction. Iowegian L. Doolittle , H.J. Ma , M.S. Champion . Digital low-level RF control using non-IQ sampling . Proceedings of LINAC2006 , Knoxville, Tennessee USA , 2006 . Digital low-level RF control using non-IQ sampling https://en.wikipedia.org/wiki/Discrete_Fourier_transform https://en.wikipedia.org/wiki/Discrete_Fourier_transform W. Kester , Converting oscillator phase noise to time jitter, Tutorial MT-008 , Analog Devices , 2009 . Converting oscillator phase noise to time jitter, Tutorial MT-008 10.1007/s41365-016-0120-8 This work was supported by the U.S. Department of Energy (DE-AC02-05CH11231) and the China Scholarship Council 2016-04-17 2016-05-20 2016-06-07 2016-08-30 2016-10-20 2021-04-11 © China Science Publishing & Media Ltd. (Science Press), Shanghai Institute of Applied Physics, the Chinese Academy of Sciences, ChineseNuclear Society and Springer Nature Singapore Pte Ltd. 2016 This is a post-peer-review, pre-copyedit version of an article published in Nuclear Science and Techniques. The final authenticated version is available online at: http://dx.doi.org/10.1007/s41365-016-0120-8 http://dx.doi.org/10.1007/s41365-016-0120-8 . 2016 Nuclear Science and Techniques 05 27