Unity and VR

In HEP experiments, physicists typically develop detector descriptions and event-visualization tools within offline software frameworks. These event display tools are usually built upon the widely used HEP software, such as Geant4 [29] or ROOT [30], which provide user-friendly visualization capabilities that facilitate software development. With the upgrades to ROOT and its EVE package [31], the development of event-display tools has become more efficient. Several recent HEP experiments, including ALICE, CMS [11], BESIII [32], JUNO [33, 34], and Mu2e [35], adopt ROOT EVE for developing event display software. However, owing to the limited visualization technique support in ROOT, its display capabilities do not fully meet the diverse requirements of physicists, and most ROOT applications remain confined to the Linux platform.

To enhance visualization quality, interactivity, and multi-platform support, several event display tools have been developed based on external visualization software. Unity is widely applied in the field of HEP and is used in projects including BelleII, BESIII, ALICE, ATLAS, and JUNO. Unity is a professional video and game development engine based on C#, and visualization software built on Unity offers several advantages.

Additionally, Unity provides a fast turnaround during the development cycle. Projects can be executed immediately, running quickly on VR devices for easier debugging without the need to compile and link executable files [41].

Compared to 3D-based event visualization software, VR technology significantly enhances the user’s visual experience. VR applications are typically conducted using HMDs. According to Steam VR hardware statistics [42], more than half of the users utilize Meta Quest 2 and Quest 3. These devices, based on the Android operating system, offer sufficient immersion and are widely used in various fields, including gaming, social interaction, and education. Equipped with accelerometers, gyroscopes, and cameras, these devices can track the user’s head and hand movements, enabling interaction and navigation within virtual environments. Additionally, the controllers facilitate interaction with the Spatial UI in the virtual environment. VR technology provides synthesized sensory feedback, creating a strong sense of immersion and presence in a simulated environment.

Most HEP experiments are typically conducted in underground or restricted areas that are inaccessible during data collection. VR technology enables the public to explore these experiments in an immersive environment and observe detector operations and event data collection. This offers a fundamental understanding of the types of scientific research conducted in HEP, which is highly beneficial for both educational and outreach purposes.

Furthermore, by simulating particle emissions and their interactions with detectors, VR provides physicists with an immersive platform for refining offline simulations and reconstruction software [43-46]. It can also enhance the simulation accuracy. For JUNO, considering the deformation of the stainless steel truss, offsets need to be applied to the PMT positions based on limited survey data [47-49]. Overlap checks and position tuning using the VR event display tool are particularly helpful. Additionally, VR enables physicists to analyze rare events as though they are physically present within the inner detector environment, providing an alternative approach for data analysis and inspiring creativity.

VR application in HEP

In recent years, VR applications have been developed for several HEP experiments, event visualization, and outreach. These software include Belle2VR [5] for the BelleII experiment, ATLASrift [7, 8] for the ATLAS experiment, and Super-KAVE [12, 13] for the Super-K experiment.

Belle2VR is an interactive VR visualization tool developed using Unity, which is designed to represent subatomic particle physics. This application allows users to explore the BelleII detector and observe particle jets generated in high energy e⁺e^- collisions. The Super-KAVE application immerses the user in a scaled representation of the Super-K detector, allowing them to explore the virtual space, switch between event datasets, and change the visualization modes [12, 13]. In addition to providing VR modes for exploring the detector and standard event displays, the application features a supernova event visualization technique that simulates the conversion of a star into a supernova. This leads to thousands of neutrino events within approximately ten seconds. It serves as a valuable outreach tool, offering a new example of visualization techniques for various applications in neutrino particle physics. ATLASrift, a VR application developed for the ATLAS experiment, is primarily used for data visualization and outreach [9]. Users move around and inside the detector, as well as explore the entire underground experimental cavern and its associated facilities, including shafts, service halls, passageways, and scaffolds.

Software structure and data flow

The event display software should provide visualization capabilities, including detector geometry, event data information at different levels, and interactive controls. For JUNO VR software visualization, the first step involves converting and importing the detector geometry and event data information into Unity for display, followed by the development of the interactive controls. As shown in Fig. 1, the JUNO event-display software consists of four components.

Fig. 1Full-screen View

The software framework and data flow in JUNO VR

Detector geometry conversion

The detector geometry in HEP experiments is typically complex, consisting of up to millions of detector units. The description of these detectors is commonly developed using specialized geometric languages, such as GDML and Detector Description for High-Energy Physics (DD4hep) [56, 57]. The JUNO experiment, along with BESIII, PHENIX [58], and LHCb [59], uses GDML to describe and optimize the geometry of detectors for conceptual design and offline software development. GDML is a detector description language based on Extensible Markup Language (XML) [60] that describes detector information through a set of textual tags and attributes, providing a persistent description of the detector. The geometry description files of detectors typically include essential information about the detector model, such as lists of materials, positions, rotations, solids, and hierarchical structures of the detector.

Because the GDML format does not directly support import into Unity, some previous HEP applications involving Unity typically required the manual construction of geometric models. Given that HEP detectors are usually highly complex, creating 3D detector models in Unity is particularly challenging. However, Unity supports the direct import of several 3D file formats, including FBX, DAE [61], DXF [62], and OBJ [63]. Among these, FBX stands out as a widely used 3D asset format because of its ability to handle intricate scene structures. This includes not only geometry but also animations, materials, textures, lighting, and cameras, making it a highly suitable choice for HEP applications involving complex 3D models.

A method that can automatically convert GDML or DD4hep to the FBX format is essential for detector construction in Unity. Several studies have proposed automated methods for converting GDML files into FBX files, significantly facilitating Unity-based development. For instance, the BESIII collaboration group suggested using FreeCAD [64], a 3D CAD and modeling software, in conjunction with the CAD data optimization software Pixyz [65], with the STEP [66] format as an intermediate conversion format [17]. The CMS collaboration group employs SketchUp software for auxiliary data conversion [67].

Recently, methods have been proposed to directly convert GDML files to FBX files [53]. This research, based on the above method, enables a fast and automatic conversion process from GDML to FBX, which can be completed in just a few minutes and saves significant time in the conversion process. This approach is particularly beneficial during the recent geometric updates of the JUNO detector at the commissioning stage, enabling the swift conversion of the updated FBX file, which includes the latest geometry model of the real detector units after installation.

Event data conversion

In HEP experiments, event data are typically stored in files with binary raw data format or ROOT format. ROOT, an efficient data analysis framework, is widely adopted for high-performance data input and output operations. However, because Unity cannot directly read ROOT files, it is necessary to extract the required event information based on the EDM and convert it into a text format that Unity can process.

The essential information for event display comprises three main components: detector unit hits, Monte Carlo (MC) truth, and reconstruction data. The detector unit hits include the hit time and hit charge for each detector unit, such as a PMT. MC truth provides detailed truth information, such as simulated vertices and photon trajectories (including 3D coordinates and propagation with time), which facilitates a deeper analysis of the particle direction and relative velocity. Reconstruction data typically contain the reconstructed vertex positions, energy information, and additional track information for muon events like direction. Together, this information serves as the foundation for developing event display functionalities and interactive control modules based on the Spatial UI.

Furthermore, the identifiers used for the detector units in offline software may differ from the names of the geometric objects in Unity. In HEP experiments, the detector identifier system assigns a unique ID to each detector unit and plays a critical role in various applications, including data acquisition, simulation, reconstruction, and analysis. Therefore, establishing an accurate mapping between the detector identifiers in offline software and geometric objects, such as PMT in Unity, is essential to ensure the accurate display of an event. Based on the EDM readout rules and leveraging the mapping between the identifier module and geometric objects in Unity, an automated readout and conversion interface was developed to export event display information.

For the JUNO VR software, multiple types of datasets are provided, including radioactive background, Inverse Beta Decay (IBD) [68], cosmic ray muons, and other types of events. The event display dataset was designed to encompass both simulated and real data event types. Simulated events were produced using the JUNO offline software to facilitate detector simulation, commissioning, and optimization of reconstruction algorithms. Because JUNO has not yet commenced formal data acquisition, real data events are obtained from the Data Challenge dataset [69], which has data structures identical to those expected during actual operation. With the event data conversion interface, datasets with various types of data are ready to be displayed in the Unity-based visualization and VR software.

Spatial UI and interactive control

The Spatial UI serves as an interface that facilitates interaction between the user and the VR application. For the JUNO VR project, we developed two Spatial UIs: the subdetector geometry control panel and the event display control panel, as shown in Fig. 2.

Fig. 2Full-screen View

(Color online) The Spatial UI in JUNO VR. On the left is the JUNO VR event display control panel, and on the right is the sub-detector geometry control panel

The subdetector geometry panel primarily controls the visualization attributes of the geometries of various subdetectors, including the Central Detector (CD) large PMTs, CD small PMTs, Top Tracker, and water pool PMTs. Detailed information about the subdetectors of JUNO is provided in Sect. 4.1. In addition to the sensitive detectors such as PMTs, an "Other structure" toggle controls the display of passive structures, such as the steel structure, acrylic ball, PMT support structures, and liquid filling pipelines. Additionally, the "Data type" drop-down is used to switch between different types of events collected during real data acquisition or from simulation. The "Photon trail mode" toggle enables the switching of display modes for photon paths, either represented by green lines or in a manner closely resembling the particle motion.

The event display panel is designed to implement the core functionality for event visualization, which includes a toggle for switching the display mode between simulation and data types, a slider for controlling the display of an event with its timeline evolution, a drop-down menu for selecting different types of events, and a button to play the event animation. A "Draw Hit" button initiates the animation of the full event hit process, which plays within a period of time window, with the time slider moving in sync with the event timeline, enabling the user to track the current time of the event.

Interactive control is achieved using controllers, gesture operations, eye tracking, and other input methods in HMDs. The following discussion focuses on testing the interactive control for the Meta Quest 3. For other HMDs, the cross-platform support provided by the extended reality interaction toolkit in Unity minimizes the development differences between various devices. Simple adaptations based on the specific features of HMDs are sufficient for operation.

The controller buttons resemble a typical gamepad with the addition of side buttons. The X&Y buttons on the left controller are used to control the visibility of the sub-detector geometry panel. When displayed, the position of this panel is based on the user’s orientation and appears at the front left of the user’s view. Users can drag or hide the panel to avoid obstructing their view when visualizing events. The A&B buttons on the right controller are used to control the visibility of the event display panel. When displayed, the panel appeared at the front right of the user’s view. Based on the gyroscope and accelerometer hardware of the Meta Quest 3, these planes were always oriented perpendicular to the user’s view orientation.

The joystick on the left controller controls the user’s 3D movement, based on both the controller input and the user’s view orientation. For example, when the user’s head orientation is directed towards the upper right, pushing the joystick upwards moves the user in the virtual space toward that direction. Figure 3 illustrates the user’s viewpoint during motion in the JUNO VR application. The event depicted is a simulated muon event. Additional details presented in the figure are described in Sect. 4. The joystick on the right controls the user’s viewpoint direction. Additionally, the user can change their head orientation to switch perspectives. The side button was used for interaction confirmation. Furthermore, when interacting with the Spatial UI, if the controller’s laser pointer touches the corresponding component, audio and highlight feedback are provided, making the interaction smoother for the user.

Fig. 3Full-screen View

(Color online) User perspective during motion while checking the display information of a simulated muon event in the JUNO VR application. The CD small PMTs are not shown. Detailed information about the subdetectors of JUNO is provided in Sect. 4.1

Detector units

The schematic design of the JUNO detector is illustrated in Fig. 4 [23]. The detector includes a water pool, CD [47], and Top Tracker [70]. The CD is the heart of the JUNO experiment and is filled with 20 ktons of liquid scintillator [71, 72] to serve as the target for neutrino detection. The liquid scintillator is housed within a spherical acrylic vessel with a thickness of 120 mm and an inner diameter of 35.4 m. This vessel is supported by a spherical stainless-steel structure with an inner diameter of 40.1 m. To detect photons, the CD is equipped with 17,612 20-inch PMTs and 25,600 3-inch PMTs. The CD is surrounded by a water pool containing 35 ktons of highly purified water, which effectively shields the detector from external radioactivity originating from the surrounding rocks. The water pool is also instrumental in vetoing cosmic ray muons, with 2,400 20-inch PMTs deployed as part of the water Cherenkov detector. The Top Tracker, located at the top of the water pool, plays a key role in measuring and vetoing muon tracks [73, 74].

Fig. 4Full-screen View

(Color online) Schematic view of the JUNO detector

As described in Sect. 3.2, the JUNO detector geometry was converted from the GDML file and matched between the identifier module and Unity geometry for each detector unit. The visualization effects of the entire JUNO detector in the VR application are shown in Fig. 5.

Fig. 5Full-screen View

(Color online) JUNO detector in the VR application

The light blue cylindrical structure represents the water pool, with the water pool PMTs positioned outward, as indicated by the yellow portion of the spherical structure. At the top of the water pool, the reddish-brown structure represents the Top Tracker detector. From the interior view in the JUNO VR, the spherical acrylic vessel is shown in light gray, as depicted in Fig. 2, although it is almost fully transparent in reality to allow more photons to pass through. Surrounding this vessel is a stainless steel structure, shown in dark gray in Fig. 5. The CD detector PMTs, oriented toward the center of the sphere, are designed to receive photons with their photocathodes, so that only the white tail structures of every PMTs are visible in Fig. 5.

Owing to the hardware capabilities of Meta Quest 3, there is no need to optimize the grid of detector units or replace them with simplified geometric shapes. Most of the geometric details of the detector units are preserved, achieving effects that are difficult to accomplish in event displays based on the ROOT. Additionally, for the detector units, to more closely replicate the effect of real PMTs, we assigned different material properties to the detector units, including visualization attributes such as color, reflectivity, and metallicity, to achieve the best display effect.

MC simulation event display

MC simulation is crucial for detector design and assists physicists in evaluating the performance of the detector and tuning the reconstruction algorithms. There are various types of signal and background events in JUNO, and we currently focus primarily on radioactive backgrounds, IBD signals, and muon events.

The IBD event, , is the major signal event for detecting electron anti-neutrinos in the JUNO experiment [22, 23]. JUNO identifies and reconstructs IBD events by detecting positron and neutron capture signals. This dual-signal characteristic helps to effectively identify antineutrino signal events while suppressing the large background events.

For the IBD event, there are both positron and neutron signals, whose photon paths are displayed in green and red, respectively, as shown in Fig. 6. The triggered detector units are color-coded from cyan to dark blue based on the number of hits in the event, with bluer colors indicating a higher number of hits. PMTs that were not triggered are displayed in yellow by default. Furthermore, in the time evolution of an event, the color of the fired PMTs changes with time according to the associated timing information. The neutron-induced photon paths are delayed by approximately 170 μs relative to those from the positron, and this delay can be visualized using the time slider in the JUNO VR environment.

Fig. 6Full-screen View

(Color online) Event display for a simulated IBD event in the JUNO VR application. The green lines represent the photon paths of the positron, and the red lines indicate the photon paths of the neutron. The yellow spheres represent PMTs that are not triggered, whereas the spheres with a color gradient from light blue to blue indicate PMTs with an increasing number of hits

One major background event type is cosmic-ray muon events. Muons are secondary particles produced by high-energy cosmic rays in the earth’s atmosphere and possess strong penetrating power. Despite JUNO being located 650 m deep underground, a small fraction of muons can still penetrate the overlying shielding and enter the detector, generating muon events.

Figure 7 presents the event information for the simulated muon events. The photon trajectories are represented by light green lines. These paths gradually extended over time, depicting the propagation of photons. In the simulated event shown, the directions of these photon paths may change, indicating their interactions with the detector materials. For a muon event, as a muon penetrates the detector, it continuously produces photons while depositing its energy in the liquid scintillator.

Fig. 7Full-screen View

(Color online) Event display for a simulated muon event in the JUNO VR application. The green lines represent the photons generated along the path of the muon that penetrated the detector. The controllers and lasers emitted from the controllers represent the user’s interactive control

Event reconstruction plays a key role in JUNO data processing, reconstructing the vertex and energy of an event, which is essential for determining the neutrino mass hierarchy. For point-like events, such as IBD signals, almost all photon paths originate from the same event vertex. Figure 8 shows the reconstructed vertex and MC truth. The initial particle production vertex (red sphere), derived from the MC truth, indicates where the positron is created. The weighted energy deposit vertex (green sphere) marks the positron’s annihilation point in the liquid scintillator. The reconstructed vertex (purple sphere) is produced using the event reconstruction algorithm. The reconstruction bias (light yellow line) represents the discrepancy between the reconstructed vertex and the energy deposit vertex. A shorter distance indicates a more accurately reconstructed vertex. In an ideal scenario, the reconstructed vertex converges to the true vertex.

Fig. 8Full-screen View

(Color online) Comparison of the reconstructed vertex (purple) with the weighted energy deposit vertex (green) and the particle production vertex (red) from MC truth in a simulated event. The yellow line indicates the reconstruction bias

Real data event display

For the real-data event, we utilize the Data Challenge dataset [69], whose data structures and processing pipeline are identical to those employed during data acquisition. This ensures that the software functions seamlessly once the experiment enters formal operation. The event composition in this dataset is the same as that in the MC simulation, encompassing radioactive background events, IBD signals, and muon events.

Figure 9 presents the event information for a muon event derived from real data events. The reconstructed muon travels through the detector along the magenta line. The left and right sides represent the reconstructed incident and exit points of the muons, respectively. A time offset was established by dividing the track distance by the speed of light. Users can observe the trajectory of the muon using the Spatial UI. Because the exact point of photon emission along the path cannot be determined, the photon information is not displayed in this mode. Using the reconstructed hit time, the corresponding point on the trajectory was linked to the relevant PMT unit. Once the photon particles arrive at the PMT units, the triggered PMTs will change color accordingly.

Fig. 9Full-screen View

(Color online) Display of a reconstructed muon event from the datasets in the JUNO VR application. The translucent part represents the CD acrylic sphere and its supporting components. The magenta line indicates the reconstructed muon track by connecting the points where the muon enters and exits the JUNO detector

Moreover, by exploiting Unity’s robust visualization capabilities, a specialized mode was developed to simulate photon paths using particle-like effects instead of simple line trajectories to display the propagation of particles more realistically.