Scientific Achievement
A system to process the very large datasets (700 GB per 15 sec) produced by the 4D Camera electron detector using high performance computing.
Significance and Impact
The new streaming workflow significantly improves data turnaround time, enables real-time decision-making, and minimizes potential for human error. The higher throughput also allows scientists to accomplish complex experiments previously not possible.
Research Details
- The 4D Camera was designed, built, and commissioned to leverage the NERSC Superficiality API.
- Demonstrations of heavy and light atom imaging provide full information about nanoparticle atomic structure.
- A full battery stack was imaged at the nanoscale.
P. Ercius, et al., Micro. and Microanal. (2024). doi:10.1093/mam/ozae086. accepted
Research Summary
A new detector installed at the National Center for Electron Microscopy at the Molecular Foundry at LBL operates at 87,000 frames per second producing raw data sets that are up to 700 GB in size. The data, generated in under 15 seconds, would fill approximately 3 typical laptop hard drives.
We utilize modern web technology and fast networking to process the data on the NERSC high performance computing platform.
The use of high performance detector technology and high performance computing together provide faster data acquisition and faster processing. This allows scientists to perform complex experiments that were previously not possible. We demonstrate the detectors ability to image light atoms such as Fluorine at very high resolution and also investigate the structure of a large battery stack at lower resolution. This shows the flexibility of the detector to operate in very different regimes.
