Article Metrics


Online attention

Handling high data rate detectors at Diamond Light Source

DOI: 10.1088/1742-6596/425/6/062008 DOI Help

Authors: Ulrik Pedersen (Diamond Light Source) , Nick Rees (Diamond Light Source) , Mark Basham (Diamond Light Source) , Frederik Ferner (Diamond Light Source)
Co-authored by industrial partner: No

Type: Journal Paper
Journal: Journal Of Physics: Conference Series , VOL 425

State: Published (Approved)
Published: March 2013

Abstract: An increasing number of area detectors, in use at Diamond Light Source, produce high rates of data. In order to capture, store and process this data High Performance Computing (HPC) systems have been implemented. This paper will present the architecture and usage for handling high rate data: detector data capture, large volume storage and parallel processing. The EPICS area Detector frame work has been adopted to abstract the detectors for common tasks including live processing, file format and storage. The chosen data format is HDF5 which provides multidimensional data storage and NeXuS compatibility. The storage system and related computing infrastructure include: a centralised Lustre based parallel file system, a dedicated network and a HPC cluster. A well defined roadmap is in place for the evolution of this to meet demand as the requirements and technology advances. or processing the science data the HPC cluster allow efficient parallel computing, on a mixture of ×86 and GPU processing units. The nature of the Lustre storage system in combination with the parallel HDF5 library allow efficient disk I/O during computation jobs. Software developments, which include utilising optimised parallel file reading for a variety of post processing techniques, are being developed in collaboration as part of the Pan-Data EU Project ( These are particularly applicable to tomographic reconstruction and processing of non crystalline diffraction data.

Journal Keywords: Compatibility; Computer Codes; Diffraction; Light Sources; Mixtures; Parallel Processing; Performance; Processing; Storage

Subject Areas: Technique Development

Facility: Swiss Light Source