James Hannah
The DAFNI project has completed its Year Three hardware procurement, which has included adding over 1.5PB of storage to the platform.
In 2020 the DAFNI platform team has focused on procuring flash and disk storage in order to provide a solid set of infrastructure on which to host models and growing datasets.
To provide working storage for the demands of increasingly data-hungry models and workflows, the team have invested in an expandable NVM-e/SSD flash storage appliance from Pure Storage. This appliance will be used to provision high-performance PostgreSQL/PostGIS databases, and as “scratch space” for I/O bound models. This appliance can be extended in future procurements to meet the demands of users on the platform.
Additionally, the team have procured a number of state-of-the-art Dell servers containing 1.68PB of raw storage to serve as medium-term storage of datasets and model outputs. These servers will be added to the DAFNI Kubernetes environment to ensure that users and collaborators will have ample space on the platform to store and process their datasets and outputs over the next few years.
An example of one user of this storage are the six UKCRIC Urban Observatories, real-world laboratories which will produce huge amounts of raw data over the years. DAFNI’s hardware purchases this year will provide a space outside of each observatory for this research data to live, and will provide the data to DAFNI users from across the community in a central database repository.
The new DAFNI hardware will be connected via a high-performance 100Gbps network to the existing servers and storage and be housed at STFC’s Rutherford Appleton Laboratory, home of huge UK computing infrastructures such as JASMIN and IRIS.
The new aggregate hardware specs are as follows:
- 27 server nodes
- 16.8TB of memory
- 792 CPU cores
- ~1.6PB server storage
- 10×Nvidia V100 GPUs (paired)
- 10 iSCSI storage devices – 2PB storage total:
- Dell Equallogic: A high-IOPS pool with 127TB of “fast” storage (for VM block devices, databases etc)
- Dell Equallogic: A high-density pool with 205TB of “slow” storage (for NFS, home directories, nearline backups etc)
- Pure Storage: a very high-throughput SSD/Flash storage pool of 20TB (with dedup, compression and remote monitoring)
- 8 Mellanox SN2100 switches
- 128×100Gbps ports
- 2×40Gbps connections to the STFC core network
- 1×10Gbps management uplink to the RIG systems network
There will be ongoing hardware purchases to provide data storage solutions as the community grows on DAFNI and their requirements become fully realised.