Dr Robert Bell1, Mr Jeroen van den Muyzenberg2, Mr Steve McMahon3, Mr Peter Edwards1
1CSIRO IMT SC, Clayton, Australia
2CSIRO IMT SC, now Griffith University
3CSIRO IMT SC
High Performance Computing centres provide storage to complement compute services. Typically, they configure their highest-performing filesystem as a ‘scratch’ area, providing space for the temporary storage of data, and shared among all the users.
HPC service providers use scheduling to ensure compute resources are allocated to the stakeholders and users in some way reflecting need, entitlement and fairness. The same criteria need to apply to shared storage.
Quotas are typically used to control storage usage, but to support large problems over-allocation is needed, along with some mechanism to clear out old data to make way for the new. This has proved to be a difficult problem as filesystems have grown to meet the compute needs, storing hundreds of millions of files.
This paper canvasses ways to manage shared filesystems for temporary storage, and then provides a new algorithm for flushing old files that is highly scalable and responsive.
Robert Bell first worked for CSIRO as a vacation student at CSIRO Division of Meteorological Physics in November 1967.
From 1974, he worked for about 15 years in the CSIRO Division of Atmospheric Research, in programming various models of the ocean and atmosphere, and latterly in managing the computing group.
From 1990, he moved into providing support and services for CSIRO scientific computing (including a joint centre with the Bureau of Meteorology). He is currently responsible for the administration of CSIRO’s HPC National Partnerships.
He has majored on data storage facilities for science, having nurtured the CSIRO SC Data Store for over 26 years.
Since September 2015, he has been seconded part-time to the Bureau of Meteorology’s Scientific Computing Services group.
He is driven to provide services for science, particularly in computing and storage services, and in user support, having been a user himself of such services in the past.