Scratch Management and Scalable Flushing

Dr Robert Bell1, Mr Jeroen van den Muyzenberg2, Mr Steve McMahon3, Mr Peter Edwards1

1CSIRO IMT SC, Clayton, Australia

2CSIRO IMT SC, now Griffith University

3CSIRO IMT SC

 

High Performance Computing centres provide storage to complement compute services.  Typically, they configure their highest-performing filesystem as a ‘scratch’ area, providing space for the temporary storage of data, and shared among all the users.

HPC service providers use scheduling to ensure compute resources are allocated to the stakeholders and users in some way reflecting need, entitlement and fairness.  The same criteria need to apply to shared storage.

Quotas are typically used to control storage usage, but to support large problems over-allocation is needed, along with some mechanism to clear out old data to make way for the new.  This has proved to be a difficult problem as filesystems have grown to meet the compute needs, storing hundreds of millions of files.

This paper canvasses ways to manage shared filesystems for temporary storage, and then provides a new algorithm for flushing old files that is highly scalable and responsive.


Biography:

Robert Bell first worked for CSIRO as a vacation student at CSIRO Division of Meteorological Physics in November 1967.

From 1974, he worked for about 15 years in the CSIRO Division of Atmospheric Research, in programming various models of the ocean and atmosphere, and latterly in managing the computing group.

From 1990, he moved into providing support and services for CSIRO scientific computing (including a joint centre with the Bureau of Meteorology).  He is currently responsible for the administration of CSIRO’s HPC National Partnerships.

He has majored on data storage facilities for science, having nurtured the CSIRO SC Data Store for over 26 years.

Since September 2015, he has been seconded part-time to the Bureau of Meteorology’s Scientific Computing Services group.

He is driven to provide services for science, particularly in computing and storage services, and in user support, having been a user himself of such services in the past.

ABOUT AeRO

AeRO is the industry association focused on eResearch in Australasia. We play a critical coordination role for our members, who are actively transforming research via Information Technology. Organisations join AeRO to advance their own capabilities and services, to collaborate and to network with peers. AeRO believes researchers and the sector significantly benefit from greater communication, coordination and sharing among the increasingly different and evolving service providers.

Conference Managers

Please contact the team at Conference Design with any questions regarding the conference.
© 2017 Conference Design Pty Ltd