Ms Crystal Chua1, Mr David Jericho1, Mr Michael Usher1, Mr Gavin Kennedy1
1AARNet Pty Ltd, ,
AARNet’s CloudStor is one of the most widely adopted collaboration tools across the Australian research data ecosystem. The AARNet Cloud Services team deploys and operates a significant software stack across multiple geographically dispersed sites. AARNet continuously seeks to improve the CloudStor service offerings, and in 2018, we introduced a particularly exciting new feature to the AARNet CloudStor ecosystem: S3 gateways as a means of convenient, high-speed data transfer directly to our backend storage.
Unfortunately, initial deployments of the S3 gateways revealed a critical issue; resource contention between different access pathways to the backend EOS filesystem. This risked outages across the entire CloudStor ecosystem, and highlighted to us the need to relieve the throughput as much as possible while also reducing the blast radius of an outage, and we chose to do this by sharding the application stack.
This presentation begins with our design goals and aspirations for the redesign of the CloudStor S3rvice – to increase overall uptime and stability of the platform as a whole, and to ensure that, wherever possible, individual elements could no longer impact the rest of the service. We then discuss the AARNet Cloud Services team’s effort put into building and deploying the solution we designed, and all the things we learned along the way. We’ll then end off with a happy update on the current state of the S3 gateways, including some performance analyses, as well as our future plans to enhance the resilience of the rest of CloudStor using the same design principles.
Crystal Chua is a Cloud Services Storage Engineer at AARNet, based in Brisbane, Australia. She attended university at the University of Queensland, working as a systems programmer in the Core Infrastructure Services team while completing her Master of Information Technology degree. Currently, Crystal works on architecture, storage and integration for AARNet’s continental-scale cloud storage platform, CloudStor, ensuring the service meets and exceeds customer expectations.