Containerised Development of a Scientific Data Management System

Mr Ben Leighton1, Mr Andrew Freebairn2, Mr Ashley Sommer3, Dr Jonathan Yu 1, Dr Simon Cox1

1CSIRO, Clayton, Australia,

2CSIRO, Canberra, Australia,

3CSIRO, Brisbane, Australia


Containers provide a lightweight way of creating and maintaining software system deployments as executable code. Containers provide many benefits including ensuring software deployments are portable, reproducible, isolated, fast, efficient, provable and versioned. Containers can be repurposed and composed to create powerful new systems while also minimising manual repetition, reducing complexity and maximising reuse.

Container based approaches are proving revolutionary in facilitating reproducibility, portability and reuse of complex scientific software and compositions. Technologies like Docker provides a single human and machine readable specification of software configuration and deployment. When well architected, these specifications provide portability between systems and infrastructures and break down traditional barriers that exist across local servers, the cloud, development, test, production, and inhouse and client environments.

In this presentation, we detail a free-form collective scientific development environment that facilitates efficient knowledge transfer and common technology investment while providing large degrees of developer and stakeholder freedom. In particular, we will focus on development activities centered around the open source CKAN product, demonstrating how we engineered an improved Docker container based deployment and attached innovative backup, restore, and development specific containers to maximise the portability of the development environment and minimise the difference between local developer environments and server based dev, test and prod environments. We will demonstrate how, across two different projects we used these technologies to rapidly develop, reuse and harmonise CKAN deployments for Scientific Data Management. The patterns we propose are readily applicable to a wide variety of development problems in Science and beyond. Finally we will outline some emerging approaches for Continuous Deployment of Containers and Container Development Workflows.


Ben Leighton is a Software Engineer and Data Scientist working in Land and Water at CSIRO. He works on engineering collaborative technologies for data, and code. His research interests include reusable, reproducible, and portable environmental science and science systems.


AeRO is the industry association focused on eResearch in Australasia. We play a critical coordination role for our members, who are actively transforming research via Information Technology. Organisations join AeRO to advance their own capabilities and services, to collaborate and to network with peers. AeRO believes researchers and the sector significantly benefit from greater communication, coordination and sharing among the increasingly different and evolving service providers.

Conference Managers

Please contact the team at Conference Design with any questions regarding the conference.
© 2019 Conference Design Pty Ltd