Use of Open-Source software for ingesting, processing, displaying and delivering data from the CosmOz soil moisture sensor network

Mr Ashley Sommer1, Mr Matthew Stenson1, Mr David McJannet1

1CSIRO Land And Water, Dutton Park, Australia


CSIRO’s CosmOz soil moisture sensor network’s software solution for ingesting, processing, displaying, and delivering soil moisture data is showing its age, has become slow, and it’s time for a replacement.

In 2010, CSIRO Land and Water installed cosmic-ray sensors in locations around Australia to form the CosmOz network. These novel sensors use cosmic rays from space to measure average soil moisture over an area of 30 hectares to depths of 10 to 50 cm. This technique has a major advantage over conventional on-ground soil moisture sensing technology that can only measure moisture content within small volumes of soil.

The sensors have been generating hourly timeseries data for nine years, with raw data transmitted via satellite and delivered over email to the now legacy Cosmoz Processing scripts, stored in a proprietary Relational Database, displayed to consumers via a WordPress site, and made available to download in text files. While that software stack has worked well enough for nine years, it has become very slow as accumulated data volumes increase.

In late 2019 our team designed and built a new CosmOz Data Processing Pipeline, CosmOz REST API, and CosmOz Web UI, using modern Open-Source software solutions. Components were implemented using MongoDB, InfluxDB, Python3, EmberJS, amCharts4, OpenAPI 2.0, and Docker.

The Open-Source software packages selected have delivered high quality data ingest and processing solutions and provide easy to use data visualisation and powerful API access options. This presentation will describe the data processing steps and provide examples of the GUI available to users.


Ashley is an Informatics Software Engineer working at CSIRO Land and Water in Brisbane.

Ashley has a BA in Information Technology from Griffith University, and has been working in the Environmental Informatics Group in L&W since 2015. Ashley is an open-source software enthusiast and active open-source project contributor. These days primarily a Python developer, Ashley is project maintainer of several popular open-source python libraries, and core-contributor of many others.


AeRO is the industry association focused on eResearch in Australasia. We play a critical coordination role for our members, who are actively transforming research via Information Technology. Organisations join AeRO to advance their own capabilities and services, to collaborate and to network with peers. AeRO believes researchers and the sector significantly benefit from greater communication, coordination and sharing among the increasingly different and evolving service providers.