From a Data Rivulet to a River: Lessons learnt from upgrading the Deterministic Seven-Day Streamflow Forecast System to provide Probabilistic Flow Ensembles at the Bureau of Meteorology

Patrick Sunter1, Daehyok Shin1, Prasantha Hapuarachchi1, Maree Carroll1, Sophie Zhang1

1Australian Bureau Of Meteorology, Melbourne, VIC, Australia

 

This presentation will discuss the challenges faced, and how we addressed them, in a multi-year project to upgrade the Australian Bureau of Meteorology’s (BoM) Seven-Day Streamflow Forecasting service to provide ensemble probabilistic forecasts.

The project involved integrating many new statistical approaches, algorithms and data sources – several of which originated in collaborative research with the CSIRO and Australian universities – into a production-ready system able to publish results daily for several hundred locations on the Bureau’s website.

We will discuss the ways this work challenged our existing systems and the ways we addressed those challenges, including:
• data management and provenance: requiring new approaches to handle version control of much larger data artefacts and model representations, including moving to Git Large File Storage (LFS) for managing hydrological model configuration and verification data;
• performance and scalability: including updating our Python software that previously worked effectively on deterministic Numerical Weather Prediction (NWP) grids to deal with higher-resolution ensemble forecasts;
• system integration: the challenge of integrating new R&D software into production architectures, including dealing with legacy systems;
• Redesigning outputs for better scientific communication: Including updating graphical plots that balance communicating the extra information included in probabilistic forecasts, while not overwhelming generalists with too much information.
Finally, we will attempt to draw out the most relevant lessons learnt from this project for other eResearch practitioners and other scientific software engineers.


Biography:

Patrick Sunter has worked in the field of software engineering of scientific computing applications for more than a decade, participating in multiple collaborative projects in research and industry. Building on a base of software engineering post-graduate training, he has worked across the domains of geophysics, materials science, and spatial information to develop software to support modelling and analysis of complex problems.

Patrick joined the Australian Bureau of Meteorology’s Water Forecasting Services section in 2016, and since then has worked on upgrades to the software and information systems that underpin the Bureau’s seasonal and short-term streamflow forecasting services.

ABOUT AeRO

AeRO is the industry association focused on eResearch in Australasia. We play a critical coordination role for our members, who are actively transforming research via Information Technology. Organisations join AeRO to advance their own capabilities and services, to collaborate and to network with peers. AeRO believes researchers and the sector significantly benefit from greater communication, coordination and sharing among the increasingly different and evolving service providers.