Mr Ben Leighton1, Ms Julia Anticev1, Mr Alex Khassapov1
1Csiro, Clayton, Australia
The aim of the Energy Use Data Model (EUDM) project led by CSIRO Energy is to make Australian energy-use data accessible to the wider research community. A subset of this energy data, sensor readings from substations, have been provided by electricity distributors from across Australia. The EUDM project has harmonized this data to a standard format. The current set of data constitutes around 500 million observations. A goal of the project is to further add value to these harmonized datasets through generation of select pre-processed analytics products. Here we describe our initial work on “Medium Size Data Analytics” and show that, with no assumptions about time series alignment, large time series joins can be generated in reasonable time working within a familiar relational database paradigm, utilizing simple infrastructure, and with a minimum of python code.
Ben Leighton is a Software Engineer and Data Scientist working at CSIRO Land and Water. His work includes engineering collaborative technologies for data, and code. His research interests are reusable, reproducible, and portable environmental science and science systems.