Pangeo: Scalable Geoscience Tools in Python — Xarray, Dask, and Jupyter

Dr James Munroe1

1Memorial University of Newfoundland

 

Earth scientists face serious challenges when working with large datasets. Pangeo is a rapidly growing community and software ecosystem for scalable geoscience based on open source scientific python. Pangeo’s three core packages are:

  1. Jupyter, a web-based tool for interactive computing
  2. xarray, a data-model and toolkit for working with N-dimensional labeled arrays
  3. Dask, a flexible parallel computing library

When combined with distributed computing, these tools can help geoscientists perform interactive analysis on datasets up to petabytes in size. In this interactive, tutorial we will demonstrate how to employ this platform using real science examples from physical oceanography and hydrology. Participants will follow along using Jupyter notebooks to interact with xarray and Dask running on public cloud.

Agenda (Tentative):

  • Introduction to Pangeo Project and Software Ecosystem
  • Hands-on interactive tutorial of xarray
  • Break / Discussion
  • Hands-on interactive tutorial of dask
  • Lunch
  • Getting started with cloud-native data analysis
  • Break / Discussion
  • Deploying your own Pangeo platform on cloud or HPC computing resources

Learning Objectives: Participants will learn how to:

  • Recognize the software packages that comprise the Pangeo platform and explain how they work together
  • Load datasets using xarray from netCDF files, openDAP endpoints, and Zarr stores
  • Analyze data using xarray’s label-based operations and groupby feature
  • Work with very large xarray datasets using Dask

ABOUT AeRO

AeRO is the industry association focused on eResearch in Australasia. We play a critical coordination role for our members, who are actively transforming research via Information Technology. Organisations join AeRO to advance their own capabilities and services, to collaborate and to network with peers. AeRO believes researchers and the sector significantly benefit from greater communication, coordination and sharing among the increasingly different and evolving service providers.

Conference Managers

Please contact the team at Conference Design with any questions regarding the conference.
© 2017 Conference Design Pty Ltd