Parsl: A Python-based Parallel Scripting Library

Prof. Daniel S. Katz1, Yadu Babuji2, Kyle Chard2,3, Ben Clifford2, Ian Foster2,3, Zhuozhao Li2, Michael Wilde2,4, Anna Woodard2, Justin Wozniak2

1University of Illinois at Urbana-Champaign , , , 2University of Chicago & Argonne National Laboratory, , , 3Globus, , , 4Parallel Works, ,


Python is increasingly the lingua franca of scientific computing. It is used as a higher level language to wrap lower-level libraries and to compose scripts from various independent components. However, scaling and moving Python programs from laptops to supercomputers remains a challenge. Here we present Parsl (, a parallel scripting library for Python. Parsl makes it straightforward for developers to implement parallelism in Python by annotating functions that can be executed asynchronously and in parallel, and to scale analyses from a laptop to thousands of nodes on a supercomputer or distributed system. We explain the features of Parsl, and show an example of how resources are specified, how parallel functions are decorated, and how a Parsl program takes advantage of these features.


Daniel S. Katz is Assistant Director for Scientific Software and Applications at the National Center for Supercomputing Applications (NCSA), Research Associate Professor in Computer Science (CS), Research Associate Professor in Electrical and Computer Engineering (ECE), and Research Associate Professor in the School of Information Sciences (iSchool) at the University of Illinois Urbana-Champaign. He is also Guest Faculty at Argonne National Laboratory, and Adjunct Faculty at the Center for Computation & Technology (CCT), Louisiana State University (LSU). His interest is in the development and use of advanced cyberinfrastructure to solve challenging problems at multiple scales. His technical research interests are in applications, algorithms, fault tolerance, and programming in parallel and distributed computing, including HPC, Grid, Cloud, etc. He is also interested in policy issues, including citation and credit mechanisms and practices associated with software and data, organization and community practices for collaboration, and career paths for computing researchers.


AeRO is the industry association focused on eResearch in Australasia. We play a critical coordination role for our members, who are actively transforming research via Information Technology. Organisations join AeRO to advance their own capabilities and services, to collaborate and to network with peers. AeRO believes researchers and the sector significantly benefit from greater communication, coordination and sharing among the increasingly different and evolving service providers.