Yabi - A Workflow Management System and Workflow Engine for Data Reduction Pipeline

Introduction

Yabi is a 3-tier application stack to provide users with an intuitive, easy to use, abstraction of compute and data environments. Developed at the Centre for Comparative Genomics, Yabi has been deployed across a diverse set of scientific disciplines and high performance computing environments.
Yabi is deployed at IA2 Data Center to allow accredited users to run HARPS-N and GIANO-B  data reduction pipelines on private and public data from TNG archive.

Yabi Architecture

Yabi key features are:

• simply web based access
• easy tool addition and maintenance
• handling of disparate compute and storage resources ie. PBSPro, SGE, Torque, SSH, SFTP, Amazon S3, Swift
• easy graphical workflow creation environment

Yabi architecture consists of three components:

1. Client. It is typically a web browser, although a command line client also exists (yabish), which interacts with the front end application in the same way as a web browser. All traffic is via HTTPS, and users are required to log in to activate a session. Cookies are used to maintain a user session.
2. Yabi Application. It is a Python WSGI web application based on Django framework.
3. Yabu Backend. It makes use of the Celery Task Engine to queue user jobs before distribution to the appropriate execution backend via RabbitMQ.

Authentication mechanisms available for Yabi are:

• Database authentication
• LDAP authentication
• Kerberos and LDAP authentication

Yabi Interoperability with Other Workflow Management Systems

To model and run complex and recurrent tasks, research communities have developed different workflow management systems, sometimes targeted to a specific experiment and so unlikely reusable. Therefore, it is valuable for a good workflow management system to be interoperable with other workflow management systems, to rely on a widely accepted workflow language, or at least to easily export and import workflow designed with workflow management systems.

Yabi describes and manages jobs with an internal MySQL database.

Even if Common Workflow Language (CWL) is not natively supported in Yabi, tools, jobs, workflows, etc. can be exported in JSON files that can be easily translated in CWL and vice versa.
In addition, automated transfer of workflows from other workflow management systems, like Galaxy, to Yabi is also possible (see "EDGY—Export of Data from Galaxy to Yabi, Automated Workflow Transfer to Command Line Tool.", Molik et al., 2016).

Contact

Andrea Bignamini