Support Ukrain
Adaltas logoAdaltasAdaltas logoAdaltas

Apache Airflow

Apache Airflow is an open source workflow management platform created by AirBnB engineers in 2014. Users can create, schedule and monitor complex workflows programatically while a rich user interface provides powerful visualization tools.

Workflows are authored as DAGs (directed acyclic graphs) in Python scripts implementing the "configuration as code" principle. This approach enables rapid iteration on data pipelines and a high degree of scalability.

Airflow was accepted as Apache Incubator project in March 2016 and has been an Apache top-level project since January 2019. It has established itself as de-facto standard in workflow management and is used by data engineers around the globe.

Related articles

Automate a Spark routine workflow from GitLab to GCP

Automate a Spark routine workflow from GitLab to GCP

Categories: Big Data, Cloud Computing, Containers Orchestration | Tags: Learning and tutorial, Airflow, Spark, CI/CD, GitLab, GitOps, GCP, Terraform

A workflow consists in automating a succession of tasks to be carried out without human intervention. It is an important and widespread concept which particularly apply to operational environments…

Ferdinand DE BAECQUE

By Ferdinand DE BAECQUE

Jun 16, 2020

Canada - Morocco - France

International locations

10 rue de la Kasbah
2393 Rabbat
Canada

We are a team of Open Source enthusiasts doing consulting in Big Data, Cloud, DevOps, Data Engineering, Data Science…

We provide our customers with accurate insights on how to leverage technologies to convert their use cases to projects in production, how to reduce their costs and increase the time to market.

If you enjoy reading our publications and have an interest in what we do, contact us and we will be thrilled to cooperate with you.