DevOps for DataOps: Building a CI/CD Pipeline for Apache Airflow DAGs

Introduction

In this post, we will learn how to use GitHub Actions to build an effective CI/CD workflow for our Apache Airflow DAGs. We will use the DevOps concepts of Continuous Integration and Continuous Delivery to automate the testing and deployment of Airflow DAGs to Amazon Managed…

--

--

--

AWS Senior Solutions Architect | 8x AWS Certified Pro | Polyglot Developer | DataOps | DevOps | Technology consultant, writer, and speaker

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

web visualization using tork-a rwt

Interview with Zhengsong Tu: A Journey from Newbie to Apache APISIX Committer

Using EFS instead of EBS service on the AWS as Create/launch Application using Terraform

Day 6 — Check the Inspector!

Cars get old too

How to Make Software Development Your Next Career

Go-Tenable: A CLI tool for scanning and vulnerability management

How To Scrape Aliexpress Data in just 3 steps

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Gary A. Stafford

Gary A. Stafford

AWS Senior Solutions Architect | 8x AWS Certified Pro | Polyglot Developer | DataOps | DevOps | Technology consultant, writer, and speaker

More from Medium

Using environment variables to control a dynamic Airflow DAG

Airflow Monitoring via Slack

Apache Airflow Testing Guide

Applied Apache Airflow- Pros/Cons

Apache Airflow components with Open Source Technologies