Lakehouse Automation on AWS with Apache Airflow

Introduction

In the following video demonstration, we will learn how to programmatically load and upload data from Amazon Redshift to an Amazon S3-based Data Lake using Apache Airflow. Since we are on AWS, we will be using the fully-managed Amazon Managed Workflows for Apache Airflow (Amazon MWAA). Using Airflow, we will COPY raw…

--

--

--

AWS Senior Solutions Architect | 8x AWS Certified Pro | Polyglot Developer | DataOps | DevOps | Technology consultant, writer, and speaker

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

3 git commands you should better know

How to Add the Syncfusion Blazor Chips Component to a Blazor WebAssembly App

How foodpanda reduced 45% of our BigQuery cost with reservations slots

Symfony Internals #2: Data Validation

The complete guide to the creation of a fully automated CD pipeline for an Angular frontend…

Domino Server Administrator — Your FAQ’s answered

ESP32 Devlog 3 — Barometric Sensor

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Gary A. Stafford

Gary A. Stafford

AWS Senior Solutions Architect | 8x AWS Certified Pro | Polyglot Developer | DataOps | DevOps | Technology consultant, writer, and speaker

More from Medium

DevOps for DataOps: Building a CI/CD Pipeline for Apache Airflow DAGs

My thoughts on AWS Managed Workflows for Apache Airflow

Bilbo Baggins going on an adventure

What makes Apache Airflow: most efficient platform to manage your Data Engineering workflows.

Abstracting Data Loading with Airflow DAG Factories