Lakehouse Automation on AWS with Apache Airflow

Introduction

In the following video demonstration, we will learn how to programmatically load and upload data from Amazon Redshift to an Amazon S3-based Data Lake using Apache Airflow. Since we are on AWS, we will be using the fully-managed Amazon Managed Workflows for Apache Airflow (Amazon MWAA). Using Airflow, we will COPY raw…

--

--

--

AWS Senior Solutions Architect | 8x AWS Certified Pro | Polyglot Developer | DataOps | DevOps | Technology consultant, writer, and speaker

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

AWS Lambda — Things to Know!

Metadata abuse in AWS

Wolf in sheep’s clothing

A Look into IBM RPG

AMA Recap Vol.108 — — Hololoot: New dimension of the NFTs

Using PCA algorithm to improve performance for our product recommender — Part 1

Firebase Dynamic Links in iOS

Apollo Federation in Go part2

How to create buttons in Jupyter

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Gary A. Stafford

Gary A. Stafford

AWS Senior Solutions Architect | 8x AWS Certified Pro | Polyglot Developer | DataOps | DevOps | Technology consultant, writer, and speaker

More from Medium

Build your own AWS Glue Databrew operator for Apache Airflow

Infrastructure as code built data lake with AWS Glue, S3, Athena, and Terraform

How to run Meltano in a container on Google Cloud Composer

Extracting data from a web service via AWS Glue