Building an ETL pipeline with Airflow and ECS

 By Medium - 2020-12-22

Description

ETL is an automated process that takes raw data, extracts and transforms the information required for analysis, and loads it to a data warehouse. There are different ways to build your ETL pipeline…

Summary

  • ETL is an automated process that takes raw data, extracts and transforms the information required for analysis, and loads it to a data warehouse.
  • This configuration is built using Fargate Tasks: an IAM that helps pulling images from your docker register, we are using ECR here.
  • The architecture described in this post makes it easy to run ETL workflows as quickly as possible and sufficiently reliable for some use cases.

 

Topics

  1. Backend (0.42)
  2. Database (0.07)
  3. Management (0.06)

Similar Articles

Creating beautiful maps with Python

By Medium - 2021-01-31

I always liked city maps and a few weeks ago I decided to build my own artistic versions of it. After googling a little bit I discovered this incredible tutorial written by Frank Ceballos. It is a…

Learning To Use Docker

By DEV Community - 2020-12-27

This is the second post in my series called "Demystifying Docker". While having read the first post i... Tagged with beginners, tutorial, devops, docker.