Building a Data Platform in 2021. How to build a modern, scalable data

By Medium - 2021-03-07

Description

This is a tough metaphor for me to use as a proud cat parent, but the sentiment has never been more accurate when it comes to data in the 21st century. While it’s true that you can solve most of your…

Summary

  • How to build a modern, scalable data platform to power your analytics and data science projects.
  • On the DYI spectrum of this component are popular tools such as Airflow, which many companies use to build robust, end to end data pipelines.
  • Transformation If the data warehouse component is the most critical piece of the modern data stack, then the transformation component is the most overlooked.
  • In the past it was acceptable that end users would consume information through dashboards and external analytics tools, but it is becoming increasingly apparent that unless data professionals can get their insights back into systems of record, their work may be all for naught.

 

Topics

  1. Backend (0.42)
  2. Database (0.22)
  3. Security (0.11)

Similar Articles

The Growing Importance of Metadata Management Systems

By Gradient Flow - 2021-02-02

Metadata will be the foundation for data governance solutions, data catalogs, and other enterprise data systems. By Assaf Araki and Ben Lorica. Introduction As companies embrace digital technologie…

15 Essential Steps To Build Reliable Data Pipelines

By Medium - 2020-12-01

If I learned anything from working as a data engineer, it is that practically any data pipeline fails at some point. Broken connection, broken dependencies, data arriving too late, or some external…