Introduction to Data Engineering

By KDnuggets - 2021-03-14

Description

The Q&A for the most frequently asked questions about Data Engineering: What does a data engineer do? What is a data pipeline? What is a data warehouse? How is a data engineer different from a data sc ...

Summary

  • The Q&A for the most frequently asked questions about Data Engineering: Sometimes, a great internal tool may later become an open-source product of the company.
  • For example, one of the data product teams at Lyft built a data discovery tool called Amundsen, which was open-sourced in 2019.
  • A company’s data is often stored in different transactional systems (or even worse, as text files), and transactional data is highly normalized and suboptimal for analytics.

 

Topics

  1. Backend (0.44)
  2. Database (0.2)
  3. Security (0.11)

Similar Articles

The Growing Importance of Metadata Management Systems

By Gradient Flow - 2021-02-02

Metadata will be the foundation for data governance solutions, data catalogs, and other enterprise data systems. By Assaf Araki and Ben Lorica. Introduction As companies embrace digital technologie…

15 Essential Steps To Build Reliable Data Pipelines

By Medium - 2020-12-01

If I learned anything from working as a data engineer, it is that practically any data pipeline fails at some point. Broken connection, broken dependencies, data arriving too late, or some external…