Microsoft Azure ADF - Dynamic Pipelines

 By SQLServerCentral - 2021-03-09

Description

Azure Data Factory (ADF) is a cloud based data integration service that allows you to create data-driven workflows in the cloud for orchestrating and

Summary

  • Azure Data Factory (ADF) is a cloud based data integration service that allows you to create data-driven workflows in the cloud for orchestrating and automating data movement and data transformation.
  • For the dynamic pipeline config settings, I will talk about the first pipeline, the Router data factory pipeline.
  • This means config is not available and it makes no sense to move further in the pipeline.
  • The property, useSourceFileName, has a value “true” or “false”, which will be looked at in the Data Factory in this ‘IF condition’.

 

Topics

  1. Backend (0.51)
  2. Database (0.18)
  3. Security (0.1)

Similar Articles

15 Essential Steps To Build Reliable Data Pipelines

By Medium - 2020-12-01

If I learned anything from working as a data engineer, it is that practically any data pipeline fails at some point. Broken connection, broken dependencies, data arriving too late, or some external…