Azure Data Factory Pipeline Return Value from child pipeline to Parent Pipeline

In this article we will perform step by step demo on how to customize Pipeline Return Value from child pipeline and use it in Parent Pipeline in Azure Data Factory.

Along with in Azure Data Factory, Azure Synapse Pipelines also allows us to customize the return value from child pipeline and subsequently consume that return value in parent pipeline.

We can use the Set Variable activity to return values from the child pipeline to the calling pipeline.

Currently this feature “Pipeline return value” is in public preview.

In calling pipeline-child pipeline paradigm, Introduce pipeline return value, a dictionary of key value pairs, that allows communications between child pipelines & parent pipeline.

Scenario: When we need to use this feature?
  • Child pipeline is required for complex pipeline with various activities with dependencies.
  • Solution is to logically grouping the related activities in child pipeline or separate pipeline.
  • We need this feature to maintain the dependencies by getting output from previous or child pipeline.

So using this new “Pipeline return value” feature, activities can return a value from the child Pipeline.

Important note:

Pipeline variables are user-defined. ADF Pipeline return value is a system variable that you can set here then use in the parent consuming pipeline.

For DEMO on all other LIST of IMPORTANT articles on Azure Data Factory, Azure Synapse and Azure Databricks, please scroll down to bottom of this article.

Use Case:

We can use “Pipeline return value” for several scenario in Azure data factory including few below:

  • Script Activity : As in this demo we will use this parent pipeline’s Script Activity to log integer return value or expression value from child pipeline.
  • Web Activity: As a string value, we can define URL as pipeline return value to be used in Web Activity in parent pipeline.
  • Wait Activity: We can use wait period as int value from child pipeline to be used for a wait activity
DEMO: How to Pipeline Return Value from Child Pipeline in Azure Data Factory?

In this demo we will use Set Variable and Execute Pipeline activities and below high level steps

  1. Child Pipeline setup
  2. Use Set Variable Activity : customize return value using Set Variable inside child pipeline
  3. Parent Pipeline setup: ADF Execute Pipeline activity to invoke child pipeline and consume return value

Let’s have a quick step by step demo of Azure Data Factory Pipeline return value:

Child Pipeline setup:

Create Child pipeline and named it as ChildPipeline.

  1. Now add a Set Variable activity
  2. give a proper name like what it set or return to the parent pipeline . Example as ‘Set Record Processed Value’.
  3. Now navigate to the Settings tab
  4. In ‘Variable type’ field click on “Pipeline return value (preview)“
  5. For ‘Values’ field, click on +New option to add new value like set the static value or expression value.
Azure Data Factory – Return value from a Child Pipeline to a Parent Pipeline
Azure Data Factory setting return value on Child Pipeline

Now once we click on +New option, we need to add the proper Variable name like ‘RecordProcessed’, select the data type as Int and give static or expression value. here we are assigning int value as 1010 as shown in below image.

Azure Data Factory Pipeline assign value Set Variable value in Child Pipeline
Azure Data Factory Pipeline assign value Set Variable value in Child Pipeline

Azure Data Factory Pipeline pipeline return value data type:

There are several data types which includes expression as well.

Azure Data Factory Pipeline pipeline return value data type
Azure Data Factory Pipeline pipeline return value data type

Now debug this child pipeline to see the output from Set Variable Activity:

Azure Data Factory Pipeline Debug Child Pipeline: see output from Set Variable Activity
Azure Data Factory Pipeline Debug Child Pipeline: see output from Set Variable Activity

Below is output from Set Variable Activity in Child Pipeline of Azure Data Factory Pipeline

Azure Data Factory : Set Variable Activity debug output in Child Pipeline
Azure Data Factory : Set Variable Activity debug output in Child Pipeline
Parent pipeline Setup:

Create parent pipeline and named it as ParentPipeline

  1. Add the execute pipeline; This activity will execute ChildPipeline.
  2. Name it to Execute Child Pipeline
  3. Navigate to Settings tab
  4. Select ‘ChildPipeline’ from drop down list.
Azure Data Factory Pipeline Invoke Child Pipeline
Azure Data Factory Pipeline Invoke Child Pipeline: Pipeline return value

For this demo add a Script activity to log the Record Processed for each pipeline run and rename it to ‘Record Processed In Child Pipeline’.

Alternative: But you can use Set Variable Activity instead of Script activity in your case and create a variable in the parent pipeline to store the returned value.

Now in Settings tab and click on +New option add script parameter and give a name and type along with return value assigned from child pipeline using expression shown in subsequent image.

Note, please select the linkedsevice for database and provide the required query as required.

Azure Data Factory Pipeline Script Activity Input parameter
Azure Data Factory Pipeline Script Activity Input parameter

Please click on the value textbox at above image to see the details expression of return value from child pipeline as shown in below image:

@activity('Execute Child Pipeline').output.pipelineReturnValue.RecordProcessed

Here, pipelineReturnValue is system variable and RecordProcessed is the variable name from Set Variable Activity of child pipeline.

Now let us debug the parent pipeline, here pipeline return value of the child pipeline becomes the activity output of the Azure Data Factory Execute Pipeline Activity.

Now click on the input of script activity (Record Processed In Child Pipeline) to see the below output from execute pipeline activity i.e. from child pipeline:

Azure Data Factory Pipeline parent pipeline debug
Azure Data Factory Pipeline parent pipeline debug

Below is the return value i.e. input execute pipeline activity i.e. from child pipeline

Azure Data Factory Pipeline : Return value from child pipeline
Azure Data Factory Pipeline : Return value from child pipeline

So in this article we performed step by step demo on how to return value from child pipeline and consume it in parent pipeline using Set Variable Activity.

Thanks for reading; Please feel free to comment below for any question/query/suggestion or issue you are facing.

DEMO on all LIST of IMPORTANT articles:
Tutorial from Azure Synapse:

MERGE command in Azure Synapse Analytics Dedicated SQL pool: https://sarnendude.com/merge-command-in-azure-synapse-analytics-dedicated-sql-pool/

Azure Synapse Link for Azure SQL Database using Change Feed : Demo: https://sarnendude.com/azure-synapse-link-for-azure-sql-database-using-change-feed-demo/

Azure Synapse Intelligent Cache for Apache Spark: https://sarnendude.com/azure-synapse-analytics-intelligent-cache-for-apache-spark/

Flowlet transformation in Azure Data Factory and Azure Synapse pipeline: https://sarnendude.com/azure-synapse-analytics-intelligent-cache-for-apache-spark/

Azure Synapse Tutorial: Three In ONE Service: https://sarnendude.com/azure-synapse-tutorial/

Tutorial from Azure Data Factory:

Azure Data Factory Data Flow: Change Data Capture Architecture & Demo: https://sarnendude.com/azure-data-factory-data-flow-support-change-data-capture/

How to create and use Flowlet transformation in Azure Data Factory and Azure Synapse pipeline: https://sarnendude.com/how-to-create-and-use-flowlet-transformation-in-azure-data-factory-and-azure-synapse-pipeline/

Azure Managed Identity Authentication for Azure SQL Database by Azure Data Factory: https://sarnendude.com/azure-managed-identity-authentication-for-azure-sql-database-by-azure-data-factory/

Cast Transformation DEMO in Mapping Data flow of Azure Data Factory: https://sarnendude.com/cast-transformation-in-mapping-data-flow-of-azure-data-factory-synapse-analytics/

Azure Data Factory and Synapse Analytics provides Script Activity to execute DML & DDL script: https://sarnendude.com/azure-data-factory-and-synapse-analytics-provides-script-activity/

How ABN AMRO Bank uses Data Mesh architecture on Microsoft Azure for faster data insights & business decisions: https://sarnendude.com/azure-data-mesh-architecture-abn-amro-bank/

Azure Data Factory end to end Data Lineage demo using Azure Purview: https://sarnendude.com/azure-data-factory-end-to-end-data-lineage-demo-using-azure-purview/

Azure Data Factory data flows: ‘incremental extract’ feature allows to read only changed rows on Azure SQL DB sources: https://sarnendude.com/azure-data-factory-data-flows-incremental-extract-feature-allows-to-read-only-changed-rows-on-azure-sql-db-sources/

Tutorial from PySpark and Azure Databricks:

Delta Lake’s Change Data Feed (CDF) Demo in Azure Databricks: https://sarnendude.com/delta-lakes-change-data-feed-cdf-demo-in-azure-databricks/

How to Read CSV file in PySpark easily in Azure Databricks: https://sarnendude.com/how-to-read-csv-file-in-pyspark-easily-and-load-into-dataframe/

How to Write CSV file in PySpark easily in Azure Databricks: https://sarnendude.com/how-to-write-csv-file-in-pyspark-easily-in-azure-databricks/

Leave a Reply