Table of contents
  1. How to Monitor ADF Pipelines
  2. Ways to Monitor ADF Pipelines
  3. 1. Using Azure Portal
    1. Monitoring Tab
    2. Pipeline Runs
    3. Activity Runs
  4. 2. Setting Alerts and Metrics
    1. Creating Alerts
    2. Viewing Metrics
  5. 3. Log Analytics and Azure Monitor
    1. Enable Diagnostic Logs
    2. Using Log Analytics
    3. Azure Monitor
  6. 4. Custom Monitoring Solutions
    1. Using PowerShell
    2. Using Logic Apps

How to Monitor ADF Pipelines

No matter how good your workflow is, errors will happen. Here are some ways to monitor your ADF pipelines. My favorite way is to set up email alerts with a specific subject line and severity. Alright, here are all the ways you can monitor your pipelines.

Ways to Monitor ADF Pipelines

  1. Using Azure Portal
  2. Setting Alerts and Metrics
  3. Log Analytics and Azure Monitor
  4. Custom Monitoring Solutions

1. Using Azure Portal

Monitoring Tab

  • Go to ADF: Open the Azure portal and find your Data Factory instance.
  • Monitor Tab: Click on the “Monitor” tab on the left side. This section shows details about pipeline runs, activity runs, and triggers.

Pipeline Runs

  • See Pipeline Runs: Here, you can see all pipeline runs. Filter by status (Succeeded, Failed, In Progress) and time range.
  • Run Details: Click on a pipeline run to see detailed info, including the status of each activity in the pipeline.

Activity Runs

  • Activity Details: This section shows individual activities within each pipeline run. You can see input, output, and error messages for each activity.

2. Setting Alerts and Metrics

Creating Alerts

  • Go to Alerts: In your ADF instance, go to the “Alerts” section.
  • Create Alert: Click on “New alert rule” to make a new alert.
  • Set Alert: Choose the condition (e.g., pipeline failure), threshold, and notification method (e.g., email, SMS).

Viewing Metrics

  • Metrics: You can check metrics like pipeline run duration, activity duration, and trigger runs. Metrics help you understand performance and spot issues.

3. Log Analytics and Azure Monitor

Enable Diagnostic Logs

  • Diagnostic Settings: In your ADF instance, go to “Diagnostic settings” and turn on diagnostic logs. Send these logs to a Log Analytics workspace, Event Hub, or Storage Account.
  • Log Analytics Workspace: If you use Log Analytics, you can run queries to analyze the logs.

Using Log Analytics

  • Query Logs: In the Log Analytics workspace, run queries to find details about pipeline runs, failures, and performance.
  • Sample Query:
    ADFPipelineRun
| where Status == 'Failed'
| project RunId, PipelineName, Start, End, Status, ErrorMessage

Azure Monitor

  • Integration: You can use Azure Monitor with ADF for centralized monitoring.
  • Alerts: Create alerts based on log queries and metrics using Azure Monitor.

4. Custom Monitoring Solutions

Using PowerShell

  • Automation: Use PowerShell scripts to automate monitoring and reporting of pipeline runs.
  • Sample Script: 
    $dataFactoryName = "your_data_factory_name"
$resourceGroupName = "your_resource_group_name"

$pipelineRuns = Get-AzDataFactoryV2PipelineRun -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName
$failedRuns = $pipelineRuns | Where-Object { $_.Status -eq 'Failed' }
  
foreach ($run in $failedRuns) {
    Write-Output "Pipeline: $($run.PipelineName) | Run ID: $($run.RunId) | Status: $($run.Status)"
}

Using Logic Apps

  • Automated Workflows: Create Logic Apps to handle pipeline monitoring tasks automatically, like sending notifications or triggering other workflows based on pipeline statuses.