Why am I receiving fewer PagerDuty alerts than degraded assets in Dagster+?

Problem Description

  Users may notice that they receive fewer PagerDuty alerts than the actual number of degraded assets shown in their Dagster+ deployment. This discrepancy occurs when multiple assets degrade simultaneously but the alerting system doesn't send individual notifications for each asset.

Symptoms

• Multiple assets showing as degraded in Dagster+ (e.g., 27 assets)

• Fewer PagerDuty alerts received than expected (e.g., only 10 alerts for 27 degraded assets)

• Alert policy appears to be configured correctly to cover all assets

Root Cause

  Asset health alerts only notify once when the health status changes, not for every degraded asset. When multiple assets degrade simultaneously, the alerting system may batch or deduplicate notifications. Additionally, PagerDuty may have rate limits or batching behavior that prevents all alerts from being delivered when many are triggered at once.

Solution

  Verify your alert policy scope covers all intended assets and check PagerDuty integration configuration for rate limits.

Step-by-Step Resolution

1. Review your alert policy configuration to ensure it covers all assets you want to monitor

2. Check if your alert policy has filters (asset groups, tags, or owners) that might exclude some assets

3. Verify your PagerDuty integration is correctly configured and not hitting event limits

Alternative Solutions (if applicable)

  Create separate alert policies for different asset groups to get more granular control over notifications and reduce batching effects.

Prevention

  Set up multiple alert policies grouped by asset categories or criticality levels to ensure better alert distribution and avoid overwhelming notification services with simultaneous alerts.

Related Documentation

• Creating Alert Policies

• Configuring Alert Notification Services