How to Handle Pod Eviction in Kubernetes Deployments

When running Dagster in Kubernetes environments, pod evictions can occur due to resource constraints or node scaling operations. To prevent runs from hanging indefinitely, you can implement the following solutions:

"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"

Configure job-specific timeouts in your Dagster configuration to ensure runs don't hang indefinitely:

@job(
    config={
        "execution": {
            "config": {
                "max_concurrent": 1,
                "timeout_seconds": 7200  # 2 hours
            }
        }
    }
)

Pod evictions can occur due to:

• Node running out of memory

• Node scale-down operations

• Resource limit constraints to mitigate these issues, consider requesting more resources during pod creation and implementing appropriate timeout configurations.

For more information about run monitoring and timeout configurations, see:

• Run Monitoring Documentation