Problem

Error in the logs:

DagsterUserCodeUnreachableError: Could not send a request to your Dagster Cloud agent since no agents have recently heartbeated. Check your agent and restart it if it is no longer running.

Solutions

• Set up monitoring alerts:

  • Dagster agent downtime alert

  • AWS CloudWatch usage alert - CloudWatch alerts on the agent's service memory usage can help you proactively manage memory and prevent the agent from running out of memory (OOM error).

• If the agent experiences an OOM issue even after increasing the memory limit:

  • Review the ECS console to understand what exactly happened to the task

  • Use a memory profiling tool such as py-spy. This tool can run without interrupting the running program and provides insights into memory usage patterns, which can help identify memory leaks or areas of inefficient memory use. For more information, see Profiling hanging or slow code with py-spy.