Troubleshooting sensor tick failures with "DagsterUserCodeUnreachableError" in Hybrid K8s Deployments

Last updated: October 9, 2025

If you are using Dagster+ Hybrid deployed on Kubernetes and the sensor evaluation error is this below variant, it could be due to the code server pod that is running the sensor code becoming unresponsive.

Redeploying the code location would likely resolve the problem as a short-term solution by spinning up a brand new pod.

A common cause is sporadically hanging code in sensors gradually burning through threads over time.

To prevent this issue from recurring:

  1. Upgrade to Dagster Agent version 1.7.2 or later, which includes automatic detection and redeployment of unresponsive pods

  2. For debugging stuck pods, use py-spy to analyze thread activity by following the guide in our GitHub discussion

To use py-spy for debugging, you'll need to configure appropriate security context settings on the pod before it starts. These settings cannot be modified on an already running pod.

dagster._core.errors.DagsterUserCodeUnreachableError: Unable to reach the user code server for sensor <name_of_sensor>. Sensor will resume execution once the server is available.

  File "/dagster/python_modules/dagster/dagster/_daemon/sensor.py", line 291, in __exit__
    raise DagsterUserCodeUnreachableError(

The above exception was caused by the following exception:
dagster._core.errors.DagsterUserCodeUnreachableError: dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE

Stack Trace:
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 843, in _process_api_request
    api_result = self._handle_api_request(
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 703, in _handle_api_request
    serialized_sensor_data_or_error = client.external_sensor_execution(
  File "/dagster/dagster/_grpc/client.py", line 466, in external_sensor_execution
    return self._query(
  File "/dagster/dagster/_grpc/client.py", line 173, in _query
    self._raise_grpc_exception(
  File "/dagster/dagster/_grpc/client.py", line 156, in _raise_grpc_exception
    raise DagsterUserCodeUnreachableError(

The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.20.28.136:4000: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-02-28T14:10:56.570462062+00:00", grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:172.20.28.136:4000: Failed to connect to remote host: Connection refused"}"
>