Hi all, What's the best way to troubleshoot the fo...
# ask-communityj
Jay Sundaram
03/13/2021, 3:53 AMHi all, What's the best way to troubleshoot the following?
• my Docker image is in ECR
• my ECSAgent is running on an AMI instance
• Prefect Cloud Dashboard indicates my ECSAgent is online
• labels match: 'prefect agent ecs start --label my-flows' and ECSRun(labels=['my-flows'])
• I navigate to my project > Flow and select the latest version and click QUICK RUN
and then nothing happens. The ECSAgent doesn't find the flow to submit for execution.
Since the flow is not submitted, there are no logs to inspect.
I've had success with a similar set up except where my Docker image was pulled from Docker Hub instead.
Is there more logging somewhere else?
Appreciate any tips!
Jay Sundaram
03/13/2021, 5:47 PMSolution: i updated the invocation of the Flow constructor to include S3 as the storage (the default is local storage- and thus my AMI ECSAgent did not have access to the Flow's metadata).
Might be helpful to include some output when the Flow is registered along the lines of:
"You've registered your flow configured for local storage (default since none was specified). Your flow will be processed by a local agent monitoring for the following labels: [flow's labels]."
I see the INFO line below indicates the flow has been uploaded to S3.
If the absence of that line is the only indicator, it might be helpful to provide a little more info.
Copy code
python my_favorite_flow.py
Result check: OK
[2021-03-13 12:33:19-0500] INFO - prefect.S3 | Uploading my-important-flow/2021-03-13t17-33-19-588984-00-00 to my-favorite-s3-bucket
Flow URL: <https://cloud.prefect.io/my-prefect-cloud-gmail-com-s-account/flow/a5b83b1f-4dea-40eb-9dcf-6db3f2d019b2>
└── ID: 5e5a9240-fc14-4467-a5df-659b0a66ea61
└── Project: my-prefect-project
└── Labels: ['s3-flow-storage']c
Chris White
03/13/2021, 7:20 PM
Hi @Jay Sundaram! Thanks for the feedback and for following up with your solution; the UI should have displayed a small warning icon on the flow run to alert you to the fact that there were no agents that had the appropriate labels to run your flow. That being said, we’re aware this can be a little confusing at times and so we’re always looking for ways to improve the debug experience — I like your suggestion of a more verbose message for the labels in the display so we’ll take that into consideration!
j
Jay Sundaram
03/13/2021, 9:05 PMHi @Chris White, Ah yes- so all agents are aware of the registered flows but only the agent that shares the same label with those flows will submit them for execution.
If that is the case, yes- it would be helpful if the UI could report that a registered ECSRun flow was detected but that the storage for that flow was set to local i.e.: a deployment environment in which there are no RUNNER agents executing in that same environment (at least that that Cloud/Server are aware of).
3 Views