https://prefect.io logo
f

Fina Silva-Santisteban

02/05/2021, 11:42 PM
Hi Prefect community! We’re currently running the prefect server, prefect docker agent with a local docker image and I’m using Docker storage which points at my local Dockerfile. That works fine, but switching the backend from server to cloud somehow doesn’t. We have Prefect Cloud seats and I’ve successfully authenticated my agent, and I’ve successfully registered my flows. When I try to run a flow using the Prefect Cloud UI, my prefect docker agent console says
Deploying flow run...
but the flow gets stuck in the
Submitted
stage. What am I missing?
(Once this setup works the plan is to make the next incremental change, which is to use an AWS ECR instead of a local image)
c

Chris White

02/06/2021, 1:00 AM
Hi Fina! Usually runs stuck in submitted states are caused by the image either failing to get pulled or failing to initialize properly, but your agent should detect those situations and either fail the run or create a log. A few questions: • can you see a container ID referenced in the logs? If so can you see if that container produced any logs or issues? • Is your agent running a recent version of prefect? • if you restart your agent with the —verbose flag and place the run back in a scheduled state do you notice anything?
f

Fina Silva-Santisteban

02/09/2021, 12:36 AM
Thank you for your help @Chris White! Yes the prefect cloud dashboard shows me a container ID in the logs section:
Copy code
Submitted for execution: Container ID: 92c(...)
But when I check my local containers by running
docker ps
it doesn’t show anything! (It does show containers as expected when I run things with the prefect server instead of prefect cloud) I’m currently using prefect
0.14.5
. The agent doesn’t seem to have a
-verbose
flag(?) What am I missing?
c

Chris White

02/09/2021, 2:27 AM
Hm very interesting;
prefect agent start --verbose
should be a documented / exposed CLI flag — would you mind opening a GitHub issue with the details so we can investigate further?
f

Fina Silva-Santisteban

02/09/2021, 5:12 PM
@Chris White I think I had misunderstood: Since I’m using local Docker Storage and the Docker Agent the command I use to run the agent is
prefect agent docker start
. That command doesn’t have a verbose flag. However
prefect agent start --verbose
does exist, but that agent throws an error if I try to use it with my setup, and that error makes sense:
Error while deploying flow: TypeError('Unsupported Storage type: Docker')
. Should I open a github issue to request the verbose flag for the docker agent? Besides that, the problem still seems to be that the container doesn’t exist, even though I get provided with a container ID. What are the best next steps?
AHA there is no verbose flag but there is
--show-flow-logs
! And that is finally giving me an error message (vs silently failing):
Copy code
raise ClientError(
prefect.utilities.exceptions.ClientError: Malformed response received from Cloud - please ensure that you have an API token properly configured.
I’ll take a closer look at my token setup and let you know how that goes!
DOUBLE AHA: I had accidentally given
PREFECT__CLOUD__AGENT__AUTH_TOKEN
the same value as
PREFECT__CLOUD__AUTH_TOKEN
! Fixed that and now everything is running wonderfully! Thanks so much for your help @Chris White! Finding the logs was really the key to solving this!
c

Chris White

02/09/2021, 6:45 PM
Ahhh nice!! Glad you figured it out 😄
🙏 1
@Marvin archive “Flow Run stuck in Submitted - how to debug?”
marvin 1