https://prefect.io logo
Title
f

Felix Sonntag

08/04/2022, 11:23 AM
Hey, Facing some issues with Orion 2.0.2 and not sure how to debug those. It seems to fail when calling
update_flow_run
in
engine.py
. I’m running a flow via the docker-container infrastructure type. It seems to start, but there’s no output on the UI, in fact the flow run status stays in “Pending” state, even after it fails on the work queue. I also see some TF log outputs which appear when importing TF, but the actual log output of my script is not showing up. Are there any hints on how to solve this?
a

Anna Geller

08/04/2022, 11:57 AM
A flow run usually gets stuck in a pending state when the image cannot be pulled and container cannot be deployed. Did you use a custom image? If so, did you specify it on your deployment manifest?
The usual workflow would be to: 1) Build a deployment for your flow via CLI (assuming an S3 storage block with the name
dev
):
prefect deployment build yourflow.py:yourflow --name dev --tag dev -sb s3/dev --infra docker-container --output yourflow.yaml
this will generate a YAML manifest 2) Then, build your image:
docker build -t yourimage .
As a result, it will generate a sha that you can pass to your deployment manifest as shown in the screenshot Then, all is left is to apply the deployment and run it:
prefect deployment apply yourflow.yaml
f

Felix Sonntag

08/05/2022, 3:57 AM
Thanks for checking! Yes, I have done these steps of course 🙂
I think otherwise it also wouldn’t have done the TF import steps
If I re-run a few times, it sometimes gets further, but only to a point where it’s about to start my flow run and then I get errors due to faulty inputs (something I need to fix myself). But it always takes like 5-10 minutes, and mostly fails beforehand.
*this is just on Orion on my local machine
I.e. the
Docker container … has status 'running'
for a long time, without logs on the Docker container or the queue 😕 Prefect seems to do stuff, but I can’t tell what.
a

Anna Geller

08/05/2022, 11:15 AM
interesting - and if you run it in a Process rather than DockerContainer infrastructure block? and does it work if you run the docker container without Prefect?
f

Felix Sonntag

08/06/2022, 9:55 AM
Running it in a process works without any issues. And you mean starting the given image as a container? That also works of course 🙂
a

Anna Geller

08/06/2022, 11:50 AM
I was asking all those questions since there is not enough info to help atm. Can you share your Dockerfile? If you're willing to wait, there will be more recipes and docs for Docker in the next 2 weeks
f

Felix Sonntag

08/06/2022, 2:07 PM
Okay, I figured out the issue: The VPN is messing with
host.docker.internal
, i.e. connections time out. So no issue with prefect Thanks a lot though 🙂
🙌 1
a

Anna Geller

08/06/2022, 4:57 PM
Nice work, glad you figured it out