https://prefect.io
Join Slack
Hey all in the last couple weeks our docker + ECS flow runs have started failing with a cryptic mess...
# ask-community
j
Hey all in the last couple weeks our docker + ECS flow runs have started failing with a cryptic message:
Copy code
19:27:36.656 | INFO    | prefect.flow_runs.runner - Opening process...
/usr/local/bin/python3.12: No module named prefect.engine
19:27:36.671 | ERROR   | prefect.flow_runs.runner - Process for flow run 'petite-snail' exited with status code: 1
19:27:36.701 | INFO    | Flow run 'petite-snail' - Downloading flow code from storage at '.'
I'm perfectly able to run our flows in a 
process
worker or by invoking the flows directly through python. Anyone have any ideas?
✅ 1
j
👋 How are you installing prefect? It is baked into your image in ECS? Or installed dynamically? Are you using a virtual env? By default (unless you've modified it) in your container prefect will run 
prefect flow-run execute
to launch a 
Runner
. That 
Runner
will then launch your flow in a child process with 
python -m prefect.engine
it looks like 
prefect
is installed for the first part of that but not the second part
j
We use the default prefect image
prefecthq/prefect:3-latest
Ya, that looks right. When I run 
python -m prefect.engine
I get that same error. Any ideas?
j
just to clarify, where are you running that command?
j
/opt/prefect/project
j
in ECS?
can you run 
prefect version
as well?
j
oh, ya, ECS but also using a local docker worker
Copy code
Version:             3.0.11
API version:         0.8.4
Python version:      3.12.7
Git commit:          a17ccfcf
Built:               Thu, Oct 24, 2024 5:36 PM
OS/Arch:             linux/aarch64
Profile:             ephemeral
Server type:         server
Pydantic version:    2.9.2
j
Sorry 😅 you're running 
prefect version
and 
python -m prefect.engine
both locally on your machine (not inside of a docker container) and 
python -m prefect.engine
gives an error and 
prefect version
gives the above output. Also when using an 
ecs
worker and work pool, the container exits with the original error you pasted. Do I have that right?
j
Running both from within a docker container
Yes to the rest
j
gotcha! and the image for the container you're getting that error in is 
prefect:3-latest
?
j
umungst others
j
hm the latest prefect image should be up to 
3.1.15
thanks for bearing with all the questions, trying to figure out how I can reproduce
j
No problem at all. Thank you so much for the help!
Oh, I guess I am installing prefect by hand. We're using hatch and prefect is one of our dependencies
j
oh interesting! I am not super familiar with hatch, but could it be installing 
prefect
somewhere a subprocess couldn't access it? The 2nd process that gets runs works like:
Copy code
command = [sys.executable, "-m", "prefect.engine"]
j
Oh sorry that was misleading. We maintain deps with hatch but install them in the container with pip
j
just into the system python within the container?
j
ya
j
darn, theres goes that theory
j
Another piece of debugging info: I can run the flow directly by calling it directly with python
j
and i'm assuming your doing like 
from prefect import ...
at the top of that file?
j
yep
j
what about:
Copy code
python -c "from prefect import engine"
j
Copy code
cannot import name 'engine' from 'prefect'
j
it is possible you have another file that is named 
prefect
or 
engine
?
j
I have a folder at the base of my project called prefect... would that do it?!
j
AAAAHH that might be it. Python is not so great about 2 things with the same name
can you try renaming it?
j
yes, omg...
🙌 1
That was it!!!!
🙌 1
holy moly
Jake, thank you so much!
j
No problem! glad we were able to get to the bottom of it 😄
9 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by