hi all -
getting a strange behaviour:
when running a flow on my (type Process) agent I have 2 stages under
pull
stage
1. cloning from git
2. installing requirements
one of my flow requirements is boto3 (and obviously it is in the requirements.txt file)
every first flow run after agent's startup fails with this error:
"Flow could not be retrieved from deployment."
"exc_info":{"type":"ScriptError","message":"Script at 'flows/full_scan.py' encountered an exception: ModuleNotFoundError(\"No module named 'boto3'\")"
and the second flow onwards succeeds
if I install boto3 manually on the agent's terminal it works fine
Copy code
but if I add this step to pull:
- prefect.deployments.steps.run_shell_script:
id: force-install-boto3
script: python3 -m pip install boto3==1.28.0
stream_output: true
the aforementioned behaviour continues
help?
b
Bianca Hoch
10/04/2023, 8:56 PM
Hi Lior!
> very first flow run after agent's startup fails with this error:
> "Flow could not be retrieved from deployment."
>
>
"exc_info":{"type":"ScriptError","message":"Script at 'flows/full_scan.py' encountered an exception: ModuleNotFoundError(\"No module named 'boto3'\")"
>
> and the second flow onwards succeeds
>
Just to clarify, this is with the
run_shell_script
added to your pull step? First flow run fails, then the next runs successfully?
l
Lior Barak
10/05/2023, 7:40 AM
@Bianca Hoch exactly
b
Bianca Hoch
10/05/2023, 4:43 PM
Quick follow up from our call - try using the k8s worker or process worker instead of an agent to see if the behavior changes in regards to the pull step
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.