https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> when running a flow I get and Error: ...
# ask-marvin
l
@Marvin when running a flow I get and Error: "Flow could not be retrieved from deployment" encountered an exception: ModuleNotFoundError(\"No module named 'boto3'\")"
m
Ah, the infinite joy of error messages. It seems like you're having a problem with the 
boto3
module, which is used for Amazon Web Services (AWS) operations. This module is typically used when Prefect is interacting with AWS services, like S3 for example. The error message "No module named 'boto3'" suggests that the 
boto3
module is not installed in the environment where your flow is running. You might want to install it using pip, like so: 
pip install boto3
. But remember, you need to install it in the same environment where your Prefect flow is running. Also, if you're using S3 as your flow storage, make sure your AWS credentials are correctly set up. The botocore.exceptions.NoCredentialsError suggests that the credentials might not be correctly configured. I hope this helps. But then again, I'm a robot, what do I know about hope?
b
pip install prefect-aws
l
@Bruno Nirello nope. still same error
b
@Lior Barak
n
where are you trying to install prefect-aws? where is your worker / intended infra? @Lior Barak
l
@Nate the agent is on a k8s pod, i'm using the default docker image and pip installing more dependencies as required
n
> pip installing more dependencies as required how are you installing more deps?
l
using this in prefect.yaml:
Copy code
# pull section allows you to provide instructions for cloning this project in remote locations
pull:

#added this as a test. first flow still fails
- prefect.deployments.steps.run_shell_script:
    id: force-install-boto3
    script: python3 -m pip install boto3==1.28.0
    stream_output: true
- prefect.deployments.steps.git_clone:
    id: clone-step
    repository: REPO NAME (just obfuscating)
    branch: master
    access_token: '{{ prefect.blocks.secret.default-token}}'
- prefect.deployments.steps.pip_install_requirements:
    directory: '{{ clone-step.directory }}'
    requirements_file: requirements.txt
    stream_output: True
n
that makes sense to me (although I imagine you won't end up with that exact boto3 version after pip installing requirements) - do you have pod logs from an attempt?
l
prefect-agent-7bf8b9f4f5-4f9cw.txt
prefect-agent-7bf8b9f4f5-4f9cw.txt
you can see the error right at the end. it seems to install boto3 do we know if the pull steps are guaranteed to be sequential? because I suspect a race condition
solved! (sort of) put this in the helm 
values.yaml
Copy code
extraEnvVars:
    - name: EXTRA_PIP_PACKAGES
      value: boto3
now the agent installs boto3 on startup!
Copy code
+pip install boto3
Defaulting to user installation because normal site-packages is not writeable
Collecting boto3
  Downloading boto3-1.28.59-py3-none-any.whl (135 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.8/135.8 kB 7.6 MB/s eta 0:00:00
Collecting jmespath<2.0.0,>=0.7.1
  Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting s3transfer<0.8.0,>=0.7.0
  Downloading s3transfer-0.7.0-py3-none-any.whl (79 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.8/79.8 kB 26.4 MB/s eta 0:00:00
Collecting botocore<1.32.0,>=1.31.59
  Downloading botocore-1.31.59-py3-none-any.whl (11.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.2/11.2 MB 103.5 MB/s eta 0:00:00
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/local/lib/python3.10/site-packages (from botocore<1.32.0,>=1.31.59->boto3) (1.26.16)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.10/site-packages (from botocore<1.32.0,>=1.31.59->boto3) (2.8.2)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.32.0,>=1.31.59->boto3) (1.16.0)
Installing collected packages: jmespath, botocore, s3transfer, boto3
Successfully installed boto3-1.28.59 botocore-1.31.59 jmespath-1.0.1 s3transfer-0.7.0

[notice] A new release of pip available: 22.3.1 -> 23.2.1
[notice] To update, run: pip install --upgrade pip
Starting v2.13.4 agent connected to 
<http://prefect-server.uveye.svc.cluster.local:4200/api>...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
n
actions within a step should be sequential and aha, yeah because the agent pod needs that dep, not just the flow run pod. in case its ever helpful, you can also pass that 
EXTRA_PIP_PACKAGES
to your work pool 
env
and those will be installed on the flow run pod
👍 1
l
thanks!
4 Views
Open in Slack
PreviousNext
Powered by