Andy Dienes
04/11/2023, 5:58 PMray.JobSubmissionClient
. how do I install ray
(or pip packages more generally?) I tried a variety of things:
1. importing prefect.settings
in my flow directly and setting `prefect.settings.EXTRA_PIP_PACKAGES = ["list", "of", "packages"]
2. adding it to the requires
field of push
and pull
in my prefect.yaml
3. adding EXTRA_PIP_PACKAGES: ray
under job_variables
in the deployment.yaml
none of these worked. what is the recommended way to install packages for flows without burning a whole new image every timealex
04/11/2023, 6:29 PMprefecthq/prefect
docker image then you can set the EXTRA_PIP_PACKAGES
environment variable to install additional dependencies. You can set that for a deployment in a project with the env
key under job_variables
, e.g.:
work_pool:
job_variables:
env:
EXTRA_PIP_PACKAGES: ray
Andy Dienes
04/11/2023, 8:16 PMbotocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied
when trying to run a flow. I have absolutely no idea what is causing this since it's the exact same code I ran successfully before. is there some kind of cache it keeps internally of s3 objects it's trying to list out? I don't even see ListObjects
in the list of options for s3 IAM permissions, and in the storage block I made it has only four get/put
-directory/path
aws s3api list-objects --bucket my-bucket --max-items 10
manually from aws cli it returns correctly, and I'm using the same credentials as I put in that blockalex
04/11/2023, 8:25 PMS3
block? Are you using it as part of your flow?Andy Dienes
04/11/2023, 8:27 PMpull
from the prefect.yaml), and with the block as
from prefect.filesystems import S3
s3_block = S3.load("test-block")
but now, it seems that neither pattern works with no change to the definition of the flow or to the configspull:
- prefect_aws.projects.steps.pull_project_from_s3:
requires: prefect-aws>=0.3.0
bucket: qr-space
folder: prefect
# - prefect.projects.steps.set_working_directory:
# directory: flows
I was trying to change / understand the working directory, since I need to upload the working_directory
to my ray cluster so that it can run tasksprefect.yaml
causes the
ListObjects operation: Access Denied
error. that error goes away when I remove the two commented out linesalex
04/11/2023, 8:33 PMprefect-ray
? https://prefecthq.github.io/prefect-ray/Andy Dienes
04/11/2023, 8:36 PMray.JobSubmissionClient
ListObjects
issue (and surprise at comments impacting behavior! lol)alex
04/11/2023, 8:39 PMset_working_directory
and pull_project_from_s3
steps. That way you know what directory you are downloading to by switching to the desired directory and then downloading to it.Andy Dienes
04/11/2023, 8:48 PMalex
04/12/2023, 2:01 PM