https://prefect.io logo
a

Andy Dienes

04/11/2023, 5:58 PM
I'm trying to have my flow use a
ray.JobSubmissionClient
. how do I install
ray
(or pip packages more generally?) I tried a variety of things: 1. importing
prefect.settings
in my flow directly and setting `prefect.settings.EXTRA_PIP_PACKAGES = ["list", "of", "packages"] 2. adding it to the
requires
field of
push
and
pull
in my
prefect.yaml
3. adding
EXTRA_PIP_PACKAGES: ray
under
job_variables
in the
deployment.yaml
none of these worked. what is the recommended way to install packages for flows without burning a whole new image every time
a

alex

04/11/2023, 6:29 PM
If you’re using the
prefecthq/prefect
docker image then you can set the
EXTRA_PIP_PACKAGES
environment variable to install additional dependencies. You can set that for a deployment in a project with the
env
key under
job_variables
, e.g.:
Copy code
work_pool:
  job_variables:
    env:
      EXTRA_PIP_PACKAGES: ray
a

Andy Dienes

04/11/2023, 8:16 PM
@alex I really appreciate all your help here. I have one more --- I am sometimes getting now
Copy code
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied
when trying to run a flow. I have absolutely no idea what is causing this since it's the exact same code I ran successfully before. is there some kind of cache it keeps internally of s3 objects it's trying to list out? I don't even see
ListObjects
in the list of options for s3 IAM permissions, and in the storage block I made it has only four
get/put
-
directory/path
and when I call
Copy code
aws s3api list-objects --bucket my-bucket --max-items 10
manually from aws cli it returns correctly, and I'm using the same credentials as I put in that block
a

alex

04/11/2023, 8:25 PM
How are you using the
S3
block? Are you using it as part of your flow?
a

Andy Dienes

04/11/2023, 8:27 PM
so, earlier today I had actually run the flow both ways: without referencing the block in the flow at all (it could still
pull
from the prefect.yaml), and with the block as
Copy code
from prefect.filesystems import S3
s3_block = S3.load("test-block")
but now, it seems that neither pattern works with no change to the definition of the flow or to the configs
oh.. ok I found how to replicate the error
Copy code
pull:
    - prefect_aws.projects.steps.pull_project_from_s3:
          requires: prefect-aws>=0.3.0
          bucket: qr-space
          folder: prefect
# - prefect.projects.steps.set_working_directory:
#     directory: flows
I was trying to change / understand the working directory, since I need to upload the
working_directory
to my ray cluster so that it can run tasks
but actually, adding those comments onto the end of the
prefect.yaml
causes the
Copy code
ListObjects operation: Access Denied
error. that error goes away when I remove the two commented out lines
a

alex

04/11/2023, 8:33 PM
Hmm, that’s interesting. If you’re trying to run prefect tasks on a Ray cluster, have you seen our Ray integration
prefect-ray
? https://prefecthq.github.io/prefect-ray/
a

Andy Dienes

04/11/2023, 8:36 PM
I have, it looks snazzy. although my impression is that is more meant to let ray be an engine for prefect tasks --- I would like to keep the number of prefect tasks very small (but the number of ray jobs very high), so I want the ray job to be somewhat opaque to the orchestrator and just use prefect flows to control the
ray.JobSubmissionClient
in any case, I think that's slightly orthogonal; once I have the working directory somewhere I can access in a prefect flow I can send this to the ray cluster. just fighting through this
ListObjects
issue (and surprise at comments impacting behavior! lol)
a

alex

04/11/2023, 8:39 PM
Gocha, one thing I would try is switching the
set_working_directory
and
pull_project_from_s3
steps. That way you know what directory you are downloading to by switching to the desired directory and then downloading to it.
a

Andy Dienes

04/11/2023, 8:48 PM
on second look, it seems like a totally flaky/random failure unrelated to the commented lines lol (thankfully)
I'll go file a gh issue
last one --- it looks like the pods that spin up to run the tasks, either on success or failure, stick around instead of deleting themselves. I imagine this is for observability / logging? but is there a way to configure these to delete as soon as a task finishes. I saw a similar question here https://github.com/PrefectHQ/prefect/issues/5547 but not sure how this works for workers vs for agents
a

alex

04/12/2023, 2:01 PM
On your kubernetes work pool you can set the Finished Job TTL which will tell your cluster to clean up completed jobs after a given number of seconds.
2 Views