I'm trying to have my flow use a

ray.JobSubmissionClient

. how do I install

ray

(or pip packages more generally?) I tried a variety of things: 1. importing

prefect.settings

in my flow directly and setting `prefect.settings.EXTRA_PIP_PACKAGES = ["list", "of", "packages"] 2. adding it to the

requires

field of

push

and

pull

in my

prefect.yaml

3. adding

EXTRA_PIP_PACKAGES: ray

under

job_variables

in the

deployment.yaml

none of these worked. what is the recommended way to install packages for flows without burning a whole new image every time

If you’re using the

prefecthq/prefect

docker image then you can set the

EXTRA_PIP_PACKAGES

environment variable to install additional dependencies. You can set that for a deployment in a project with the

env

key under

job_variables

, e.g.:

work_pool:
  job_variables:
    env:
      EXTRA_PIP_PACKAGES: ray

@alex I really appreciate all your help here. I have one more --- I am sometimes getting now

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

when trying to run a flow. I have absolutely no idea what is causing this since it's the exact same code I ran successfully before. is there some kind of cache it keeps internally of s3 objects it's trying to list out? I don't even see

ListObjects

in the list of options for s3 IAM permissions, and in the storage block I made it has only four

get/put

-

directory/path

How are you using the

S3

block? Are you using it as part of your flow?

so, earlier today I had actually run the flow both ways: without referencing the block in the flow at all (it could still

pull

from the prefect.yaml), and with the block as

from prefect.filesystems import S3
s3_block = S3.load("test-block")

but now, it seems that neither pattern works with no change to the definition of the flow or to the configs

Hmm, that’s interesting. If you’re trying to run prefect tasks on a Ray cluster, have you seen our Ray integration

prefect-ray

? https://prefecthq.github.io/prefect-ray/

I have, it looks snazzy. although my impression is that is more meant to let ray be an engine for prefect tasks --- I would like to keep the number of prefect tasks very small (but the number of ray jobs very high), so I want the ray job to be somewhat opaque to the orchestrator and just use prefect flows to control the

ray.JobSubmissionClient

Gocha, one thing I would try is switching the

set_working_directory

and

pull_project_from_s3

steps. That way you know what directory you are downloading to by switching to the desired directory and then downloading to it.

on second look, it seems like a totally flaky/random failure unrelated to the commented lines lol (thankfully)

On your kubernetes work pool you can set the Finished Job TTL which will tell your cluster to clean up completed jobs after a given number of seconds.