https://prefect.io logo
e

Emma Willemsma

10/08/2020, 8:40 PM
Does anyone know if there's a way to set up workflow dependency imports so that they're only imported during flow runs? Say I have something like this:
Copy code
from prefect import task, Flow
from prefect.environments.storage import Docker
import boto3

@task(log_stdout=True)
def use_boto3():
    print('Using boto3 version {}'.format(boto3.__version__))

with Flow('Sample Flow') as flow:
    use_boto3()

storage = Docker(python_dependencies=['boto3'])
storage.add_flow(flow)
storage.build()
flow.register(project_name='Dev', build=False)
I would like to be able to run this to register my flow without having to have boto3 installed (since it will only be used during the flow run anyway)
I want to be able to specify different deps for different flows, and the python_dependencies parameter on the storage object seems like a nice easy way to do that. However, I then also need to maintain an environment with all of the deps for all of the workflows just so that I can register the flows, which is less nice
d

Dylan

10/08/2020, 8:48 PM
@Emma Willemsma this is definitely an interesting idea
I don’t think it’s possible now but I’ll pass along this feedback to the team
e

Emma Willemsma

10/08/2020, 8:49 PM
Cool thanks!
d

Dylan

10/08/2020, 8:49 PM
👍
@Emma Willemsma I stand corrected
Copy code
@task(log_stdout=True)
def use_boto3():
    import boto3
    print('Using boto3 version {}'.format(boto3.__version__))
👍 1
That should work
Meaning, if you declare you imports on a per-task basis, you don’t need to include them in your registration environment
as a matter of best practice you should include the import in each task that needs boto3
e

Emma Willemsma

10/08/2020, 8:58 PM
Oh I see. Are there any examples like this in the docs?
d

Dylan

10/08/2020, 9:00 PM
I think some of our logging examples used to have this
I’m in a meeting at the moment but I’ll get back to you as soon as I can
e

Emma Willemsma

10/08/2020, 9:00 PM
Awesome, thanks!