Channels
#prefect-community
Title
e
Emma Willemsma
10/08/2020, 8:40 PMDoes anyone know if there's a way to set up workflow dependency imports so that they're only imported during flow runs? Say I have something like this:
Copy code
from prefect import task, Flow
from prefect.environments.storage import Docker
import boto3
@task(log_stdout=True)
def use_boto3():
print('Using boto3 version {}'.format(boto3.__version__))
with Flow('Sample Flow') as flow:
use_boto3()
storage = Docker(python_dependencies=['boto3'])
storage.add_flow(flow)
storage.build()
flow.register(project_name='Dev', build=False)I want to be able to specify different deps for different flows, and the python_dependencies parameter on the storage object seems like a nice easy way to do that. However, I then also need to maintain an environment with all of the deps for all of the workflows just so that I can register the flows, which is less nice
d
Dylan
10/08/2020, 8:48 PM
@Emma Willemsma this is definitely an interesting idea
I don’t think it’s possible now but I’ll pass along this feedback to the team
e
Emma Willemsma
10/08/2020, 8:49 PMCool thanks!
d
Dylan
10/08/2020, 8:49 PM
👍
@Emma Willemsma I stand corrected
Copy code
@task(log_stdout=True)
def use_boto3():
import boto3
print('Using boto3 version {}'.format(boto3.__version__))👍 1
That should work
Meaning, if you declare you imports on a per-task basis, you don’t need to include them in your registration environment
as a matter of best practice you should include the import in each task that needs boto3
e
Emma Willemsma
10/08/2020, 8:58 PMOh I see. Are there any examples like this in the docs?
d
Dylan
10/08/2020, 9:00 PM
I think some of our logging examples used to have this
I’m in a meeting at the moment but I’ll get back to you as soon as I can
e
Emma Willemsma
10/08/2020, 9:00 PMAwesome, thanks!