Emma Willemsma

    Emma Willemsma

    1 year ago
    Does anyone know if there's a way to set up workflow dependency imports so that they're only imported during flow runs? Say I have something like this:
    from prefect import task, Flow
    from prefect.environments.storage import Docker
    import boto3
    
    @task(log_stdout=True)
    def use_boto3():
        print('Using boto3 version {}'.format(boto3.__version__))
    
    with Flow('Sample Flow') as flow:
        use_boto3()
    
    storage = Docker(python_dependencies=['boto3'])
    storage.add_flow(flow)
    storage.build()
    flow.register(project_name='Dev', build=False)
    I would like to be able to run this to register my flow without having to have boto3 installed (since it will only be used during the flow run anyway)
    I want to be able to specify different deps for different flows, and the python_dependencies parameter on the storage object seems like a nice easy way to do that. However, I then also need to maintain an environment with all of the deps for all of the workflows just so that I can register the flows, which is less nice
    Dylan

    Dylan

    1 year ago
    @Emma Willemsma this is definitely an interesting idea
    I don’t think it’s possible now but I’ll pass along this feedback to the team
    Emma Willemsma

    Emma Willemsma

    1 year ago
    Cool thanks!
    Dylan

    Dylan

    1 year ago
    👍
    @Emma Willemsma I stand corrected
    @task(log_stdout=True)
    def use_boto3():
        import boto3
        print('Using boto3 version {}'.format(boto3.__version__))
    That should work
    Meaning, if you declare you imports on a per-task basis, you don’t need to include them in your registration environment
    as a matter of best practice you should include the import in each task that needs boto3
    Emma Willemsma

    Emma Willemsma

    1 year ago
    Oh I see. Are there any examples like this in the docs?
    Dylan

    Dylan

    1 year ago
    I think some of our logging examples used to have this
    I’m in a meeting at the moment but I’ll get back to you as soon as I can
    Emma Willemsma

    Emma Willemsma

    1 year ago
    Awesome, thanks!