Good morning everyone question for y’all. I have a experimental set up where I have: 1. settings.py: Basically metadata in a dictionary 2. pipeline.py: where I define my flow 3. folder tasks: with several task.py (with different names of course) For now settings.py has the metadata of a single experiment including each experiment directory. Hence it is called/imported in each task file as well as the main pipeline file in order to have it as a global variable sharer across files. For example the results folder path is passed to tasks decorators so that each task knows where to save the result of the task and where to look when cacheing (i am using file based cacheing). The approach works perfectly when I just run one experiment, for which I detail its metadata in settings.py manually. However I have been trying to set a battery of experiments where only one or a few values of the metadata change. For that I attempted to set a second settings file (battery_experiments.py). This file contains lists of values for certain arguments in settings.py. Each of these are used in a for loop where I make several calls of the

@flow

in pipeline.py, each time updating settings.py metadata with values from battery_experiments.py. The issue is that even though the global variables within settings change in each iteration (I print them out), it seem like the task decorator keeps taking the first iteration value always. I am guessing that this happens because the decorators are compiled before anything is ran, so they keep the values in settings.py before any update is done. I am not sure how to make the iterative updates in global variables to be reflected in the task decorator input. Does any of you know if this is possible at all? Thanks a lot!

I'm not sure I totally follow, might need to see your code, but tasks are instantiated/defined at import time, so if the settings are being passed to the task decorator, that won't change over the course of the runtime

@Austin Weisgrau I see. Is there a way to pass a list to a

@task

decorator argument at import time and use each value at each iteration of the flow call? I guess I could set the import of the tasks within the loop so that at each iteration the value in the task decorator changes? that may actually work but leaves the code in a weird structure with an import at teh end instead of all at the very beginning?

I don't totally follow but if you paste some code examples for what you're thinking I could try and help

My code is fairly large but I can set a made up example with the logic simplified. Will do that in a second.

I think if there are values that you want to be different at runtime, you'd want to pass them as arguments to the task functions themselves rather than passing them to the task decorator...

values = [1, 2, 3]

@task
def mytask(value):
   ...

@flow
def myflow(value):
    mytask(value)

for value in values:
    myflow(value)

settings.py

metadata ={
   'path':'to_be_defined'
}

task.py

import settings

@task(cache_key_fn=cache_file_based_fn,
      persist_result=True, 
      result_storage=LocalFileSystem(basepath=settings.metadata['path'])
)
def do_task(args):
   ...
   return

ah i see...you could potentially instantiate the task at runtime within the flow and execute it immediately:

def mytask():
   ...

@flow
def myflow():
    task(mytask, settings)(args)

yeah the way you've set it up won't work because of the reason i stated above, the task decorator is evaluated at import time but you're changing the metadata at run time

Isnt that the same than importing the fucntion within the loop?

Oh! You can change the decorator arguments in a task call?

a decorator is evaluated at import time, so if you use a decorator, the value you pass the decorator is going to be wahtever it is when the program first loads

Oh I see

a decorator is equivalent to the following:

from prefect import task

@task
def mytask():
   ...

is the same as:

from prefect import task

def mytask():
   ...

mytask = task(mytask)

What is passed at run time overrides the task decorator?

the difference is that when you use the decorator, the decorator is evaluated immediately at runtime, vs in the second example, you can control when the "decorator" is evaluated

Gotcha

no it doesn't override, you have to EITHER use the decorator OR do the second method i'm describing

Okay!

🤔 not quite, because the first time you import do_task, the decorator will evaluate, and it won't override the import and reevaluate the decorator the second time

Yeah, but inst it importing at every loop iteration?

no, once a module is imported, it won't reimport

I was just looking at that yeah

import importlib
importlib.reload(module)

I'm honestly not sure, I haven't used that method before, I just know it exists

Okay, thanks a lot for all the help!