https://prefect.io logo
Join Slack
Powered by
Is there a way to ensure that a new definition of ...
# ask-community
m
Is there a way to ensure that a new definition of a Task or Flow overwrites an old one? I'm using Jupyter to tinker with a Task, and if I make a tweak and re-run, it gives me this and the new behavior doesn't happen.
Copy code
UserWarning: A task named 'download_image' and defined at '/tmp/ipykernel_409/4031849273.py:14' conflicts with another task. Consider specifying a unique `name` parameter in the task definition:
I've been using variations on this to get around it:
Copy code
@task(
    name=f'my_task-{datetime.datetime.now().strftime("%Y%m%d%H%M%S%s%f")}',
)
but it's a pretty annoying bit of boilerplate to have to reach for every time.
n
hmm is this a problem you encounter outside of jupyter? i know jupyter sessions keep old objects around, which explains those warnings imo. you could also suppress warnings in a jupyter environment with stdlib 
warnings
m
@Nate Nope, just in Jupyter. But I want them to not happen in Jupyter.
Cuz it's not just a warning, the behavior doesn't change to match the latest one.
n
ahh i guess i was looking at this warning you sent
Copy code
UserWarning: A task named 'download_image' and defined at '/tmp/ipykernel_409/4031849273.py:14' conflicts with another task. Consider specifying a unique `name` parameter in the task definition:
which is bc your local prefect object registry is finding a flow by that name already in memory, which makes sense to me if you don't restart the kernel (flush variables) between flow definitions - is it possible that you're calling a previously defined version of the 
download_image
task from a different jupyter cell which is not the one you're expecting?
m
Exactly, I don't want to restart the the kernel every time I tinker with it.
That's definitely what happened, I copy & paste a cell and then make a modified version a little bit underneath. I want the new one to overwrite the old one (like regular Python function definition)
n
yeah i think the difference here is that the object registry is keeping those tasks around where thats not true for regular python functions - if you're sticking with jupyter, I'm not sure there's a great way to flush defined tasks from the registry without restarting the kernel - i think you could manually reset the registry, but that seems just as manual
m
oof, that's a major regression from Prefect 1 😞
I got some code from Marvin that might do in a pinch, but that's still kinda clunky as a solution.
Marvin's code, btw (which I tested & works!)
Copy code
import datetime
import functools

def unique_name_task(func=None, *, name=None, **kwargs):
    if func is None:
        return functools.partial(unique_name_task, name=name, **kwargs)

    if name is None:
        name = func.__name__
        
    @task(name=f"{name}-{datetime.datetime.now().strftime('%Y%m%d%H%M%S%f')}", **kwargs)
    @functools.wraps(func)
    def wrapper(*args, **inner_kwargs):
        return func(*args, **inner_kwargs)

    return wrapper

@unique_name_task
def my_task():
    # Your task logic here

def unique_name_flow(func=None, *, name=None, **kwargs):
    if func is None:
        return functools.partial(unique_name_flow, name=name, **kwargs)

    if name is None:
        name = func.__name__
        
    @flow(name=f"{name}-{datetime.datetime.now().strftime('%Y%m%d%H%M%S%f')}", **kwargs)
    @functools.wraps(func)
    def wrapper(*args, **inner_kwargs):
        return func(*args, **inner_kwargs)

    return wrapper
n
i saw that - i agree thats a bit clunky/excessive - in my limited experience working in jupyter i just always restart the kernel bc I don't do a lot of dev in there so it never became a big pain, i prefer ipython for interactive scripting. i can check internally to see if anyone has better ideas
m
Awesome, thanks!
Open in Slack
PreviousNext
Powered by