Channels
#prefect-community
Title
m
Matt Alhonte
06/27/2023, 1:49 AMIs there a way to ensure that a new definition of a Task or Flow overwrites an old one? I'm using Jupyter to tinker with a Task, and if I make a tweak and re-run, it gives me this and the new behavior doesn't happen.
Copy code
UserWarning: A task named 'download_image' and defined at '/tmp/ipykernel_409/4031849273.py:14' conflicts with another task. Consider specifying a unique `name` parameter in the task definition: Copy code
@task(
name=f'my_task-{datetime.datetime.now().strftime("%Y%m%d%H%M%S%s%f")}',
)n
Nate
06/27/2023, 3:52 AMhmm is this a problem you encounter outside of jupyter? i know jupyter sessions keep old objects around, which explains those warnings imo. you could also suppress warnings in a jupyter environment with stdlib
warningsm
Matt Alhonte
06/27/2023, 8:19 PM@Nate Nope, just in Jupyter. But I want them to not happen in Jupyter.
Cuz it's not just a warning, the behavior doesn't change to match the latest one.
n
Nate
06/27/2023, 8:24 PMahh i guess i was looking at this warning you sent
Copy code
UserWarning: A task named 'download_image' and defined at '/tmp/ipykernel_409/4031849273.py:14' conflicts with another task. Consider specifying a unique `name` parameter in the task definition:download_imagem
Matt Alhonte
06/27/2023, 8:28 PMExactly, I don't want to restart the the kernel every time I tinker with it.
That's definitely what happened, I copy & paste a cell and then make a modified version a little bit underneath. I want the new one to overwrite the old one (like regular Python function definition)
n
Nate
06/27/2023, 8:38 PMyeah i think the difference here is that the object registry is keeping those tasks around where thats not true for regular python functions - if you're sticking with jupyter, I'm not sure there's a great way to flush defined tasks from the registry without restarting the kernel - i think you could manually reset the registry, but that seems just as manual
m
Matt Alhonte
06/27/2023, 8:56 PMoof, that's a major regression from Prefect 1 😞
I got some code from Marvin that might do in a pinch, but that's still kinda clunky as a solution.
Marvin's code, btw (which I tested & works!)
Copy code
import datetime
import functools
def unique_name_task(func=None, *, name=None, **kwargs):
if func is None:
return functools.partial(unique_name_task, name=name, **kwargs)
if name is None:
name = func.__name__
@task(name=f"{name}-{datetime.datetime.now().strftime('%Y%m%d%H%M%S%f')}", **kwargs)
@functools.wraps(func)
def wrapper(*args, **inner_kwargs):
return func(*args, **inner_kwargs)
return wrapper
@unique_name_task
def my_task():
# Your task logic here
def unique_name_flow(func=None, *, name=None, **kwargs):
if func is None:
return functools.partial(unique_name_flow, name=name, **kwargs)
if name is None:
name = func.__name__
@flow(name=f"{name}-{datetime.datetime.now().strftime('%Y%m%d%H%M%S%f')}", **kwargs)
@functools.wraps(func)
def wrapper(*args, **inner_kwargs):
return func(*args, **inner_kwargs)
return wrappern
Nate
06/27/2023, 9:02 PMi saw that - i agree thats a bit clunky/excessive - in my limited experience working in jupyter i just always restart the kernel bc I don't do a lot of dev in there so it never became a big pain, i prefer ipython for interactive scripting. i can check internally to see if anyone has better ideas
m
Matt Alhonte
06/27/2023, 9:03 PMAwesome, thanks!