Tim Galvin
06/22/2023, 11:47 AM@task decorated function blocking when calling it as is, e.g.
@task
def square_num(a):
return a**2
@flow
def main_flow():
squares = [square_num(i) for i in range(4)]
here each time square_num is called in the loop, it is not a PrefectFuture that is stored, it is the evaluated number. To get the behaviour that is similar to the @dask.delayed type decorated function, the .submit method should be used.
I ask this question after mocking up a port of a dask workflow over to prefect. If the __call__ of a Task instance behaved similar to its submit method, the conversion would basically be as simple as replacing @delayed with @task and calling it a day, I mocked up a extra Task keyword argument in my own prefect fork called block_on_call, and could tweak this __call__ vs .submit() behaviour on a per @task basis. Maybe it is because of my previous dask exposure, but this kind of feels more natural to me, and kind of makes this toy porting problem pretty effortless.
Is it a case that the current Task.__call__ behaviour is to support a DAG-less type use case/style? Would there be much interest in a pull request of the above make sense/be useful, or am I missing the larger intent here? Does my rambling make sense?Ryan Peden
06/22/2023, 7:26 PM@task decorator.Tim Galvin
06/23/2023, 1:47 AM@task to a function and call the function as it is written without prefect involved? The net effect is the same, modulo the monitoring on the prefect server. My initial feeling would have been the concurrent processing that prefect provides would be more valued than the monitoring in a blocking type mode.
But I say all this as one user - and I really hope that I am not a painful one!
My little patch in my own test environment can revert this behaviour on a per-task basis, with the default behaviour being what is currently implement, i.e. The old behaviour is only used when @task(block_on_call=False) which is not the default. I would be happy to submit a pull request for review if it makes senseTim Galvin
06/23/2023, 5:37 AM