https://prefect.io logo
#prefect-community
Title
# prefect-community
e

egk

04/14/2022, 5:39 PM
Is it possible to return a Socket from a task?
k

Kevin Kho

04/14/2022, 5:40 PM
If it’s not serializeable by cloudpickle, you need to turn off
checkpointing
for the task and you can’t use
DaskExecutor
because both will attempt to serialize it.
e

egk

04/14/2022, 5:42 PM
Sockets are not serializable. The two tasks need to run from the same process. So perhaps the question is - can two tasks run in the same process?
k

Kevin Kho

04/14/2022, 5:49 PM
It will be the same process with the default LocalExecutor
Copy code
with Flow(...) as flow:
    socket = task_a()
    b = task_b(socket)
    c = task_c(socket)
will work as long as
task_a
checkpointing is turned offf
e

egk

04/14/2022, 5:51 PM
It should also work with the
DockerExecutor
, right? Are Executors per Flow or per Task?
k

Kevin Kho

04/14/2022, 5:53 PM
We only have
LocalExecutor
,
LocalDaskExecutor
and
DaskExecutor
. Docker is a Run Configuration. Both Run Configuration and Executor are per Flow. All executors will then run relative to the new container spun up for DockerRun.
e

egk

04/14/2022, 5:54 PM
Thanks for the help. Considering moving my Airflow project to Prefect since Airflow has some limitations I'm not comfortable with. Mostly the fact that what is being ran is each task individually, rather than the entire flow (which seems to be the case with Prefect).
k

Kevin Kho

04/14/2022, 5:56 PM
Yeah, which results in the passing of data between tasks not being first class like your Socket (as far as I’m aware)
e

egk

04/14/2022, 5:59 PM
I found a pretty hacky way to achieve this with Airflow but I don't like it.
k

Kevin Kho

04/14/2022, 6:00 PM
Tell me more 😄. Am curious
e

egk

04/14/2022, 6:15 PM
I assume you're familiar with Airflow's terminology. I want to create a DAG that runs a local server, does something over that server, and then shuts it down. Given that the local server is a task that cannot exit, I need to somehow dynamically launch the task that does something over the server. How could I do that? I thought about perhaps placing it in a different DAG and then triggering a DAG run, however I'd like to avoid that and keep all of them within the same DAG. My current solution is to run in parallel a sensor that listens for a file which I dynamically create upon launching the server, however this is pretty redundant.
Do you understand my goal, or was I not clear enough?
Honestly, I don't need neither Airflow nor Prefect for their "data science" purposes. I just want a managed way to run tasks and workflows.
Do you have any idea how could this be better implemented in Airflow?
k

Kevin Kho

04/14/2022, 6:31 PM
Yeah that makes sense and I can see why it would be hard in Airflow. In Prefect, we have temporary resources. For an Airflow implementation…I think what you have sounds right already, but I don’t know Airflow too well
e

egk

04/14/2022, 6:35 PM
How come Prefect doesn't have SSH tasks?
k

Kevin Kho

04/14/2022, 6:36 PM
I think in general the “Why is there no _ task?” doesn’t have a deeper reason other than noone has written/contributed it. But this is also the first time I’ve seen the request for one.
8 Views