Is it possible to return a Socket from a task?

If it’s not serializeable by cloudpickle, you need to turn off

checkpointing

for the task and you can’t use

DaskExecutor

because both will attempt to serialize it.

Sockets are not serializable. The two tasks need to run from the same process. So perhaps the question is - can two tasks run in the same process?

It will be the same process with the default LocalExecutor

with Flow(...) as flow:
    socket = task_a()
    b = task_b(socket)
    c = task_c(socket)

will work as long as

task_a

checkpointing is turned offf

It should also work with the

DockerExecutor

, right? Are Executors per Flow or per Task?

We only have

LocalExecutor

,

LocalDaskExecutor

and

DaskExecutor

. Docker is a Run Configuration. Both Run Configuration and Executor are per Flow. All executors will then run relative to the new container spun up for DockerRun.

Thanks for the help. Considering moving my Airflow project to Prefect since Airflow has some limitations I'm not comfortable with. Mostly the fact that what is being ran is each task individually, rather than the entire flow (which seems to be the case with Prefect).

Yeah, which results in the passing of data between tasks not being first class like your Socket (as far as I’m aware)

I found a pretty hacky way to achieve this with Airflow but I don't like it.

Tell me more 😄. Am curious

I assume you're familiar with Airflow's terminology. I want to create a DAG that runs a local server, does something over that server, and then shuts it down. Given that the local server is a task that cannot exit, I need to somehow dynamically launch the task that does something over the server. How could I do that? I thought about perhaps placing it in a different DAG and then triggering a DAG run, however I'd like to avoid that and keep all of them within the same DAG. My current solution is to run in parallel a sensor that listens for a file which I dynamically create upon launching the server, however this is pretty redundant.

Yeah that makes sense and I can see why it would be hard in Airflow. In Prefect, we have temporary resources. For an Airflow implementation…I think what you have sounds right already, but I don’t know Airflow too well

How come Prefect doesn't have SSH tasks?

I think in general the “Why is there no _ task?” doesn’t have a deeper reason other than noone has written/contributed it. But this is also the first time I’ve seen the request for one.