I have a postgres database that can only be accessed by establishing an ssh tunnel. If possible I would like to establish the tunnel as a task, then use the existing postgres tasks to query the database via the tunnel. However, I believe that a

CreateSshTunnel

task does not work in general, as it is stateful and local - i.e. I don't think it would work when using the Dask executor (for example), as the tunnel task could execute on a different node than the postgres task. So what would a correct way to structure this be?

If I’m understanding this correctly, you should encapsulate all of that logic inside of 1 task. As I understand it, resources created within a task are always local to wherever the task is being run, so that you’d be able to use the tunnel to connect. After you get your results, you can then pass those downstream as required

That's fair, but then I don't get to take advantage of the existing postgres task. But I guess there's no alternative?

That portion I’m not 100% certain on since I’ve never used the existing postgres task. There might be a way to subclass the task and deal with it that way if you want to look into that?

Yeah, I'll look into subclassing it, thanks

@Alex Cano is correct - Prefect has an implicit assumption that each task runs in a completely different environment, so you wouldn’t be able to share resources. That said, if you wanted to guarantee that all of your tasks ran locally and used a

LocalExecutor

, you could pass your tunnel object around. I wouldn’t recommend this unless you had no other choice, however, as it will only work under those controlled settings.

After thinking about it more, subclassing a task to add an orthogonal functionality like this is actually kind of messy, and breaks SRP. Instead I'm working on creating a task-wrapper class that can be subclassed to generically add extra functionality to any task.