Hello all. Does anyone have any experience with or resources for using Prefect with GCP serverless? (cloud functions/cloud run) We are currently running a lot of our workflows using GCP serverless (via cloud functions or cloud run); and we have been using pub/sub queues as an orchestration tool to communicate between tasks. The pub/sub 10 minute message acknowledgement limit is starting to become a problem as some our tasks are taking longer than that to complete, so we wish to look into alternative tools.

This architecture sounds pretty complicated. Why do you need cross-task communication? It feels like it defeats the purpose of serverless a bit?

We have some preprocessing workflows which have been broken down into multiple tasks, each requiring vastly different resources. These individual tasks have so far been small enough that each can be run via a cloud function or cloud run instance. The way it works at the moment, and we have found it to be working well for our purpose is: Workflow 1 is broken down into three tasks, and triggered when a file is written to a cloud bucket. The trigger kicks off task A in this workflow, which writes a message to a pub/sub queue when completed. Task B subscribes to the queue and is started whenever a new message is available in the queue. It then writes a completion message to a different pub/sub queue when completed. Task C is then run when triggered by Task B's message to the queue. From my understanding of Prefect, is that this type of workflows is something it can help with?

Why do we have to poll? Instead of

with Flow(...) as flow:
    a = task_a()
    task_b(upstream_tasks=[a])