Anyone have experience / recs for doing Prefect <> GCP serverless (any form)? This is part of some citizen journalism web scraping efforts (twitter, fb, ...), so will be needing to vet & recycle IPs, and thus expect nodes to be somewhat short-lived (1min-1mo), and with many native serverless, 15min anyways

Hi @Leo Meyerovich (Graphistry) - welcoming input from the community here if they've used Prefect in this way before, but my recommendation would be to treat your serverless/transient infrastructure as external to Prefect, using Prefect to do things like kick off Google Cloud Functions and maintain the teardown of nodes/reassignment of IP addresses, but not running Prefect Server/Infrastructure in the same transient way. This'll allow you to use Prefect as the task master without needing to worry about scale-up/down/redeployment of your Prefect components and can even let you leverage the stateful engine of Prefect to monitor those tasks.

Yeah if I'm reading that right, we're indeed planning by default something like: -- keep using our core prefect schedulers etc stack -- spin up cloud functions or whatever, have them phone home to register under a tag like 'cloud function' -- users would then opt-in to that tag we're new to serverless and newish to prefect, so specifics around that would be helpful. we originally did a heavy docker stack for prefect agents b/c needed a lot of gpu etc. libs, but for the crawler, looks like we can get away with lightweight python deps, so thinking this will be more about wrangling weird cloud infra config (and afaict prefect doesn't support native plugins for running on Cloud Functions or something similar in GCP?)

Hm, unfortunately I can't give much guidance here (perhaps the community can chime in), but anything you can do with Python, you should be able to do with Prefect. We don't have a stock Cloud Functions task but GCP offers a python library for interacting with Cloud Functions which you should be able to leverage.