With the recent release of the Cloud Run Jobs feature on Google Cloud, are there any plans to make a...

Nash Taylor

05/22/2022, 4:26 PM

With the recent release of the Cloud Run Jobs feature on Google Cloud, are there any plans to make an Agent out of it? Am I correct in thinking that would be a great fit for prefect flows?

💡 1

Jeremiah

05/22/2022, 4:28 PM

I was super excited to see long-running jobs on GCR too. A serverless setup for Prefect 2.0 has always been high on my wishlist but we haven’t started work on it yet. I’m hoping to be in a position to make headway later this year.

Nash Taylor

05/22/2022, 4:32 PM

That would be awesome. Honestly I kind of chuckled when I saw the Vertex Training agent, because I've been up against the same problem that probably led to that being chosen as an agent. I actually just recently moved our training infrastructure off of that in favour of straight Compute Engine, because I finally figured out how to mimic the one thing that Vertex Training does that I couldn't. (Shutting down the instance when it's done)

Nash Taylor

05/22/2022, 4:34 PM

Basically, we use a cloud-init script (pydantic typed of course!) to pull the specified docker image, run the job, do any cleanup and logging, and then call the

google.cloud.compute.InstancesClient().delete_unary() method

with its own instance ID from the metadata server

Nash Taylor

05/22/2022, 4:35 PM

Good lord, the slack formattting of backticks on mobile is horrendous

Jeremiah

05/22/2022, 4:35 PM

Slack formatting in general…

Nash Taylor

05/22/2022, 4:37 PM

I couldn't even escape the italics on "horrendous" so I had to end the message there

3 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.