https://prefect.io logo
Title
n

Nash Taylor

05/22/2022, 4:26 PM
With the recent release of the Cloud Run Jobs feature on Google Cloud, are there any plans to make an Agent out of it? Am I correct in thinking that would be a great fit for prefect flows?
đź’ˇ 1
j

Jeremiah

05/22/2022, 4:28 PM
I was super excited to see long-running jobs on GCR too. A serverless setup for Prefect 2.0 has always been high on my wishlist but we haven’t started work on it yet. I’m hoping to be in a position to make headway later this year.
n

Nash Taylor

05/22/2022, 4:32 PM
That would be awesome. Honestly I kind of chuckled when I saw the Vertex Training agent, because I've been up against the same problem that probably led to that being chosen as an agent. I actually just recently moved our training infrastructure off of that in favour of straight Compute Engine, because I finally figured out how to mimic the one thing that Vertex Training does that I couldn't. (Shutting down the instance when it's done)
Basically, we use a cloud-init script (pydantic typed of course!) to pull the specified docker image, run the job, do any cleanup and logging, and then call the
google.cloud.compute.InstancesClient().delete_unary() method
with its own instance ID from the metadata server
Good lord, the slack formattting of backticks on mobile is horrendous
j

Jeremiah

05/22/2022, 4:35 PM
Slack formatting in general…
n

Nash Taylor

05/22/2022, 4:37 PM
I couldn't even escape the italics on "horrendous" so I had to end the message there