https://prefect.io logo
Title
k

kasteph

03/08/2023, 10:37 PM
How are people deploying deployments to GCS with GitHub Actions? My workflow authenticates just fine and even runs the block registration without any problems but when I check the bucket, the code is not copied. This is what I'm running from GitHub Actions:
prefect block register -f blocks/gcp.py
          for f in deployments/*.py; do
            python "$f";
          done
s

Sean Williams

03/08/2023, 10:46 PM
Do you need to upload the file to GCS before you register it? https://github.com/google-github-actions/upload-cloud-storage
k

kasteph

03/08/2023, 10:47 PM
Hm I didn't think so, because when I run that command from
docker compose exec
on the machine hosting Prefect, then it actually copies it.
r

redsquare

03/08/2023, 11:03 PM
what does your deploy look like
k

kasteph

03/08/2023, 11:55 PM
The Deployment?
here's an example:
if __name__ == "__main__":
    deployment = build_deployment_from_flow(
        flow=log,
        name="run",
        work_queue_name="work",
        path=str(Path().resolve()),
        entrypoint="deployments/log.py:log",
    )
    deployment.apply()
and build_deployment_from_flow is a util I wrote:
def build_deployment_from_flow(
    flow: str, name: str, work_queue_name: str, path: str, entrypoint: str
):
    args = {
        "flow": flow,
        "name": name,
        "work_queue_name": work_queue_name,
    }

    if os.getenv("DEPLOY_ENV") == "prod":
        print("using GCS Block")
        return Deployment.build_from_flow(
            **args,
            path=path + "/deployments",
            storage=GCS.load("flows"),
            entrypoint=entrypoint.split("/")[-1]
        )

    return Deployment.build_from_flow(
        **args,
        **{
            "path": path,
            "entrypoint": entrypoint,
        }
    )
r

redsquare

03/09/2023, 12:14 AM
why are you registering a block versus simply creating a gcs storage block
k

kasteph

03/09/2023, 12:15 AM
What do you mean by creating a GCS Storage Block? I'm doing that too in blocks/gcp.py:
prefect_flows = GCS(bucket_path="some-temp/prefect-flows/")
    prefect_flows.save("flows", overwrite=True)
r

redsquare

03/09/2023, 12:16 AM
prefect block register -f blocks/gcp.py
register is for new block types
k

kasteph

03/09/2023, 12:16 AM
Oh I see
I was just following some GitHub repo linked from the Prefect recipes repo
r

redsquare

03/09/2023, 12:17 AM
try just python blocks/gcp.py
k

kasteph

03/09/2023, 12:17 AM
cool running
python blocks/gcp.py
seems to work fine.
r

redsquare

03/09/2023, 12:17 AM
then see
took me ages to see that, late where I am
k

kasteph

03/09/2023, 12:18 AM
ah thanks a lot
👍 1
weird, it is now getting registered on Prefect Cloud but the code is still not being copied to the bucket 😕
r

redsquare

03/09/2023, 12:45 AM
sure DEPLOY_ENV is prod?
k

kasteph

03/09/2023, 12:45 AM
yup
r

redsquare

03/09/2023, 12:50 AM
how is it authing gcs
k

kasteph

03/09/2023, 12:51 AM
GKE workload identity, it works just fine. i have a separate step in the workflow that lists the bucket and adds an empty file to it as a sanity check
r

redsquare

03/09/2023, 12:54 AM
no infrastructure block for the deploy?
k

kasteph

03/09/2023, 12:54 AM
nope
r

redsquare

03/09/2023, 12:57 AM
try
deployment.apply(upload=True)
I always have my apply in build_from_flow so it auto uploads
k

kasteph

03/09/2023, 1:00 AM
I always have my apply in build_from_flow so it auto uploads
Oh, what does this mean? Is this just
deployment.apply(upload=True)
you mentioned before?
r

redsquare

03/09/2023, 1:01 AM
yeah
k

kasteph

03/09/2023, 1:04 AM
oh that's useful, thanks
if you follow this approach, does this mean that you don't have a CI/CD then?
still kind of confused by that whole flow with prefect since there's a bunch of outdated stuff from last year
r

redsquare

03/09/2023, 1:05 AM
we use gh actions
k

kasteph

03/09/2023, 1:06 AM
right, but from github actions, do you run the deployments with something like:
python deployments/flow_with_deployment.py
then?
r

redsquare

03/09/2023, 1:07 AM
we iterate around all project folders - deploying each project (if changed)
k

kasteph

03/09/2023, 1:07 AM
oh thanks, i have something like that already:
for f in deployments/*.py; do
            python "$f";
          done
r

redsquare

03/09/2023, 1:08 AM
use it with changed files so we dont deploy everything everytime
k

kasteph

03/09/2023, 1:08 AM
that makes sense
r
k

kasteph

03/09/2023, 1:12 AM
thanks this is super helpful!
r

redsquare

03/09/2023, 1:15 AM
no worries, good luck with it all
k

kasteph

03/09/2023, 1:15 AM
thanks