https://prefect.io
Join Slack
How are people deploying deployments to GCS with GitHub Actions? My workflow authenticates just fine...
# ask-community
k
How are people deploying deployments to GCS with GitHub Actions? My workflow authenticates just fine and even runs the block registration without any problems but when I check the bucket, the code is not copied. This is what I'm running from GitHub Actions:
Copy code
prefect block register -f blocks/gcp.py
          for f in deployments/*.py; do
            python "$f";
          done
s
Do you need to upload the file to GCS before you register it? https://github.com/google-github-actions/upload-cloud-storage
k
Hm I didn't think so, because when I run that command from 
docker compose exec
on the machine hosting Prefect, then it actually copies it.
r
what does your deploy look like
k
The Deployment?
here's an example:
Copy code
if __name__ == "__main__":
    deployment = build_deployment_from_flow(
        flow=log,
        name="run",
        work_queue_name="work",
        path=str(Path().resolve()),
        entrypoint="deployments/log.py:log",
    )
    deployment.apply()
and build_deployment_from_flow is a util I wrote:
Copy code
def build_deployment_from_flow(
    flow: str, name: str, work_queue_name: str, path: str, entrypoint: str
):
    args = {
        "flow": flow,
        "name": name,
        "work_queue_name": work_queue_name,
    }

    if os.getenv("DEPLOY_ENV") == "prod":
        print("using GCS Block")
        return Deployment.build_from_flow(
            **args,
            path=path + "/deployments",
            storage=GCS.load("flows"),
            entrypoint=entrypoint.split("/")[-1]
        )

    return Deployment.build_from_flow(
        **args,
        **{
            "path": path,
            "entrypoint": entrypoint,
        }
    )
r
why are you registering a block versus simply creating a gcs storage block
k
What do you mean by creating a GCS Storage Block? I'm doing that too in blocks/gcp.py:
Copy code
prefect_flows = GCS(bucket_path="some-temp/prefect-flows/")
    prefect_flows.save("flows", overwrite=True)
r
Copy code
prefect block register -f blocks/gcp.py
register is for new block types
k
Oh I see
I was just following some GitHub repo linked from the Prefect recipes repo
r
try just python blocks/gcp.py
k
cool running 
python blocks/gcp.py
seems to work fine.
r
then see
took me ages to see that, late where I am
k
ah thanks a lot
👍 1
weird, it is now getting registered on Prefect Cloud but the code is still not being copied to the bucket 😕
r
sure DEPLOY_ENV is prod?
k
yup
r
how is it authing gcs
k
GKE workload identity, it works just fine. i have a separate step in the workflow that lists the bucket and adds an empty file to it as a sanity check
r
no infrastructure block for the deploy?
k
nope
r
try
Copy code
deployment.apply(upload=True)
I always have my apply in build_from_flow so it auto uploads
k
I always have my apply in build_from_flow so it auto uploads
Oh, what does this mean? Is this just 
deployment.apply(upload=True)
you mentioned before?
r
yeah
k
oh that's useful, thanks
if you follow this approach, does this mean that you don't have a CI/CD then?
still kind of confused by that whole flow with prefect since there's a bunch of outdated stuff from last year
r
we use gh actions
k
right, but from github actions, do you run the deployments with something like: 
python deployments/flow_with_deployment.py
then?
r
we iterate around all project folders - deploying each project (if changed)
k
oh thanks, i have something like that already:
Copy code
for f in deployments/*.py; do
            python "$f";
          done
r
use it with changed files so we dont deploy everything everytime
k
that makes sense
r
k
thanks this is super helpful!
r
no worries, good luck with it all
k
thanks
4 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by