Tilak Maddy
12/02/2021, 11:53 AMflow.storage=Local()
to my Hello world flow ? Everything seems to be the sameAnna Geller
prefect register --project p -p flows/local_storage_local_run_implicit.py
will automatically assign a Local storage with the provided local path
flow.storage = Local(path="flows/local_storage_local_run_implicit.py")
It will also assume LocalRun run configuration with your host name label.
flow.run_config = LocalRun(labels=["yourhostname"])
When you then start an agent on the same machine:
prefect agent local start
it will assign your host name label to the agent and this is how it will match it with the flow because the LocalRun used the same label.
The reason for the default hostname label is that only a Local agent on the same machine can run this flow because this specific Local storage path wouldn’t exist on another machine (say an EC2 instance on AWS).
Does it answer your question?Tilak Maddy
12/02/2021, 1:02 PMimport prefect
from prefect import task, Flow
from prefect.storage import GitHub
from prefect.run_configs import LocalRun
@task
def gitbye():
logger = prefect.context.get("logger")
<http://logger.info|logger.info>("Git bye world!")
with Flow("git-bye-flow") as flow:
gitbye()
flow.storage = GitHub(
repo="XXX/test-repo",
path="github_flow.py",
access_token_secret="XXX"
)
flow.run_config = LocalRun(labels=["dev"])
flow.register(project_name="tutorial")
This file exists in my local machine as well as the Github Repo . However when I proceed to trigger a run from prefect cloud, my agent picks it up but then the flow "failed"
Here's what I found
An import error , that I don't know how to fix.
Could you assist me please ?Anna Geller
pip install "prefect[github]"
And then restarting the agent so that the agent now has this package in its environment:
prefect agent local start --label dev --no-hostname-label
should fix the issueTilak Maddy
12/02/2021, 6:47 PMTilak Maddy
12/02/2021, 6:51 PMprefect agent vertex start --label dev --service-account <mailto:XXX@XXX.iam.gserviceaccount.com|XXX@XXX.iam.gserviceaccount.com>
Tilak Maddy
12/02/2021, 6:51 PMAnna Geller
prefect agent vertex start --project prefect-community
Anna Geller
Tilak Maddy
12/02/2021, 7:25 PMTilak Maddy
12/02/2021, 7:26 PMAnna Geller
Tilak Maddy
12/02/2021, 7:29 PMflow.run_config = VertexRun(labels=["dev"])
Tilak Maddy
12/02/2021, 7:30 PMTilak Maddy
12/02/2021, 7:33 PMTilak Maddy
12/02/2021, 7:34 PMraise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.PermissionDenied: 403 Permission 'aiplatform.customJobs.create' denied on resource '<//aiplatform.googleapis.com/projects/edilitics1/locations/us-central1>' (or it may not exist). [reason: "IAM_PERMISSION_DENIED"
domain: "<http://aiplatform.googleapis.com|aiplatform.googleapis.com>"
metadata {
key: "permission"
value: "aiplatform.customJobs.create"
}
metadata {
key: "resource"
value: "projects/edilitics1/locations/us-central1"
}
]
Anna Geller
export GOOGLE_APPLICATION_CREDENTIALS="/Your/path/to/key.json"
pip install prefect --upgrade
pip install "prefect[gcp]" --upgrade
prefect agent vertex start --project prefect-community --label vertex
or for you the label dev instead of Vertex then.Tilak Maddy
12/02/2021, 7:39 PMprefect agent vertex start --project prefect-community --label vertex
Hey the project here refers to the project name in your gcp right ? Anyways I'll try it and get back. Thanks for all the help so farAnna Geller
Tilak Maddy
12/03/2021, 12:16 PMprefect agent vertex start --project prefect-community --label vertex
command running on my local machine forever.
I just want vertex to be running forever as an agent on GCP waiting for cloud to tell it when to trigger a run . Then it should fetch the file from Github and execute the flow.
What do I do ?Tilak Maddy
12/03/2021, 12:45 PMDiagram▾
Anna Geller
Tilak Maddy
12/03/2021, 12:56 PMTilak Maddy
12/03/2021, 8:45 PMbut will work the same way for VertexHey @Anna Geller unfortunately it did not work ! Here is the command I ran
prefect agent vertex install --label dev --service-account <mailto:XXX@XXX.iam.gserviceaccount.com|XXX@XXX.iam.gserviceaccount.com>
Usage: prefect agent vertex [OPTIONS] COMMAND [ARGS]...
Try 'prefect agent vertex -h' for help.
Error: No such command 'install'.
Anna Geller
Tilak Maddy
12/04/2021, 3:19 AMsupervisorctl
Then ironically I saw that the prefect-agent process was in RUNNING
mode. I typed fg prefect-agent
. to foreground the process and everything became normal. But look, I want it to run no matter what. I feel like this is a common issue and i can't seem to find the fix. Help meTilak Maddy
12/04/2021, 8:39 AMAnna Geller