https://prefect.io logo
Title
p

philip

06/05/2020, 1:58 AM
How can i use prefect server to access google cloud?
n

nicholas

06/05/2020, 1:59 AM
Hi @philip - can you expand on what you mean? Within a flow or deploying Prefect server or something else?
p

philip

06/05/2020, 2:00 AM
task = ShellTask()
with Flow("to_bigquery") as flow:
command = task(command='gsutil cp .... .....)
when I use
flow.run()
it can success. But when i run on Prefect Server it have an error
401 Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.
n

nicholas

06/05/2020, 2:04 AM
Ah I see @philip, thanks for clarifying 🙂 What seems to be happening here is that your local machine is authorized with
GCP
, which is why
flow.run
would succeed - it's running that in an authorized context. Where is your agent running? Your flow's runtime context doesn't seem to be authorized.
p

philip

06/05/2020, 2:09 AM
I find the problem. My agent start in the python venv, i will have the error. When I run in the original python env it can success. Thanks for your help!!
n

nicholas

06/05/2020, 2:10 AM
That makes sense! Glad I could help 😄
👍 1
p

philip

06/05/2020, 2:13 AM
But when i use `nohup`to run agent, i will get the same error😅
n

nicholas

06/05/2020, 2:15 AM
But only if nohup is installed in an environment, right?
Otherwise nohup should take on the context correctly, i believe
p

philip

06/05/2020, 2:25 AM
Excuse me, can you expand more detail?
n

nicholas

06/05/2020, 2:45 AM
Ah ok @philip you're correct that starting your agent in the python environment would cause it to lose some context including it seems the gcloud auth context.
In which case I'd recommend you explicitly authorize the context of the task you're running, by including authorization in the shell task maybe (or better by getting credentials upstream and passing them to the tasks that'll need them)
p

philip

06/05/2020, 2:51 AM
My agent is start at the shell, and i already authorization in the shell. But when i use nohup start the agent in the shell it still have the error. Only use prefect agent start in shell it can run success.
n

nicholas

06/05/2020, 3:07 AM
You should authorize the
ShellTask
or use a
GCP
service account credentials instead
p

philip

06/05/2020, 3:09 AM
Got it. Thank you!
😄 1