https://prefect.io logo
Join Slack
Powered by
What's the best way to set up a Prefect agent in G...
# ask-community
k
What's the best way to set up a Prefect agent in Google Cloud? Just a regular VM instance in compute engine?
wait, i don't need a agent if i have a worker? seems i'm pretty confused here šŸ¤”
j
Hey, this might help: https://docs.prefect.io/2.11.4/guides/upgrade-guide-agents-to-workers/#how-to-upgrade-from-agents-to-workers It explains some of the differences. but short version yes, workers are new style agents.
k
hmm i see i see. i'm completely new so i'm not even familiar with agents. i was looking to run small flows with google cloud run and prefect seems to be the right choice, but can't find exact settings for how to do it. i'll take a look at this doc and see how it goes.
thanks for the quick reply!
j
np! If you're brand new I would encourage you to sign up for prefect cloud (it's free) and use a cloud run push pool. You won't need to worry about an agent or worker. IMO easiest way to use cloud run Heres a guide: https://docs.prefect.io/2.11.4/guides/deployment/push-work-pools/?h=push
upvote 2
k
ty! will use this one
hmm i am getting this error:
Copy code
Traceback (most recent call last):
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 255, in coroutine_wrapper
    return call()
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 383, in __call__
    return self.result()
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 283, in result
    return self.future.result(timeout=timeout)
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 169, in result
    return self.__get_result()
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 346, in _run_async
    result = await coro
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/cli/deploy.py", line 257, in deploy
    await _run_single_deploy(
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/cli/deploy.py", line 511, in _run_single_deploy
    await run_steps(build_steps, step_outputs, print_function=app.console.print)
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 152, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect_docker.deployments.steps.build_docker_image
An exception occurred.
it was a local docker settings issue, fixed it
šŸ™Œ 1
it seems like i still have to run this command somewhere it seems?
Copy code
To execute flow runs from this deployment, start a worker in a separate terminal that pulls work from the 'my-cloud-run-pool' work pool:

        $ prefect worker start --pool 'my-cloud-run-pool'
j
No need! That message shouldnā€™t be displayed for push work pools and will be removed with next release
k
ah ok!
hmm iā€™m still seeing this ā€˜unhealthyā€™ status
maybe i wasnā€™t able to create a push work pools?
ah thatā€™s it. i selected regular google cloud run, not google cloud run push
i was able to start a google cloud run job! yay
šŸ™Œ 1
but it seems like the job is failing due to this:
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 124, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 95, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 292, in aresult
    return await asyncio.wrap_future(self.future)
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 316, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 28, in set_working_directory
    os.chdir(directory)
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/prefect'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 395, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/deployments.py", line 207, in load_flow_from_flow_run
    output = await run_steps(deployment.pull_steps)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 152, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.set_working_directory
I guess this is coming from the prefect.yaml file?
Copy code
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.set_working_directory:
    directory: /opt/prefect/prefect
j
Glad you got it running! yes, it looks like there isnā€™t an ā€œopt/prefect/prefectā€ in your image?
k
aah i forgot to put 
COPY . /opt/prefect/
in Dockerfile
makes sense
hmm iā€™m still getting an error
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 879, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1016, in get_code
  File "<frozen importlib._bootstrap_external>", line 1073, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/awr.py'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 395, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/deployments.py", line 222, in load_flow_from_flow_run
    flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/site-packages/prefect/flows.py", line 975, in load_flow_from_entrypoint
    flow = import_object(entrypoint)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 201, in import_object
    module = load_script_as_module(script_path)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 164, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'awr.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
docker run ls /opt/prefect
to the image does say that the file is there though
i tried specifying the docker image resource uri to the push work pool directly and now iā€™m getting a new error,
Copy code
"terminated: Application failed to start: "/usr/local/bin/python": exec format error"
but seems like it was the right thing to do
this happened last time when i forgot to add 
--platform linux/amd64
to 
docker build
iā€™ll try manually re-building
AHA
it worked!
i guess after i do 
prefect deploy
iā€™d have to manually rebuild the docker image with 
--platform linux/amd64
and 
push
it. might be something for the prefect team to fix in the future!
j
You should be able to add that to your 
prefect.yaml
! It's likely just not a default prompt of 
prefect deploy
https://github.com/PrefectHQ/prefect-docker/blob/main/prefect_docker/deployments/steps.py#L136C1-L146
then it should happen on each 
prefect deploy
k
ah ty!
so just add 
platform: linux/amd64
Hmm, we've scheduled runs for the pipeline but the auto-scheduled runs don't start, forever is in pending status. what did we do wrong?
j
do you mind sending me the workspace id and flow run id? (feel free to dm if you would prefer)
k
sure! workspace id is 
a27a3f3c-fc6c-402c-b274-ea79051c82c2
and flow run is 
helpful-dachshund
j
Appreciate it. Will take a look and get back to you. Additionally any info you have from the Cloud Run console would be helpful, if you see a job execution with that name or not etc.
k
So apparently the job was not created on the cloud run side
And the work pool is said to be unhealthy
@Jake Kaplan please let me know if you found anything! would truly appreciate it. thanks thanks
j
Hey apologies for the delayed reply. I wasn't able to discover anything on my end, from what I could see the job submitted? This an odd case that I haven't seen before, where the run is just stuck. Are you able to try your schedule on a shorter interval to see if it runs? I should have mentioned before I recommended but Push Pools are in still beta so there might be a couple edge cases here, but I will continue to investigate
k
sure! i could also try scheduling another test run to see if it works
just a no-op flow
seems like the no-op flow did run automatically!
could it be that the run flow needs to have more ram or cpu?
j
You should see the flow run go from 
PENDING
-> 
RUNNING
at least. If it ran out of CPU or memory you should see a 
CRASHED
state reported. The only thing I have seen similar is when the container fails to start but GCP reports a successful status code? but it sounds like the only thing that changed between the original and the no-op was the flow code itself. Which really shouldn't have impact here
k
Iā€™ll try deploying again and see how it goes!
oh actually now itā€™s starting! but seems like itā€™s carshing
aah itā€™s running into the problem of timeouts! the default timeout is 10 minutes for google cloud run jobs
20 Views
Open in Slack
PreviousNext
Powered by