https://prefect.io logo
Channels
Powered by
k

Kazushi Nagayama

08/18/2023, 2:23 PM
What's the best way to set up a Prefect agent in Google Cloud? Just a regular VM instance in compute engine?
wait, i don't need a agent if i have a worker? seems i'm pretty confused here 🤔
j

Jake Kaplan

08/18/2023, 2:38 PM
Hey, this might help: https://docs.prefect.io/2.11.4/guides/upgrade-guide-agents-to-workers/#how-to-upgrade-from-agents-to-workers It explains some of the differences. but short version yes, workers are new style agents.
k

Kazushi Nagayama

08/18/2023, 2:46 PM
hmm i see i see. i'm completely new so i'm not even familiar with agents. i was looking to run small flows with google cloud run and prefect seems to be the right choice, but can't find exact settings for how to do it. i'll take a look at this doc and see how it goes.
thanks for the quick reply!
j

Jake Kaplan

08/18/2023, 2:49 PM
np! If you're brand new I would encourage you to sign up for prefect cloud (it's free) and use a cloud run push pool. You won't need to worry about an agent or worker. IMO easiest way to use cloud run Heres a guide: https://docs.prefect.io/2.11.4/guides/deployment/push-work-pools/?h=push
upvote 2
k

Kazushi Nagayama

08/19/2023, 4:39 AM
ty! will use this one
hmm i am getting this error:
Copy code
Traceback (most recent call last):
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 255, in coroutine_wrapper
    return call()
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 383, in __call__
    return self.result()
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 283, in result
    return self.future.result(timeout=timeout)
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 169, in result
    return self.__get_result()
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 346, in _run_async
    result = await coro
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/cli/deploy.py", line 257, in deploy
    await _run_single_deploy(
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/cli/deploy.py", line 511, in _run_single_deploy
    await run_steps(build_steps, step_outputs, print_function=app.console.print)
  File "/Users/kazushi/opt/anaconda3/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 152, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect_docker.deployments.steps.build_docker_image
An exception occurred.
it was a local docker settings issue, fixed it
🙌 1
it seems like i still have to run this command somewhere it seems?
Copy code
To execute flow runs from this deployment, start a worker in a separate terminal that pulls work from the 'my-cloud-run-pool' work pool:

        $ prefect worker start --pool 'my-cloud-run-pool'
j

Jake Kaplan

08/19/2023, 3:14 PM
No need! That message shouldn’t be displayed for push work pools and will be removed with next release
k

Kazushi Nagayama

08/19/2023, 3:15 PM
ah ok!
hmm i’m still seeing this ‘unhealthy’ status
maybe i wasn’t able to create a push work pools?
ah that’s it. i selected regular google cloud run, not google cloud run push
i was able to start a google cloud run job! yay
🙌 1
but it seems like the job is failing due to this:
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 124, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 95, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 292, in aresult
    return await asyncio.wrap_future(self.future)
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 316, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 28, in set_working_directory
    os.chdir(directory)
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/prefect'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 395, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/deployments.py", line 207, in load_flow_from_flow_run
    output = await run_steps(deployment.pull_steps)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 152, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.set_working_directory
I guess this is coming from the prefect.yaml file?
Copy code
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.set_working_directory:
    directory: /opt/prefect/prefect
j

Jake Kaplan

08/19/2023, 3:33 PM
Glad you got it running! yes, it looks like there isn’t an “opt/prefect/prefect” in your image?
k

Kazushi Nagayama

08/19/2023, 3:43 PM
aah i forgot to put 
COPY . /opt/prefect/
in Dockerfile
makes sense
hmm i’m still getting an error
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 879, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1016, in get_code
  File "<frozen importlib._bootstrap_external>", line 1073, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/awr.py'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 395, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/deployments.py", line 222, in load_flow_from_flow_run
    flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/site-packages/prefect/flows.py", line 975, in load_flow_from_entrypoint
    flow = import_object(entrypoint)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 201, in import_object
    module = load_script_as_module(script_path)
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 164, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'awr.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
docker run ls /opt/prefect
to the image does say that the file is there though
i tried specifying the docker image resource uri to the push work pool directly and now i’m getting a new error,
Copy code
"terminated: Application failed to start: "/usr/local/bin/python": exec format error"
but seems like it was the right thing to do
this happened last time when i forgot to add 
--platform linux/amd64
to 
docker build
i’ll try manually re-building
AHA
it worked!
i guess after i do 
prefect deploy
i’d have to manually rebuild the docker image with 
--platform linux/amd64
and 
push
it. might be something for the prefect team to fix in the future!
j

Jake Kaplan

08/19/2023, 4:34 PM
You should be able to add that to your 
prefect.yaml
! It's likely just not a default prompt of 
prefect deploy
https://github.com/PrefectHQ/prefect-docker/blob/main/prefect_docker/deployments/steps.py#L136C1-L146
then it should happen on each 
prefect deploy
k

Kazushi Nagayama

08/20/2023, 4:14 PM
ah ty!
so just add 
platform: linux/amd64
Hmm, we've scheduled runs for the pipeline but the auto-scheduled runs don't start, forever is in pending status. what did we do wrong?
j

Jake Kaplan

08/23/2023, 1:19 PM
do you mind sending me the workspace id and flow run id? (feel free to dm if you would prefer)
k

Kazushi Nagayama

08/23/2023, 2:18 PM
sure! workspace id is 
a27a3f3c-fc6c-402c-b274-ea79051c82c2
and flow run is 
helpful-dachshund
j

Jake Kaplan

08/23/2023, 3:01 PM
Appreciate it. Will take a look and get back to you. Additionally any info you have from the Cloud Run console would be helpful, if you see a job execution with that name or not etc.
k

Kazushi Nagayama

08/23/2023, 3:06 PM
So apparently the job was not created on the cloud run side
And the work pool is said to be unhealthy
@Jake Kaplan please let me know if you found anything! would truly appreciate it. thanks thanks
j

Jake Kaplan

08/25/2023, 4:35 PM
Hey apologies for the delayed reply. I wasn't able to discover anything on my end, from what I could see the job submitted? This an odd case that I haven't seen before, where the run is just stuck. Are you able to try your schedule on a shorter interval to see if it runs? I should have mentioned before I recommended but Push Pools are in still beta so there might be a couple edge cases here, but I will continue to investigate
k

Kazushi Nagayama

08/27/2023, 9:09 AM
sure! i could also try scheduling another test run to see if it works
just a no-op flow
seems like the no-op flow did run automatically!
could it be that the run flow needs to have more ram or cpu?
j

Jake Kaplan

08/29/2023, 2:05 PM
You should see the flow run go from 
PENDING
-> 
RUNNING
at least. If it ran out of CPU or memory you should see a 
CRASHED
state reported. The only thing I have seen similar is when the container fails to start but GCP reports a successful status code? but it sounds like the only thing that changed between the original and the no-op was the flow code itself. Which really shouldn't have impact here
k

Kazushi Nagayama

08/30/2023, 4:27 AM
I’ll try deploying again and see how it goes!
oh actually now it’s starting! but seems like it’s carshing
aah it’s running into the problem of timeouts! the default timeout is 10 minutes for google cloud run jobs
Powered by