Ferdinand Biere
07/06/2023, 12:14 PM12:11:31.720 | INFO | Flow run 'tiny-binturong' - Downloading flow code from storage at '/Users/fb/repositories/prefect'
12:11:32.102 | ERROR | Flow run 'tiny-binturong' - Flow could not be retrieved from deployment.
Any input is highly appreciated āļøChristopher Boyd
07/06/2023, 3:36 PMFerdinand Biere
07/06/2023, 4:44 PMpull:
- prefect_aws.deployments.steps.pull_from_s3:
id: pull_code
requires: prefect-aws>=0.3.4
bucket: '{{ push_code.bucket }}'
folder: '{{ push_code.folder }}'
path
parameter in the `build_from_flow`function.
The error persists howeverNate
07/06/2023, 5:42 PMDeployment.build_from_flow
shouldn't be used along with the yaml declarative deployment + prefect deploy
- you should use one or the other
can you share your push step? the push
and pull
steps should fully configure:
⢠where to push
code at deployment time
⢠where the worker should pull
code from at runtime
and prefect deploy
will create the deployment for youFerdinand Biere
07/06/2023, 6:30 PM# Welcome to your prefect.yaml file! You can you this file for storing and managing
# configuration for deploying your flows. We recommend committing this file to source
# control along with your flow code.
# Generic metadata about this project
name: prefect
prefect-version: 2.10.18
# build section allows you to manage and build docker images
build: null
# push section allows you to manage if and how this project is uploaded to remote locations
push:
- prefect_aws.deployments.steps.push_to_s3:
id: push_code
requires: prefect-aws>=0.3.4
bucket: prefect
folder: code
# credentials: "{{ prefect.blocks.minio-credentials.idrivee }}"
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect_aws.deployments.steps.pull_from_s3:
id: pull_code
requires: prefect-aws>=0.3.4
bucket: '{{ push_code.bucket }}'
folder: '{{ push_code.folder }}'
This is my prefect.yaml file.
Sorry for the beginner-level questions. This is my first time getting my feet wet with Prefect š
Nate
07/06/2023, 6:42 PMSorry for the beginner-level questions. This is my first time getting my feet wet with Prefectno sweat - that's what this channel is for š
I have a Kubernetes Cluster for my Agent(s) and want to use S3 for code storage.sweet! i would recommend using the
prefect-worker
helm chart if you're on k8s
in fact, the fact that you're running an agent instead of a worker is likely the source of the error above, since pull
steps tell workers how to fetch code at runtime. agents are the older way of doing things that have historically worked with the whole build_from_flow
or prefect deployment build -sb s3/etc
style of deployment creation. to my knowledge, agents don't act upon pull
steps, so it makes sense to me that it would go looking at the default location (when a pull step is not defined)Marvin
07/06/2023, 6:43 PMprefect agent start -p "work-pool-name"
. The agent then polls a work pool looking for flow runs to execute[^1^][^2^][^3^].
Similarly, workers are lightweight polling services that retrieve scheduled runs from a work pool and execute them. They offer greater control over infrastructure configuration and the ability to route work to specific types of execution environments. Each worker has a type corresponding to the execution environment to which they will submit flow runs. Workers are only able to join work pools that match their type[^4^][^5^]. Workers are recommended if you're just getting started, since they work best with the prefect.yaml
style of deployment creation.
You start a worker with the command prefect worker start -p [work pool name]
. If the work pool does not exist, it will be created if the --type
flag is used[^6^].
So, in essence, workers are strongly-typed, next-gen agents.
[^1^]: [Prefect Tutorial](https://docs.prefect.io/latest/tutorial/orchestration/)
[^2^]: [Vertexai on Prefect 2](https://discourse.prefect.io/t/vertexai-on-prefect-2/2400)
[^3^]: [Prefect Work Pools](https://docs.prefect.io/latest/concepts/work-pools/)
[^4^]: [Prefect Work Pools](https://docs.prefect.io/latest/concepts/work-pools/)
[^5^]: [Prefect Migration Guide](https://docs.prefect.io/latest/guides/migration-guide/)
[^6^]: [Prefect Work Pools](https://docs.prefect.io/latest/concepts/work-pools/)
</stop>Nate
07/06/2023, 6:46 PMFerdinand Biere
07/06/2023, 6:46 PMMarvin
07/06/2023, 8:16 PMhelm repo add prefect <https://prefecthq.github.io/prefect-helm>
helm repo update
2. Create a new namespace in your Kubernetes cluster to deploy the Prefect worker[^1^]:
kubectl create namespace prefect
3. Create a Kubernetes Secret for the API Key[^1^]. You can then install the Prefect worker using the Helm chart with your custom values.yaml file[^1^]. Finally, verify your deployment[^1^].
As for Minio storage, Prefect 2 uses storage blocks to interact with storage services like Minio[^2^]. You'll need to create a storage block for Minio[^3^][^4^]. Most users find it easiest to configure new storage blocks through the Prefect server or Prefect Cloud UI[^3^]. You'll need the storage location (for example, a bucket or container name) and valid authentication details such as access keys or connection strings[^4^].
Remember, the journey of a thousand miles begins with a single step. Or in this case, a single command.
[^1^]: Prefect 2 Kubernetes Setup Guide
[^2^]: Prefect 2 Storage Concepts
[^3^]: Prefect 2 Storage Tutorial
[^4^]: Prefect 2 Infrastructure Concepts
</stop>Ferdinand Biere
07/06/2023, 9:08 PMMarvin
07/06/2023, 9:09 PMkubectl
is available[^1^].
2. Make sure you're using remote storage, as local storage is not supported for Kubernetes[^1^].
3. Verify that you have a Prefect server or Prefect Cloud API endpoint set in your agent's configuration[^1^].
4. Check that your worker is connecting to the proper account, workspace, and work pool[^2^].
If you're still having trouble, you might want to consult the Prefect community or support for more specific guidance.
Remember, even in the vastness of the universe, no error is truly insurmountable.
[^1^]: Prefect Infrastructure Concepts
[^2^]: Prefect Helm Worker Guide
</stop>Nate
07/07/2023, 12:34 AMMarvin
07/07/2023, 12:34 AM