https://prefect.io logo
Channels
Powered by
f

Ferdinand Biere

07/06/2023, 12:14 PM
Hi guys, this is probably a very stupid question, but as I have exhausted all the articles and documentation I could find I am coming here: When I build my deployment via the CLI (or a script for that matter) it gets loaded into the root of my S3 bucket, though on the K8s Agent it tries to load it with the reference to my local file system. (See the logs below) 
12:11:31.720 | INFO    | Flow run 'tiny-binturong' - Downloading flow code from storage at '/Users/fb/repositories/prefect'
12:11:32.102 | ERROR   | Flow run 'tiny-binturong' - Flow could not be retrieved from deployment.
Any input is highly appreciated ā˜€ļø
c

Christopher Boyd

07/06/2023, 3:36 PM
what does your pull step look like
in prefect.yaml
seems like you are just pointing to a local flow when you are deploying, and not pulling it from s3?
f

Ferdinand Biere

07/06/2023, 4:44 PM
Copy code
pull:
- prefect_aws.deployments.steps.pull_from_s3:
    id: pull_code
    requires: prefect-aws>=0.3.4
    bucket: '{{ push_code.bucket }}'
    folder: '{{ push_code.folder }}'
@Christopher Boyd I got it to point to the correct path using the 
path
parameter in the `build_from_flow`function. The error persists however
n

Nate

07/06/2023, 5:42 PM
this is a point of confusion we'll be working on making clearer, but at this time 
Deployment.build_from_flow
shouldn't be used along with the yaml declarative deployment + 
prefect deploy
- you should use one or the other can you share your push step? the 
push
and 
pull
steps should fully configure: ā€¢ where to 
push
code at deployment time ā€¢ where the worker should 
pull
code from at runtime and 
prefect deploy
will create the deployment for you
f

Ferdinand Biere

07/06/2023, 6:30 PM
Gotcha. Is there any updated Tutorial I could follow to have this work for me? I have a Kubernetes Cluster for my Agent(s) and want to use S3 for code storage.
Copy code
# Welcome to your prefect.yaml file! You can you this file for storing and managing
# configuration for deploying your flows. We recommend committing this file to source
# control along with your flow code.

# Generic metadata about this project
name: prefect
prefect-version: 2.10.18

# build section allows you to manage and build docker images
build: null

# push section allows you to manage if and how this project is uploaded to remote locations
push:
- prefect_aws.deployments.steps.push_to_s3:
    id: push_code
    requires: prefect-aws>=0.3.4
    bucket: prefect
    folder: code
    # credentials: "{{ prefect.blocks.minio-credentials.idrivee }}"

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect_aws.deployments.steps.pull_from_s3:
    id: pull_code
    requires: prefect-aws>=0.3.4
    bucket: '{{ push_code.bucket }}'
    folder: '{{ push_code.folder }}'
This is my prefect.yaml file. Sorry for the beginner-level questions. This is my first time getting my feet wet with Prefect šŸ˜…
n

Nate

07/06/2023, 6:42 PM
Sorry for the beginner-level questions. This is my first time getting my feet wet with Prefect
no sweat - that's what this channel is for šŸ™‚
I have a Kubernetes Cluster for my Agent(s) and want to use S3 for code storage.
sweet! i would recommend using the 
prefect-worker
helm chart if you're on k8s in fact, the fact that you're running an agent instead of a worker is likely the source of the error above, since 
pull
steps tell workers how to fetch code at runtime. agents are the older way of doing things that have historically worked with the whole 
build_from_flow
or 
prefect deployment build -sb s3/etc
style of deployment creation. to my knowledge, agents don't act upon 
pull
steps, so it makes sense to me that it would go looking at the default location (when a pull step is not defined)
catjam 1
@Marvin can u pls explain the diff between agents and workers?
šŸ™ 2
m

Marvin

07/06/2023, 6:43 PM
Edited by @Nate: Ah, the eternal dance of agents and workers in the Prefect universe. Let's break it down. In Prefect 2, agents are the original bridge between the Prefect orchestration engine and your execution environments. They organize work that can be picked up for execution. You start an agent with the command 
prefect agent start -p "work-pool-name"
. The agent then polls a work pool looking for flow runs to execute[^1^][^2^][^3^]. Similarly, workers are lightweight polling services that retrieve scheduled runs from a work pool and execute them. They offer greater control over infrastructure configuration and the ability to route work to specific types of execution environments. Each worker has a type corresponding to the execution environment to which they will submit flow runs. Workers are only able to join work pools that match their type[^4^][^5^]. Workers are recommended if you're just getting started, since they work best with the 
prefect.yaml
style of deployment creation. You start a worker with the command 
prefect worker start -p [work pool name]
. If the work pool does not exist, it will be created if the 
--type
flag is used[^6^]. So, in essence, workers are strongly-typed, next-gen agents. [^1^]: [Prefect Tutorial](https://docs.prefect.io/latest/tutorial/orchestration/) [^2^]: [Vertexai on Prefect 2](https://discourse.prefect.io/t/vertexai-on-prefect-2/2400) [^3^]: [Prefect Work Pools](https://docs.prefect.io/latest/concepts/work-pools/) [^4^]: [Prefect Work Pools](https://docs.prefect.io/latest/concepts/work-pools/) [^5^]: [Prefect Migration Guide](https://docs.prefect.io/latest/guides/migration-guide/) [^6^]: [Prefect Work Pools](https://docs.prefect.io/latest/concepts/work-pools/) </stop>
n

Nate

07/06/2023, 6:46 PM
marvin got it mostly right, made a couple clarifying edits šŸ™‚
šŸ™Œ 2
f

Ferdinand Biere

07/06/2023, 6:46 PM
Thank you!
@Marvin How do I get started with Prefect 2 using Kubernetes and a Minio storage?
m

Marvin

07/06/2023, 8:16 PM
Starting your journey with Prefect 2, Kubernetes, and Minio storage, are we? Let's dive into the cosmic soup of information. To get started with Prefect 2 and Kubernetes, you'll need to have Helm and Kubernetes CLI (kubectl) installed on your machine[^1^]. Once you have these prerequisites, follow these steps: 1. Add the Prefect Helm repository to your Helm client[^1^]:
Copy code
helm repo add prefect <https://prefecthq.github.io/prefect-helm>
   helm repo update
2. Create a new namespace in your Kubernetes cluster to deploy the Prefect worker[^1^]:
Copy code
kubectl create namespace prefect
3. Create a Kubernetes Secret for the API Key[^1^]. You can then install the Prefect worker using the Helm chart with your custom values.yaml file[^1^]. Finally, verify your deployment[^1^]. As for Minio storage, Prefect 2 uses storage blocks to interact with storage services like Minio[^2^]. You'll need to create a storage block for Minio[^3^][^4^]. Most users find it easiest to configure new storage blocks through the Prefect server or Prefect Cloud UI[^3^]. You'll need the storage location (for example, a bucket or container name) and valid authentication details such as access keys or connection strings[^4^]. Remember, the journey of a thousand miles begins with a single step. Or in this case, a single command. [^1^]: Prefect 2 Kubernetes Setup Guide [^2^]: Prefect 2 Storage Concepts [^3^]: Prefect 2 Storage Tutorial [^4^]: Prefect 2 Infrastructure Concepts </stop>
f

Ferdinand Biere

07/06/2023, 9:08 PM
@Marvin Could you please tell me more about a validation error for KubernetesWorkerJobConfiguration and where to look for more information?
m

Marvin

07/06/2023, 9:09 PM
Ah, the elusive KubernetesWorkerJobConfiguration validation error. It seems to be a bit of a cosmic mystery, as there are no specific mentions of this error in the Prefect documentation or GitHub issues. However, the KubernetesWorkerJobConfiguration is likely related to the configuration of your Kubernetes worker in Prefect[^1^]. It's possible that the validation error is due to some misconfiguration or missing values in your Kubernetes worker setup[^2^]. Here are a few things you might want to check: 1. Ensure that you have properly configured your Kubernetes cluster and that 
kubectl
is available[^1^]. 2. Make sure you're using remote storage, as local storage is not supported for Kubernetes[^1^]. 3. Verify that you have a Prefect server or Prefect Cloud API endpoint set in your agent's configuration[^1^]. 4. Check that your worker is connecting to the proper account, workspace, and work pool[^2^]. If you're still having trouble, you might want to consult the Prefect community or support for more specific guidance. Remember, even in the vastness of the universe, no error is truly insurmountable. [^1^]: Prefect Infrastructure Concepts [^2^]: Prefect Helm Worker Guide </stop>
n

Nate

07/07/2023, 12:34 AM
@Marvin can you find a link to the prefect-kubernetes collection? the docs live there on that
m

Marvin

07/07/2023, 12:34 AM
The Prefect Kubernetes collection documentation can be found at this link: Prefect Kubernetes Collection[^1^]. Remember, the answer to life, the universe, and everything might be 42, but the answer to your Prefect Kubernetes questions is likely in the documentation. [^1^]: Prefect Kubernetes Collection </stop>
Powered by