where can I find some examples of how to pass required flow parameters for prefect v1? when i go to register a flow with required parameters, I get

AttributeError: 'Parameter' object ...

. Why is the flow trying to run when I register it?

Has Agent monitoring been implemented? I just want to be able to check the status of an agent that is running in docker. Every now and then I get a 500 Internal Server Error and agent goes down.

Hi, I’ve been working through the best workflow for setting up Prefect. I’m currently using Prefect Cloud with an agent running on google compute and a GCP storage block. Any tips on how to manage the code base for flows and what gets deployed to Prefect Cloud? If i’m using a repo to version the flows, how do I ensure that prefect cloud and the gcp storage block are kept in sync? Currently i’m creating a deployment manually from the CLI which doesn’t seem like the best approach.

Hello everyone [5:00 PM] Am new to prefect, please which tutorial can I follow step by step on how to orchestrate dbt using prefect? Am presently using Fivetran, Bigquery and DBT but need Prefect to automate DBT

Hiii!!! I’m having a weird situation dont know anybody else has met it before. We are using Prefect 1.0. And I realized that if we dont reschedule the flow that has a new version, the flow will run its old code (although in UI it says its the newest version)

Hello, What is the best practice when establishing prefect on the cloud? Should it run in a cloud instance or in a docker container in a cloud instance?

Hi. I am new to Prefect 2.0. I need some assistance from the community here. I am deploying some pipelines that is scheduled to run on a daily basis. How do I pass in the runtime

date

into my flow as I would need to use that as a parameter for my pipeline? E.g runs on Wed 19Oct 2022 23:00. the date parameter

19Oct2022

is passed a parameter into the flow and will be used in the ETL script

It’s been a while since I’ve used Prefect - I knew a decent amount about deploying V1.0 server with Kubernetes, but it looks like there’s tons of changes in 2.0. These were some really good features I liked, and I want to see if they are available in V2, still: • keeping different flows in different repositories (i.e.

prefect register

as part of a CI was awesome) so that we don’t have to worry about competing dev environments. ◦ I think maybe what we’d want to do now for this is to use S3 storage, since it looks like git storage is only for public repos right now. • Different docker images for diferent flows: ◦ It looks like in V2, Agents are what the executors (?) in V1 were. Would using different images for different flows mean that we needed to have different agents available?

does anyone any examples of using the new

prefect-gcp

CloudRunJob

? the docs are pretty barebones compared to say the Azure equivalent. for me

CloudRunJob.run()

is just hanging infinitely with no output but it's hard to say where I went wrong without an example to go off of.

I am on Prefect 2.6.4 and have a question about "immediate scheduling". I have many tasks that run on a schedule, but now I have a need for a more event-driven flow run. Specifically, I need to immediately schedule a flow run from a script, but I need the call from the script to return immediately (rather than holding up the script and waiting for the flow to complete). I'm sure the answer is simple, but I'm not familiar with this pattern in Prefect 2 and can't find a reference to it in the docs or API. Thanks for the help! :-)

hey prefect community. i am new on a team that's using prefect. my mandate is to make sure we're making the most of the framework. our codebase is almost entirely OOP, and we wrap all of our flow logic in just one task for each flow. i dont think it would be wise for us to shift the paradigm we're programming in completely, but i do want us to leverage the task decorator and observability we get by writing functional code. 1. why does prefect force functional > OOP? Is this the norm for ETL, outside of prefect as well? 2. what benefits do we gain by going functional? 3. when is OOP appropriate? our extract logic is a web scraping so we have a lot of helper functions that share the web driver. also, a web driver instance isn't serializable. 4. what else should i be thinking about here?

Hello, I want to run a flow (Prefect 2.x) in a conda environment. Should I activate the environment via the deployment yaml (as described here: https://discourse.prefect.io/t/how-to-run-my-deployment-in-a-virtual-environment-such-as-conda/1423) or should/could I use a Process block for that? What would you recommend?

i'm a new user i want to save the flows to the database PostgreSQL, I added the connection and the prefect tables was added to my database, I have a record in table configuration , But the flows and the logs not inserted to my PostgreSQL database, i don’t know what i’m missing, can someone help me please?

@terrence has joined the channel

When is it best to post in slack vs discourse with a question?

Hi Prefect community! I have a question about orchestrating workflows using Prefect when there are steps within the workflow that have different infrastructure requirements. For example if I have a workflow I would like to orchestrate using Prefect that has these steps

(DBT job on Snowflake) -> (ML training job).

It would be great to provision a small Fargate task with a Docker image with just the DBT project for the first one, and then provision an ECS task with significantly larger resources and a different Docker image for the ML training job. Is there a way to do this by creating dependencies between Prefect deployments or by specifying infrastructure at the task or sub-flow level?

What are people’s production deployment strategies for Prefect flows currently? Do people make use of Kubernetes/Azure Container Instannces etc. or also just VM’s that run everything? We are currently struggling quite a bit, also regarding whether we should be versioning YAML files, and how. Because currently we version CLI commands in a Makefile that get run by CI/CD

Is the best practice to create a block for reading files from GCS or using

prefect_gcp.cloud_storage

as described here https://prefecthq.github.io/prefect-gcp/. I'm tryin to provide some direction to our data engineers

Is there a best practice for something like listeners in Prefect 2? Essentially looking to trigger a run when a file exists on an sftp.

Tip of the day: You can add

wait_for

to subflows so that they can wait for upstream tasks. In the example below,

subflow

didn’t run because the

fails

task failed. This feature is available in Prefect 2.6.6.

Tip of the day: You can now read Prefect documentation in dark mode! 🌙

Are you trying to wrap your head around a deployment (i.e, what it does, and how to use it)? In this series of short videos, you will get some basic understanding of deployment: • What is a deployment • Create a deployment • Run a deployment

Does anyone have a best-practice work-around to get dask distributed logs collected and displayed? 😇

Prefect now supports colored log levels in the 2.6.7 version. 🌈 Test code:

import asyncio
from prefect import flow, get_run_logger

@flow
async def test_async_flow():
    await asyncio.sleep(3)
    logger = get_run_logger()
    logger.debug("a debug")
    <http://logger.info|logger.info>("my info")
    logger.warning("some warning")
    logger.critical("super critical")
    logger.error("error!")
    print('all done')

@flow
def test_sync_flow():
    test_async_flow()

test_sync_flow()

View more here. Kudos to @Andrew Huang for this cool feature.

Tips of the day: In Prefect 2.6.7, you can add timeout to tasks 🕐 . If

timeout_seconds=n

and the task takes longer than

n

seconds to execute, Prefect will raise a TimeOut error. View more here. Kudos to @Peyton Runyan for this feature.

hey all, in prefect 1 is it better practice to pass a db connection as a parameter into a task, or to open a new session within each task? cc @Javier Ochoa

Hi, can you specify permission / rules on a flow when you deploy it?

Tips of the week Prefect logger allows you to leverage Rich to insert their own custom color / styling. Here is an example:

from prefect import flow, get_run_logger

@flow
def cow_say():
    logger = get_run_logger()
    <http://logger.info|logger.info>("I'm in [red]danger[/red]!")

if __name__ == "__main__":
    cow_say()

View more styles in Rich’s documentation.

Is anyone else this afternoon experiencing agent crashes? I've tried setting my workspace again, but I keep receiving this same error: prefect.exceptions.PrefectHTTPStatusError: Server error '500 Internal Server Error' for URL 'https://api.prefect.cloud/api/accounts/4e02abe6-e67a-4ffa-b8f9-258b7194df4e/workspaces/140a070a-5c2c-4de9-9bf6-33d62e675a65/flows/' Response: {'detail': {'exception_message': 'Internal Server Error'}} For more information check: https://httpstatuses.com/500

hey all, what are some best practices for debugging flow of flows in prefect 1 UI? tracing failure from parent to child flows are is arduous. also - does this get better with prefect 2 ui?