Question: I want to have a number that increases every time I run a deployment and access it for logging to know which files correspond to which deployment run. How to go about it? Using remote storage (GCS) I tried using a pickle but it appears that it doesn't save on the bucket. Maybe use another bucket or manually pull and push the object to the bucket. Figured there should be an easier way with Blocks.

I was trying to explore solutions for our use case.... We have a use case where we want to run our jobs in hybrid agents (#PrefectIsPerfect 🙂 ) So I'm trying to accomplish this in prefect 2.0 knowing that we have a little bit of a runway before the solution hits primetime... The cloud options seem to be well documented so I'm less concerned about it Our on prem infrastructure is primarily Openshift. We could spin up VMs to host Docker agents but, was curious how have people gone about running production loads on-prem

Ran into this bug earlier. The UI tells me to use the flag

--work-queue

but the CLI says no such option exists. Removing the

--work-queue

flagged resolved the issue.

Is Prefect Cloud HIPAA compliant? If so, does it have to be Enterprise plan tier?

A function to check if a block already exists would be helpful

What replaces the old

flatten

call in Prefect 2.0? I have a list of lists after mapping tasks and I used to be able to use

flatten

to coalesce that into a single large list.

Hi, just thought I’d say that the logging configuration in 2.0 is nice. I have a flow where I need to debug a

requests-oauthlib

connection, and now all I need to do is: 1. Copy

logging.yaml

2. Set my logger:

requests_oauthlib:
        level: DEBUG
        handlers: [console]
        propagate: no

3. Set

PREFECT_LOGGING_SETTINGS_PATH=logging.yaml

in my

.env

file. And it works!

What happened to

create_markdown_artifact

in Prefect 2.0? I don’t see markdown or artifacts mentioned in the migration guide.

Is it possibly to dynamically assign tags to task runs based on parameters? I’m using Prefect 1.x and want to do something like the following across multiple flows.

my_task.map(databases)

But I don’t want to overload my databases, so I want to set concurrency limits at the task level to make sure there are only ever

N < limit

tasks running against a given database at one time. I know you can set task tags when I’m creating the task object, but can I do so as the flow is executing against mapped tasks?

Thanks for the new release today! Very exciting. I'm trying to use the new python

Deployment

class. Previously, I defined a

KubernetesJob

infrastructure block in a different python script and used the

.save()

method to instantiate a block in the UI. It looks like this was removed in 2.1... How can I do this now? Moreover, I'm trying to understand what I pass in for the

Deployment

class in the

infrastructure

parameter. With the

storage

parameter, I just load the

storage

block that I previously defined. Can I do the same with the

infrastructure

block? If so, how do I load an existing

KubernetesJob

block? Or, is the new workflow to just input the

KubernetesJob

dictionary directly in your

Deployment

class? If that's the case, how do I tie this to a work queue? In general, I'm a big fan of the

KubernetesJob

block. idc whether it's an actual block or just a python dictionary I pass into the deployment. Either way, it's been extremely helpful to manage compute depending on the deployment. Would love to streamline it with CICD. I know that's in the works and I'm eager to see best practices with this and implement it.

Hi everyone, 👋 I'm participant ongoing data talks/mlops-zoomcamp and we're using prefect for workflow orchestration. I'm aware there new release. Fully readed changelogs and prefect 2.0 guides. But I'm having problem prefect deployment with mlflow artifacts. When I run my training script(with prefect task/flow) artifacts saves without any problem. But when I create deployment and start agent mlflow doesn't saves artifacts. Agent shows my all tasks and flow without any error. Happy new release! 🎉

Hi everyone! Congrats on the release of Prefect Cloud v2 🎉 I have a question about the lifecycle of Prefect Cloud. When is it a time limit (EOL) to use Prefect Cloud v1? I am currently using v1 and would like to move to v2 at some point.

Hi, Not sure if I missed a setting a something in v1, please let me know if that's the case... But, I was wondering if v2 is better (or will be) at managing its database size - in particular in regards to db partitions and retention policies? With about 1,500 flow runs / 100,000 task runs a day, we seem to be accumulating data rather quickly. About 3 months of running on v1 have given us:

+--------------+-------------+------------------+-------------------+
|table_name    |relation_size|indexes_sizes_size|total_relation_size|
+--------------+-------------+------------------+-------------------+
|log           |15 GB        |9060 MB           |24 GB              |
|task_run      |5765 MB      |17 GB             |23 GB              |
|task_run_state|13 GB        |6308 MB           |19 GB              |
|flow_run      |241 MB       |472 MB            |713 MB             |
|flow_run_state|438 MB       |118 MB            |557 MB             |
+--------------+-------------+------------------+-------------------+

I've been searching for a python workflow management and I've found the Prefect website. can I use Prefect 2.0 locally without cloud services for free without any user limitations? Is the paid subscription for using the Prefect cloud?

Morning, quick question does prefect have to be installed where the agent is running from? If so does it not create 2 database instances?. When prefect is installed on host running the webservice the .prefect directory and orion.db instance gets created. And then when installing prefect again on the host from where the flows are being run then another .prefect directory and orion.db instance gets created?

Hi (-: A very quick one. I am running a Local Agent in a Render service:

The agent seems to start successfully and listen to flows on a queue I’ve defined.

the queue name is “work-queue”. I’ve also defined the needed PREFECT_API_KEY and PREFECT_API_URL with my account and workspace in Render’s env.

(if you don’t know Render, it’s an Heroku alternative)

But when I start a flow via Prefect Cloud UI 2.0, the agent won’t receive the flow:

this local agent runs on Render on a web service. Maybe I need to open some ports? or install it on a different type of service in Render? On the other hand if you look at the logs above it dose say “Agent started! Looking for work from queue(s): work-queue...” meaning it might succeeded contacting the Prefect Cloud.

Any help would me much appreciated.

Concerning the latest changes in versions >2.0.4 : previously agents were able to find and place flows into queues by tags. From version > 2.0.4 the doc says queue names are used instead. Specifically here (https://docs.prefect.io/concepts/deployments/) it s said:

A deployment additionally enables you to:
- Assign a work queue name to delegate deployment flow runs to work queues.

But in the deployment section there is no info on how to declare a queue to be associated with a deployment. Could you please clarify?

In future versions it is expected to pass parameters on the command line

prefect deployment run command

? as plain text or reading them from file? thanks in advance

I may have bumped into a new bug: It seems none of my schedules are triggering anymore on 2.1.1, neither ones I've created with the --cron argument nor ones I've created manually in the GUI. The flow runs show up as scheduled and then late. They are never processed by any agents (nothing shows up in their logs) and they do not show up under "upcoming runs" for the work queues. I can run all of my deployments by triggering them manually, it's only the scheduled runes that don't get picked up. Does anyone have working schedules in 2.1.1?

hey people! really need your help 🙏 I am trying to build a

flow

, that should execute some tasks at the same cluster. It means that if the

flow

get a new message, it should use the same workers (cluster) as for previous messages instead of creating new ones. do you have any idea how to solve my case (coz I don't)?

Hey all, I’m currently looking into using Prefect to leverage dbt’s model graph. Ideally, we want to use Prefect’s 2 dynamic flow creation to generate a flow for each node of the dbt graph, however, we’re not being able to launch several flows to run at the same time. We’re definitely missing something, but couldn’t find it in the new docs. Can someone provide some insight? I’m running Prefect locally