Hi, I’m porting a flow from 1.0 to 2.0. My flow needs a couple of secrets. During development I would like to run the flow via a plain old python call and the ephemeral API, with

if __name__ == '__main__': myflow()

. This is without a running Orion server. The code in my flow that creates the Secret block appears to work. However, when I try use my Secret block in my 2.0 flow the prefect client raises a PrefectHTTPStatusError, and says ‘Block document not found’. In Prefect 1.0 I could use

PREFECT__CONTEXT__SECRETS__

to set the values I need, but I don’t see any mention of similar functionality for 2.0.

[Error with Prefect Docker Agent on AWS EC2] Hello, I followed the minitutorial here but did not work and got the following error (see pic) @Jamie Zieziula any idea how can I fix this? 😕

Where can I find up-to-date information about deployments and the file format for the yaml files?

What is the possible reason for a "Failed to create schedule" error message in the prefect cloud 2 when i want to add a schedule to a deployment via the cloud ui?

Hi, I'm having trouble with blocks. There are no blocks in the block catalog! Just says "0 blocks". No filters applied either. UPDATE:

prefect block type ls

does give the block types. UPDATE2: Something is off, when hosting on localhost and doing an ssh tunnel on it it works like a charm. When hosting on

0.0.0.0

the page is empty. UPDATE3: Now it works on the public IP as well ?!? Maybe it needs to start without

--host

the first time

[Prefect 2.0.4 + Prefect Cloud] Hey, I'm struggling with the deployment override api and kubernetes-job infrastructure I've created a generic kubernetes-job infrastructure block with some sane defaults for image etc. but no env variables. If I run this block without overrides, everything is fine and the flow starts as expected.

prefect deployment build r_script_automation.py:r_script_automation --name retention_cohort_analysis_deployment -t k8s -sb gcs/gcs-prefect-stprage -ib kubernetes-job/generic-k8s-job -o cohort_deployment.yaml

However, as soon as I add infrastr. overrides, the flow stays in "pending" state forever

prefect deployment build r_script_automation.py:r_script_automation --name retention_cohort_analysis_deployment -t k8s -sb gcs/gcs-prefect-stprage -ib kubernetes-job/generic-k8s-job -o cohort_deployment.yaml --override image=europe-docker.pkg.dev/vol-at/rm-datateam-repository/r-script-automation:beta-23 --override env.GIT_PYTHON_REFRESH="quiet" --override env.GOOGLE_APPLICATION_CREDENTIALS="/google/.google-secret-key.json" --override env.CHROMIUM_FLAGS="--no-sandbox"

If I manually delete the infra_overrides-block from the deployment-yaml - the flow works again. So somehow this env.xyz - overrides are the problem. Can you point me to my mistake? Edit: I tested overriding the "image" - this works. It really only fails when overriding the environment variables....

Hi, Can someone help me here ? PS C:\Users\Sachin Ranveer\Desktop\Logward\data-ingestion-service> prefect orion start prefect : The term 'prefect' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. At line:1 char:1 + prefect orion start + ~~~~~~~ + CategoryInfo : ObjectNotFound: (prefect:String) [], CommandNotFoundException + FullyQualifiedErrorId : CommandNotFoundException

Prefect

2.0.4

Hi all, I just upgraded from

2.0b8

to

2.0.4

. In

v2.0b8

errors within flows/tasks were caught by Prefect and the flow would continue, now in

2.0.4

they are being raised and crashing my run. How can I get back to the previous behavior? I’ll post my code in the thread

Hi there. I have an Airflow DAG that pulls data from an API. It starts with a sensor that polls the API until the data is ready. I am wondering what would be the best pattern to implement this in 2.0 Instead of having a long running process that polls, I was thinking I could have a flow that checks for data and reschedules itself up to N times if the data is not ready. When it finds data, it continues the process and does not reschedule itself. Could someone help with a code sample to do this in 2.0? Can you think of a better approach? Thanks!

Hey, I am executing a flow run from a deployment via client. Everything runs locally, with default storage configuration. After the flow finished succesfully, I request the state and try to access the result but

state.result()

only returns

_Result(key='82ecff4d3a454e77b08b5ce93e1dd418', filesystem_document_id=UUID('af2119c0-c631-45ed-af6a-0cff1cf396a5'))

. I tried different return value types for the flow (str, dict). What am I missing here? How can I easily access the return value of a succesful flow run? Thank you

Hi, is there a way to export the flow logs from Prefect? I would like to save the logs for at least 90 days, preferably with a copy of the flow logs imported our own logging infrastructure (Loki) for searchability.

Hello, can we connect to prefect cloud without using the

cloud login

command ? I tried setting

prefect config set PREFECT_API_URL

and

PREFECT_API_KEY

as a setting but it doesn't seem to work.

Hi, we are facing a strange behavior with a DBT task. It runs successfully, but often, a few seconds after the completion, DBT returns an error with the message “Command failed with exit code 137”, which is related to lack of memory. We have many DBT flows with heavier executions, wondering if anyone had a similar issue.

Hello, in prefect 1.0 - what can lead a flow’s mapped task to just immediately move to the finished state of

Mapped

- with no mapped tasks being created, and then to the flow itself just immediatley moving to the

Failed

stated - even though nothing failed?

I am having problems with the https://prefecthq.github.io/prefect-email/ when trying to use office 365 as my provider. It is giving me

ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:997)

my credentials look like this

email_server_credentials = EmailServerCredentials(

username="demo@company.com", password="mypasswordhere", smtp_server="smtp.office365.com", smtp_type="STARTTLS", smtp_port=587 )

[Prefect 1.0 -> 2.0 Migration Question] Hi! I have a flow that accepts lists of params as input. The flow is composed of many tasks that occur sequentially and in parallel. One of the initial tasks for a particular input param results in failure so the subsequent tasks are generally marked

Not Ready

with the

future.result()

resolving to

None

. The issue is that two of the final tasks of my flow accept futures of previous tasks as input and are listed in the

wait_for

section as well. These final two tasks instead are stuck in a

Pending

state and this causes the flow to hang instead of running to completion. Any advice on how to properly implement the ideal behavior (tasks that depend on futures from previous tasks that fail or didn't run should also fail/not run and flow should run to completion) would be greatly appreciated, thank you! In the logs, I see the flow encountered an

UpstreamTaskError

.

Hello, sorry to jump in the "I've got an issue" I'm currently following the "Update to Prefect 2.0: Notifications" YouTube and have followed the tutorial to the letter but when running the flow for "notification_block" I get the following error:- "RuntimeWarning: coroutine 'SlackWebhook.notify' was never awaited slack_webhook_block.notify("Hello from Prefect!")" When using the "Notifications" within the Prefect Cloud 2.0 for the Slack notification it works fine when the flow is completed.

Is there a new thing in prefect 2.0 where you can't return Google BigQuery Clients within a task? I know you can in Prefect 1.0 I've tried different task runners for 2.0 and they all fail

Hi 👋 For

KubernetesJob

is it possible to specify a

node-selector

and

tolerations

to be able to assign the worker pod to a specific node pool ? I can’t see it in the doc Thanks for your help

Hey folks, we've been playing around with Prefect 2.0.4 and got a question related to deployments with storage blocks, is the expectation that the deployment files are built per environment that exist? Our thought was that we would create the deployment yaml once and apply the same yaml between dev, preprod, and prod environments which are all separate instances. The problem we're running into is that when we generate the storage block (remote file system using minio) the storage block ids are different between our environments, so when we attempt to apply the deployment yaml via CLI it fails because the yaml file references the storage block id of the environment that the build command was executed on

Hey folks, we’re seeing very strange flow registration behaviour when trying to register a long-standing existing flow to Prefect Cloud (1.0). My question is, is there a 2500 max task limit in Prefect Cloud on statically defined DAGs? We’ve been registering this flow for months (and it’s been growing) and we’ve just surpassed 2500 nodes, and it’s now throwing an error, so want to check if you have an internal limit defined somewhere…

Hello. Starting to look at migrating to Prefect 2.0, and I’m a little confused about how to structure storage blocks for flows. Is it best practice to create a single block for all flows, or to create one block per flow? I’m not seeing any way when building the deployment to specify a subfolder, which makes me think option 1 is kind of forced…

Join us for PrefectLive this Wednesday, live on Twitch right now. @Taylor Curran will be showing us the cool stuff you can do with AWS Blocks! 🦜

Hello. I have a question. I want to deploy my flow in local subprocess from my VM hosted in AWS. The flow code is on my local computer with all the dependencies. When I execute the command "prefect deplyment buid ..." in my local computer with s3 as my block , should I normally start the agent on my VM in AWS. How to configure the agent so that before executing my flow on my VM, he will install the dependicies required. I read this article on how to deploy prefect 2 on AWS but it doesn't mentioned how the agant will execute the flow. Does the agent install dependicies before running the task inside the flow ? Here is the link https://discourse.prefect.io/t/how-to-deploy-prefect-2-0-flows-to-aws/1252

Hi, could anyone suggest how do we set up ssh keys in prefect kubernetes agent? we need to connect to a SFTP server that we can only let local development env work but not after register the flow.

What methods are folks using to send custom Kubernetes job templates to a

kubernetes-job

infrastructure block in Prefect 2.0? In 1.0, we could accomplish this by using the

KubernetesRun

module and passing it like so:

KubernetesRun(job_template_path="my/local/path.yaml")

It seems like we can try to mimic the functionality in 2.0 by using some hacks to pipe the output from our job template .yaml file, represent it as a one-line JSON string then force that output into

--override job=

but this doesn’t seem ideal.

This work queue uses a deprecated tag-based approach to matching flow runs; it will continue to work but you can't modify it

I noticed this warning on the Prefect Cloud UI. Is there any documentation on the recommended approach to use?

Hi all, I have some questions about prefect 2.0 versus 1.0. I use to use prefect 1.0 at a previous company, and am trying to get prefect up and running at a new company. I started down the road of prefect 2.0 but didn’t quite realize how much has changed, and am now wondering if I should stick with prefect 1.0. It looks like there may have been a useful discourse post about this topic (as reference here: https://docs-v1.prefect.io/), but that post no longer exists. Are there other resources on this? Specifically I am most worried about the new UI for prefect 2.0 as it doesn’t seem that useful for my use case - I don’t see the option to start flow runs, view and retry tasks and task graphs, etc.

is there a way to use

Parameters

or something analogous as input to a

KubernetesRun

to request a specific amount of resources on the fly? I'm working w/ Fargate and want to give users flexibility to run flows with differing amounts of compute

Hi all why can we not create service accounts in the ui for v1 - we are on the standard plan