Hi all. Can someone point me to documentation on how to have lazarus restart dead flows from the beginning? It appears the default is to restart from the failed task.

It looks like I can no longer have keyword function params defaulted to

None

defined on a mapped task? I now get this error (

TypeError: object of type 'NoneType' has no len()

) when trying to map across the subsequent function signature.

@task
def get_items_list(
    client, entity, page,
    response_attr=None, path=None, params=None
):

I've seen other questions about this, but not a solution yet. When using

Secret.load("slackbot-oauth").get()

, I get

AttributeError: 'coroutine' object has no attribute 'get'

whether I run this in an ipynb or with a python file through the terminal. Is it expected to run this command with await and async? Or is this not supposed to happen?

Is there a prefect 2 equivalent to template task names? I've searched high and low on the prefect 2 docs/slack and can't find anything. old docs: https://docs-v1.prefect.io/core/concepts/templating.html#templating-names

hi all, I've been in transition to prefect 2 (2.0.4 to be precise) from prefect < 2 (1,2 to be precise). Thanks to your help I sovled most of the problems, only one remains: in prefect < 2 I used parallelism via

multiprocessing.Process

as

processes = []
    for chunk in url_chunks:
        kwargs["mp3_urls"] = chunk
        processes.append(mp.Process(foo=foo, kwargs=kwargs))
    for p in processes:
        p.start()
    for p in processes:
        p.join()

and it worked. The main reason for using

multiprocessing.Process

is to have a large model in memory shared between processes. In fact it works when I run this exact flow with prefect via a python script calling the flow as a fucntion, it works as expected. However, when run it as

prefect deployment run ...

or via a scheduled run, the processes in

p.start()

don't run. The flow result is marked as success to my surprise. Any ideas?

my colleague can't login to prefect 1.0 due to a forgotten password, but he's unable to reset his password as it says to contact his system admin (me) - I have received no email regarding his password reset request @Dave Thomas

Hi, I am trying to run a

KubernetesAgent

locally and to bind it to minikube for testing purposes. For now, I see the Agent in the Server UI, but the tasks are not executed. How do I configure this? Thank you for you patience, I am new to Prefect.

😄  minikube v1.26.1 on Ubuntu 20.04
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🏃  Updating the running docker "minikube" container ...
🐳  Preparing Kubernetes v1.24.3 on Docker 20.10.17 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image <http://gcr.io/k8s-minikube/storage-provisioner:v5|gcr.io/k8s-minikube/storage-provisioner:v5>
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Getting an SQLAlchemy error when trying to parallelize about 3000 task runs on a Dask cluster. Confusing to me as this is something that "just worked" on Prefect 1.0, but isn't working for me on 2.0...

sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: <https://sqlalche.me/e/14/3o7r>)

Hi I am just setting up prefect for my Data engineering team ran into one issue when trying to setup ecs fargate.

prefect agent ecs start --cluster arn:aws:ecs:us-west-2:00000000:cluster/prefect-dev --label aws-ecs

Usage: prefect agent [OPTIONS] COMMAND [ARGS]...

Try 'prefect agent --help' for help.

╭─ Error ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮

│ No such command 'ecs'.                                                                                   │

╰───────────────────────────────────────────

Hey - are you guys recommending prefect 2.0 for production yet? Is there an ECSRun type setup that I can explore in order to replicate what we currently have in 1.0 ? Thanks

Hi, I’m porting a flow from 1.0 to 2.0. My flow needs a couple of secrets. During development I would like to run the flow via a plain old python call and the ephemeral API, with

if __name__ == '__main__': myflow()

. This is without a running Orion server. The code in my flow that creates the Secret block appears to work. However, when I try use my Secret block in my 2.0 flow the prefect client raises a PrefectHTTPStatusError, and says ‘Block document not found’. In Prefect 1.0 I could use

PREFECT__CONTEXT__SECRETS__

to set the values I need, but I don’t see any mention of similar functionality for 2.0.

[Error with Prefect Docker Agent on AWS EC2] Hello, I followed the minitutorial here but did not work and got the following error (see pic) @Jamie Zieziula any idea how can I fix this? 😕

Where can I find up-to-date information about deployments and the file format for the yaml files?

What is the possible reason for a "Failed to create schedule" error message in the prefect cloud 2 when i want to add a schedule to a deployment via the cloud ui?

Hi, I'm having trouble with blocks. There are no blocks in the block catalog! Just says "0 blocks". No filters applied either. UPDATE:

prefect block type ls

does give the block types. UPDATE2: Something is off, when hosting on localhost and doing an ssh tunnel on it it works like a charm. When hosting on

0.0.0.0

the page is empty. UPDATE3: Now it works on the public IP as well ?!? Maybe it needs to start without

--host

the first time

[Prefect 2.0.4 + Prefect Cloud] Hey, I'm struggling with the deployment override api and kubernetes-job infrastructure I've created a generic kubernetes-job infrastructure block with some sane defaults for image etc. but no env variables. If I run this block without overrides, everything is fine and the flow starts as expected.

prefect deployment build r_script_automation.py:r_script_automation --name retention_cohort_analysis_deployment -t k8s -sb gcs/gcs-prefect-stprage -ib kubernetes-job/generic-k8s-job -o cohort_deployment.yaml

However, as soon as I add infrastr. overrides, the flow stays in "pending" state forever

prefect deployment build r_script_automation.py:r_script_automation --name retention_cohort_analysis_deployment -t k8s -sb gcs/gcs-prefect-stprage -ib kubernetes-job/generic-k8s-job -o cohort_deployment.yaml --override image=europe-docker.pkg.dev/vol-at/rm-datateam-repository/r-script-automation:beta-23 --override env.GIT_PYTHON_REFRESH="quiet" --override env.GOOGLE_APPLICATION_CREDENTIALS="/google/.google-secret-key.json" --override env.CHROMIUM_FLAGS="--no-sandbox"

If I manually delete the infra_overrides-block from the deployment-yaml - the flow works again. So somehow this env.xyz - overrides are the problem. Can you point me to my mistake? Edit: I tested overriding the "image" - this works. It really only fails when overriding the environment variables....

Hi, Can someone help me here ? PS C:\Users\Sachin Ranveer\Desktop\Logward\data-ingestion-service> prefect orion start prefect : The term 'prefect' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. At line:1 char:1 + prefect orion start + ~~~~~~~ + CategoryInfo : ObjectNotFound: (prefect:String) [], CommandNotFoundException + FullyQualifiedErrorId : CommandNotFoundException

Prefect

2.0.4

Hi all, I just upgraded from

2.0b8

to

2.0.4

. In

v2.0b8

errors within flows/tasks were caught by Prefect and the flow would continue, now in

2.0.4

they are being raised and crashing my run. How can I get back to the previous behavior? I’ll post my code in the thread

Hi there. I have an Airflow DAG that pulls data from an API. It starts with a sensor that polls the API until the data is ready. I am wondering what would be the best pattern to implement this in 2.0 Instead of having a long running process that polls, I was thinking I could have a flow that checks for data and reschedules itself up to N times if the data is not ready. When it finds data, it continues the process and does not reschedule itself. Could someone help with a code sample to do this in 2.0? Can you think of a better approach? Thanks!

Hey, I am executing a flow run from a deployment via client. Everything runs locally, with default storage configuration. After the flow finished succesfully, I request the state and try to access the result but

state.result()

only returns

_Result(key='82ecff4d3a454e77b08b5ce93e1dd418', filesystem_document_id=UUID('af2119c0-c631-45ed-af6a-0cff1cf396a5'))

. I tried different return value types for the flow (str, dict). What am I missing here? How can I easily access the return value of a succesful flow run? Thank you

Hi, is there a way to export the flow logs from Prefect? I would like to save the logs for at least 90 days, preferably with a copy of the flow logs imported our own logging infrastructure (Loki) for searchability.

Hello, can we connect to prefect cloud without using the

cloud login

command ? I tried setting

prefect config set PREFECT_API_URL

and

PREFECT_API_KEY

as a setting but it doesn't seem to work.

Hi, we are facing a strange behavior with a DBT task. It runs successfully, but often, a few seconds after the completion, DBT returns an error with the message “Command failed with exit code 137”, which is related to lack of memory. We have many DBT flows with heavier executions, wondering if anyone had a similar issue.

Hello, in prefect 1.0 - what can lead a flow’s mapped task to just immediately move to the finished state of

Mapped

- with no mapped tasks being created, and then to the flow itself just immediatley moving to the

Failed

stated - even though nothing failed?

I am having problems with the https://prefecthq.github.io/prefect-email/ when trying to use office 365 as my provider. It is giving me

ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:997)

my credentials look like this

email_server_credentials = EmailServerCredentials(

username="demo@company.com", password="mypasswordhere", smtp_server="smtp.office365.com", smtp_type="STARTTLS", smtp_port=587 )

[Prefect 1.0 -> 2.0 Migration Question] Hi! I have a flow that accepts lists of params as input. The flow is composed of many tasks that occur sequentially and in parallel. One of the initial tasks for a particular input param results in failure so the subsequent tasks are generally marked

Not Ready

with the

future.result()

resolving to

None

. The issue is that two of the final tasks of my flow accept futures of previous tasks as input and are listed in the

wait_for

section as well. These final two tasks instead are stuck in a

Pending

state and this causes the flow to hang instead of running to completion. Any advice on how to properly implement the ideal behavior (tasks that depend on futures from previous tasks that fail or didn't run should also fail/not run and flow should run to completion) would be greatly appreciated, thank you! In the logs, I see the flow encountered an

UpstreamTaskError

.

Hello, sorry to jump in the "I've got an issue" I'm currently following the "Update to Prefect 2.0: Notifications" YouTube and have followed the tutorial to the letter but when running the flow for "notification_block" I get the following error:- "RuntimeWarning: coroutine 'SlackWebhook.notify' was never awaited slack_webhook_block.notify("Hello from Prefect!")" When using the "Notifications" within the Prefect Cloud 2.0 for the Slack notification it works fine when the flow is completed.

Is there a new thing in prefect 2.0 where you can't return Google BigQuery Clients within a task? I know you can in Prefect 1.0 I've tried different task runners for 2.0 and they all fail

Hi 👋 For

KubernetesJob

is it possible to specify a

node-selector

and

tolerations

to be able to assign the worker pod to a specific node pool ? I can’t see it in the doc Thanks for your help