Hi. Beginner question here for prefect 1.0. How would you advise the following type of pattern be implemented? I will have a number of tasks, some of which should have error handling tasks and the final task should only be invoked if the error handlers have either a) not been triggered, or b) succeeded in correcting the error. I've attached a schematic where green arrows represent successes and red failures. If there is no error handler attached to a task then a failure should fail the flow as well

Hi I am trying to work out how to set number of concurrent threads in Prefect 1 or 2. E.g. I want to achieve 4 threads in parallel, and not more. Can this be achieved without using the Cloud orchestrator, e.g. in the code / flow alone?

Dear Prefect Community, When we use the Prefect 2.0 UI with latest b6 version, the refresh button is not returning any data for any link/endpoint, we have to click the buttons within the browser, we can't use any specific end points to filter it via URL. For example, http://abc.net:5200 works, but http://abc.net:5200/runs can't be used with "refresh" button or any other link with different end point other than http://abc.net:5200. Is it a known defect?

Hello. I’m trying to test out Gitlab storage by following option #2 laid out here. I can see the environment variables

PREFECT__CLOUD__USE_LOCAL_SECRETS

and

PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN

set on my agent, but not on the job. Trying to run the flow just errors out at

Failed to load and execute flow run: KeyError('The secret GITLAB_ACCESS_TOKEN was not found.  Please ensure that it was set correctly in your tenant: <https://docs.prefect.io/orchestration/concepts/secrets.html>')

hey, using case statement in prefect 1. Is there a way to evaluate the argument to be != to the result instead of == ?

Hi everyone. I am trying to build an ETL process that pulls flows' data out of prefect, transforms it into a table, and gets loaded into a database. The request is to have a constant clean table in our database base where an analyst can grab the dataset and give a report on the consistency, reliability, and other potential metrics of our ETL processes. We are currently under the free version of 20,000 calls and have a single agent working with prefect cloud. I am wondering if anyone could point me in the right direction on getting started to pull data out of the API or the best documentation and examples out there. Any idea or resource will be highly appreciated.

Is there any documentation for orion on what happens to flows that are late? Our execution environment running the agent crashed and the Prefect response was not what we expected. Based on our testing it looks like the first step is that flows are marked late. If an agent becomes available a late flow will start. But then at some point after a few hours the late flow will be marked pending and it will not restart even if an agent becomes available. Is there a way to fine tune this behavior?

What would it take to have each task send logs both to the cloud and to a file on the local disk (A different file for each task)?

@Kevin Kho a month or so we had a quick comment about upcoming improvements to pickling in Prefect. have those gotten added yet to Prefect 1.0?

Can we register flows individually instead registering all the flows at once

Hi folks, Is there a way to get meta of flow run ( like, flow run name, started time etc.) inside the flow run. For example, few of my task inside the flow run inserts data in a database, so for monitoring and logging purpose I want to map flow run to those inserts, so think of like adding few columns to the table in db, named

prefect_flow_run_id

and

prefect_flow_version

and inserting those params in the db

Is there an easy way to set/change the Name/Slug of a Task based on the arguments? (ie, a task for running Jupyter notebooks that names the task after the filename of the notebook)

Hi, I’m receiving this error when trying to use

upstream_tasks

Does someone know what I’m doing wrong? My intention is to execute

task_3

only after

task_1

and `task_2`is finished (since it depends on the output of them) and start

task_4

only after the

task_3

is finished How can I accomplish that? (Code and output error in thread)

Dear Prefect Community, I have written some custom python to extract an load data into a snowflake database. It works well from my local machine but attempts to orchestrate these pipelines with Prefect cloud fail. I have followed @Anna Geller 's brilliant article on using prefect and AWS ACS Fargate as a serverless pipeline solution using the

prefecthq/prefect:latest-python3.10

docker image with some additional packages including the snowflake python connector and it's dependencies. Flows without snowflake interactions work perfectly when run from Prefect cloud however my EL flows which insert data into snowflake tables fail to load and execute returning >ModuleNotFoundError("No module named 'snowflake'"). Haven't been able to fine any similar reports among the community so wondering if someone could suggest what I may have done wrong. With thanks, Daniel.

Hi all, is it possible to override a default parameter value when register a flow via the CLI? For example, our flows have an environment parameter (eg dev, qa, prod) and we don’t want to hard code that in the flow code. Ideally when we register the flow, we want to set the default value for that parameter. Thanks in advance.

Hi, I am getting a

prefect.exceptions.ClientError: [{'path': ['create_project'], 'message': 'Uniqueness violation.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

when I ran

prefect delete project

-->

prefect create project

(using Prefect v1.0, with Prefect Cloud as backend). Can't seem to recreate a project with the same name

Hi , I’m planning to use prefect as a replacement of our internal framework, its worked like charm while using local agent but when i’m trying ecs agent , im facing issues , i want to run the same etl job on ec2 instances . The guide provided on prefect website (https://docs.prefect.io/orchestration/agents/ecs.html#running-ecs-agent-in-production) is not working with issue ( i even provided aws keys as env variables for task ), if i dont provide args it throws any error asking for it. I’m kind of stuck on this , kindly help me out. Thanks in advance

hi, i am getting sometimes "error occured while creating new work queue" when creating new work-queue from the ui [prefect 2.0]

Prefect 2.0: Hi, I can't seem to be able to run a deployed flow with tags -- it's not being picked up by the work queue/agent that's supposed to be handling that tag. 1. I created a work queue named

dev_queue

with the label

dev

and supporting all flow runners. It's the only work queue I have set up. 2. I ran an agent with

prefect agent start dev_queue

3. I created a deployment with the tag `dev`:

name: test_platform_flow_first_deployment
flow_name: Data Platform Demo
flow_location: ./test_platform_flow.py
parameters:
  to_print: "Hello from first deployment!"
tags:
  - dev

I verified in the UI that it has the right tag and flow runner. However, when I run the deployment, the flow run is never picked up. One suspicious thing I noticed is that the flow run doesn't inherit the

dev

tag from the deployment (although I don't know if it's a bug or a feature). When I remove all labels from the work queue, the flow runs are picked up correctly.

Is there a way to specify

flow_runner

in the deployment? I get `ValueError: Unregistered flow runner 'DockerFlowRunner'`when running

prefect deployment create my_deployment.yaml

. My deployment looks like this:

name: test_platform_flow_first_deployment
flow_name: Data Platform Demo
flow_location: ./test_platform_flow.py
parameters:
  to_print: "Hello from first deployment!"
tags:
  - dev
flow_runner: 
  type: DockerFlowRunner
  config:
    image: viadot:orion

Unfortunately the

flow_runner

config is not documented anywhere so it's hard for me to say if I'm specifying it incorrectly or it's not supported at all.

Hi. We are trying to create a base docker image with standard dependencies that we will use across multiple flows. We use ecs executor and for that we created an docker image based on the recommendation described here - https://medium.com/the-prefect-blog/the-simple-guide-to-productionizing-data-workflows-with-docker-31a5aae67c0a. Unfortunately it looks like the prefect ecs agent is not able to read the previously installed modules. We are sure that the modules are installed correctly because we see in logs that they are skipped when we try to install them using the EXTRA_PIP_PACKAGES environment package. Is there anything additionally that we have to define to make the pip installed modules available for prefect

Hi guys, after 2 days of trying to register my flow via gitlabCI I gave up, please help! here's my ci:

build-and-deploy-production:
   stage: build
   # image: prefecthq/prefect:latest-python3.9
   variables:
     DOCKER_HOST: <tcp://localhost:2375>
     DOCKER_TLS_CERTDIR: ""
   # image: python:3.9.13-buster
   image: google/cloud-sdk
   services:
     - docker:19.03.1-dind
   script:
     - pip3 install prefect
     - echo $PREFECT_PRODUCTION_SERVICE_ACCOUNT > /tmp/$CI_PIPELINE_ID.json
     - gcloud auth activate-service-account --key-file /tmp/$CI_PIPELINE_ID.json
     - gcloud config set project my project
     # - gcloud components install docker-credential-gcr
     - gcloud auth configure-docker --quiet
     - prefect auth login -k mytoken
     - cd ./asin2vec_pipeline/asin2vec
     - pip3 install -r requirements.txt
     - export GOOGLE_APPLICATION_CREDENTIALS=/tmp/$CI_PIPELINE_ID.json
     - prefect register --project "onboarding" --path flow.py --name "asin2vec_v2"

as you can see, tried with images: prefecthq/prefect:latest-python3.9, cloudsdk, and python.

with all had the same error:       InterruptedError: denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/myproject/locations/us-central1/repositories/data-pipelines" (or it may not exist)

to be clear, locally it works, when I authenticate with this service account.

Hi Team, can someone help me with one of my query, I am trying to use helm chart on kubernetes cluster, but facing issue while connecting to the backend apollo, by default the ui and graphql server is trying to connect http://localhost:4200, whereas I have been exposing the apollo using NodePort on specific node and host, PFA for the error that I am getting on graphql while creating tenants, FYI have changed the ENV variables to use the new values as well and also tried adding config.toml file to graphql pod

Hi everyone. Is there a way to get the beta version of orion on conda-forge? I can always pip install within my env, but would like to avoid that if possible

Hello, I have recently been reading the docs on Perfect 2.0, and had a few questions. • There is a mention of Storage in docs, but no examples on how it actually fits the big picture. What’s the best way to handle passing larger-than-memory data between tasks/flows? Does Storage fit here? • Let’s say I have a flow A doing preprocess, train and test, and now I want the flow B to be exactly like A except with a different training Task/Flow. Is there a way to only override parts of a Flow? Thank you.

hello, i'm trying to use prefect2.0 and launch ui by "prefect orion start", then i visit 127.0.0.1:4200， but got a very simple and ugly ui which is different from the official webside. is there any thing wrong?

Hi, I am trying to run an async flow and get: AttributeError: 'coroutine' object has no attribute 'is_failed'. Any thoughts?

if you delete a project in Cloud can you not recreate a project with the same name? I deleted some in my dev tenant so I could start fresh, and five minutes later I am still getting an error stating that the project already exists

This message was deleted.

hi community! is there a helm chart for a Kubernetes Agent that speaks to Prefect Cloud? i.e., only the agent, not the entire server application from https://github.com/PrefectHQ/server/tree/master/helm/prefect-server#prefect-server-helm-chart