Hi. We are trying to create a base docker image with standard dependencies that we will use across multiple flows. We use ecs executor and for that we created an docker image based on the recommendation described here - https://medium.com/the-prefect-blog/the-simple-guide-to-productionizing-data-workflows-with-docker-31a5aae67c0a. Unfortunately it looks like the prefect ecs agent is not able to read the previously installed modules. We are sure that the modules are installed correctly because we see in logs that they are skipped when we try to install them using the EXTRA_PIP_PACKAGES environment package. Is there anything additionally that we have to define to make the pip installed modules available for prefect

Hi guys, after 2 days of trying to register my flow via gitlabCI I gave up, please help! here's my ci:

build-and-deploy-production:
   stage: build
   # image: prefecthq/prefect:latest-python3.9
   variables:
     DOCKER_HOST: <tcp://localhost:2375>
     DOCKER_TLS_CERTDIR: ""
   # image: python:3.9.13-buster
   image: google/cloud-sdk
   services:
     - docker:19.03.1-dind
   script:
     - pip3 install prefect
     - echo $PREFECT_PRODUCTION_SERVICE_ACCOUNT > /tmp/$CI_PIPELINE_ID.json
     - gcloud auth activate-service-account --key-file /tmp/$CI_PIPELINE_ID.json
     - gcloud config set project my project
     # - gcloud components install docker-credential-gcr
     - gcloud auth configure-docker --quiet
     - prefect auth login -k mytoken
     - cd ./asin2vec_pipeline/asin2vec
     - pip3 install -r requirements.txt
     - export GOOGLE_APPLICATION_CREDENTIALS=/tmp/$CI_PIPELINE_ID.json
     - prefect register --project "onboarding" --path flow.py --name "asin2vec_v2"

as you can see, tried with images: prefecthq/prefect:latest-python3.9, cloudsdk, and python.

with all had the same error:       InterruptedError: denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/myproject/locations/us-central1/repositories/data-pipelines" (or it may not exist)

to be clear, locally it works, when I authenticate with this service account.

Hi Team, can someone help me with one of my query, I am trying to use helm chart on kubernetes cluster, but facing issue while connecting to the backend apollo, by default the ui and graphql server is trying to connect http://localhost:4200, whereas I have been exposing the apollo using NodePort on specific node and host, PFA for the error that I am getting on graphql while creating tenants, FYI have changed the ENV variables to use the new values as well and also tried adding config.toml file to graphql pod

Hi everyone. Is there a way to get the beta version of orion on conda-forge? I can always pip install within my env, but would like to avoid that if possible

Hello, I have recently been reading the docs on Perfect 2.0, and had a few questions. • There is a mention of Storage in docs, but no examples on how it actually fits the big picture. What’s the best way to handle passing larger-than-memory data between tasks/flows? Does Storage fit here? • Let’s say I have a flow A doing preprocess, train and test, and now I want the flow B to be exactly like A except with a different training Task/Flow. Is there a way to only override parts of a Flow? Thank you.

hello, i'm trying to use prefect2.0 and launch ui by "prefect orion start", then i visit 127.0.0.1:4200， but got a very simple and ugly ui which is different from the official webside. is there any thing wrong?

Hi, I am trying to run an async flow and get: AttributeError: 'coroutine' object has no attribute 'is_failed'. Any thoughts?

if you delete a project in Cloud can you not recreate a project with the same name? I deleted some in my dev tenant so I could start fresh, and five minutes later I am still getting an error stating that the project already exists

This message was deleted.

hi community! is there a helm chart for a Kubernetes Agent that speaks to Prefect Cloud? i.e., only the agent, not the entire server application from https://github.com/PrefectHQ/server/tree/master/helm/prefect-server#prefect-server-helm-chart

Hello. I'm trying to use ShellTask locally on Windows using Prefect core, but I'm having issues. I found this old Issue on Github saying that ShellTask isn't windows-compatible. Is there a fix or any workaround?

Hi! I recently added a schedule to one of my flows and pushed it to cloud. I now have the latest version and -1 version both scheduled and am unable to find where to disable the -1 version.

Hi everyone, I am having a hard time filtering graphQL flows by agent id or label name. any ideas on how to accomplish this?

Hi.. quick question can I have a task that returns a list of strings then from that list I'll loop over it and do some logic. the issue I have is

object is not iterable

is there anyway to return the result of a task as a normal python object list, dict and not a task (please note i don't know the length of list and its very dynamic)

I’m running a Prefect Flow on a Docker Agent with a DaskExecutor. The flow runs fine for awhile, then I get an error from the CloudFlowRunner

Unexpected error: KilledWorker('{{TaskRunName}}', <WorkerState '<tcp://127.0.0.1:44295>', name: 1, status: closed, memory: 0, processing: 72>)

Any idea what might be causing this?

is there a trigger like “at least one successful, and the rest skipped”. I have some logic like so

@task
def required():
    pass


@task
def optional():
    pass


@task
def final():
    pass


with Flow('example'):
    res = required()
    with case(some_case, True):
        optional_res = optional()

    final(res, upstream_tasks=[optional_res])

If optional is skipped, its okay to run final. If required fails, or optional fails, I dont want to run final

the example is simple, but final would have additional upstream dependencies, which would be treated like res

Hey there! I’m trying to figure out the easiest way to trigger a prefect cloud (1.0) job via the API - I know I can use the GraphQL API but I don’t have a ton of experience with GraphQL and I want to trigger a flow run based on knowing the flow name — not the id. I also need to trigger a flow on the latest version of the flow — and there will be several versions. This all needs to take place within the context of an automated build process (GCP cloud build) so I don’t want to have to install the cli or anything - I just want to drop in an API key via a secret and run a simple script to trigger the flow at the end of a build. Ideally it would be just a bash/curl script but I could also write a basic python script to do it.

Hey - I have a flow that has a few clocks eg

au_harness_win_place = CronClock(
    cron="33 * * * *",
    parameter_defaults={
        "race_types": ["Harness"],
        "market_type_codes": ["WIN", "PLACE"],
        "event_type_ids": None,
        "market_countries": ["AU"],
        "table_name": "harness_market_catalogue_au"
    },
    start_date=datetime.now(tz=timezone("Australia/Brisbane")),
)

that is called like so

with Flow(
    name="betfair_market_catalogues",
    storage=Storage.in_a_s3_bucket(),
    run_config=RunConfig().fargate_on_ecs(),
    schedule=Schedule(clocks=[au_nz_greys_win_place, au_harness_win_place]),
    executor=LocalDaskExecutor(scheduler="threads"),
) as flow:

If I want to call this via the cli it doesnt like the None values and the python objects, do I have to accept stringified versions of this or does prefect handle this on its own? eg, how would I run this locally

prefect run -p flows/market_catalogues/betfair_market_catalogues.py --param table_name="harness_market_catalogue_au" --param race_types=["Harness"] --param market_type_codes=["WIN", "PLACE"] --param event_type_ids=[] --param market_countries=["AU"]

doesnt do the job

We get a pickle thread lock error in a task when using paramiko. This is probably a problem with pickling ssh stuff. What is the alternative? How are people using paramiko with Prefect?

hi , i am executing my flow not from main flow. when i run it on the default executor everything is running file, when i use DaskExecutor it crashs with the following message 2022-06-22 14:06:40,649 prefect.TaskRunner INFO Task ‘train_model[0]’: Starting task run... 2022-06-22 14:06:40,650 prefect.TaskRunner INFO Task ‘train_model[1]’: Starting task run... The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. 2022-06-22 14:06:55.255 python[95140:4859256] * Terminating app due to uncaught exception ‘NSInternalInconsistencyException’, reason: ‘NSWindow drag regions should only be invalidated on the Main Thread!’ please advise.

Hello, community! I’m trying to build a flow with docker tasks. So below is some information: • prefect server • prefect docker and local agents are started • task with pulling image from hub.docker.com (it’s pulling ok) • task that create container • task that run container But when I run flow locally there is an error (screen). Could you please help with it? P.S. code in thread

We are using Prefect cloud and We want to enable task level SLA. Whenever the task in the flow is delayed, We want to get notification. is that possible?

I have a question about: restarted flows with trigger:any_successful tasks.

What is the best practice for implementing fan-out-to-fan-in patterns in Prefect 2.0? As a test scenario, I tried to create a flow that 1. Gets all AWS S3 bucket names 2. Spawns a task per bucket and counts the objects 3. Collects all count outputs and calculates the sum (Code is in the first comment) I get some errors (also in the comment) but I have the feeling that I'm using Prefect 2.0 in a way that might be wrong. How should I structure my flow to do it in the most Prefect-y way? I loosely based the idea on this amazing Prefect blog post.

Hey!I am facing issue while running prefect on VM instance of GCP.I am unable to open UI with external_IP:portNumber. Could anyone suggest solution to it?

Is there an updated timeline for the release of prefect 2.0? I see here

The Prefect 2.0 beta period is expected to last for at least the second quarter of 2022.

And that 2.0 is not recommended for production at this time. Is this still accurate? For context, we are looking to build an internal tool using prefect as the orchestrator and are debating whether to start with 2.0 or use 1.0 and migrate. We would be hoping to start using the tool Q3 this year.

I am trying to run a flow through an ECS Agent on FARGATE. I have successfully deployed the prefect agent on cluster. But I'm unable to run a registered flow. I'm using prefect 1.2.2. Logs only say

Submitted for execution: Task arn:

Can anybody suggest a solution?