https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • r

    Roman Jendrusch

    06/20/2022, 3:41 PM
    Hi. We are trying to create a base docker image with standard dependencies that we will use across multiple flows. We use ecs executor and for that we created an docker image based on the recommendation described here - https://medium.com/the-prefect-blog/the-simple-guide-to-productionizing-data-workflows-with-docker-31a5aae67c0a. Unfortunately it looks like the prefect ecs agent is not able to read the previously installed modules. We are sure that the modules are installed correctly because we see in logs that they are skipped when we try to install them using the EXTRA_PIP_PACKAGES environment package. Is there anything additionally that we have to define to make the pip installed modules available for prefect
    ✅ 1
    a
    12 replies · 2 participants
  • o

    Ofek Katriel

    06/20/2022, 8:48 PM
    Hi guys, after 2 days of trying to register my flow via gitlabCI I gave up, please help! here's my ci:
    build-and-deploy-production:
       stage: build
       # image: prefecthq/prefect:latest-python3.9
       variables:
         DOCKER_HOST: <tcp://localhost:2375>
         DOCKER_TLS_CERTDIR: ""
       # image: python:3.9.13-buster
       image: google/cloud-sdk
       services:
         - docker:19.03.1-dind
       script:
         - pip3 install prefect
         - echo $PREFECT_PRODUCTION_SERVICE_ACCOUNT > /tmp/$CI_PIPELINE_ID.json
         - gcloud auth activate-service-account --key-file /tmp/$CI_PIPELINE_ID.json
         - gcloud config set project my project
         # - gcloud components install docker-credential-gcr
         - gcloud auth configure-docker --quiet
         - prefect auth login -k mytoken
         - cd ./asin2vec_pipeline/asin2vec
         - pip3 install -r requirements.txt
         - export GOOGLE_APPLICATION_CREDENTIALS=/tmp/$CI_PIPELINE_ID.json
         - prefect register --project "onboarding" --path flow.py --name "asin2vec_v2"
    as you can see, tried with images: prefecthq/prefect:latest-python3.9, cloudsdk, and python.
    
    with all had the same error:       InterruptedError: denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/myproject/locations/us-central1/repositories/data-pipelines" (or it may not exist)
    to be clear, locally it works, when I authenticate with this service account.
    ✅ 1
    a
    11 replies · 2 participants
  • a

    Ankur Sheth

    06/20/2022, 9:17 PM
    Hi Team, can someone help me with one of my query, I am trying to use helm chart on kubernetes cluster, but facing issue while connecting to the backend apollo, by default the ui and graphql server is trying to connect http://localhost:4200, whereas I have been exposing the apollo using NodePort on specific node and host, PFA for the error that I am getting on graphql while creating tenants, FYI have changed the ENV variables to use the new values as well and also tried adding config.toml file to graphql pod
    ✅ 1
    a
    5 replies · 2 participants
  • j

    Jack Sundberg

    06/20/2022, 11:44 PM
    Hi everyone. Is there a way to get the beta version of orion on conda-forge? I can always pip install within my env, but would like to avoid that if possible
    ✅ 1
    a
    4 replies · 2 participants
  • g

    Gol Bahar

    06/21/2022, 2:14 AM
    Hello, I have recently been reading the docs on Perfect 2.0, and had a few questions. • There is a mention of Storage in docs, but no examples on how it actually fits the big picture. What’s the best way to handle passing larger-than-memory data between tasks/flows? Does Storage fit here? • Let’s say I have a flow A doing preprocess, train and test, and now I want the flow B to be exactly like A except with a different training Task/Flow. Is there a way to only override parts of a Flow? Thank you.
    ✅ 1
    k
    1 reply · 2 participants
  • y

    yu zeng

    06/21/2022, 3:08 AM
    hello, i'm trying to use prefect2.0 and launch ui by "prefect orion start", then i visit 127.0.0.1:4200, but got a very simple and ugly ui which is different from the official webside. is there any thing wrong?
    ✅ 1
    k
    a
    18 replies · 3 participants
  • a

    Apostolos Papafragkakis

    06/21/2022, 10:56 AM
    Hi, I am trying to run an async flow and get: AttributeError: 'coroutine' object has no attribute 'is_failed'. Any thoughts?
    ✅ 1
    a
    12 replies · 2 participants
  • j

    Jessica Smith

    06/21/2022, 2:04 PM
    if you delete a project in Cloud can you not recreate a project with the same name? I deleted some in my dev tenant so I could start fresh, and five minutes later I am still getting an error stating that the project already exists
    k
    24 replies · 2 participants
  • s

    Slackbot

    06/21/2022, 2:35 PM
    This message was deleted.
  • h

    Horatiu Bota

    06/21/2022, 3:10 PM
    hi community! is there a helm chart for a Kubernetes Agent that speaks to Prefect Cloud? i.e., only the agent, not the entire server application from https://github.com/PrefectHQ/server/tree/master/helm/prefect-server#prefect-server-helm-chart
    k
    3 replies · 2 participants
  • m

    Mansour Zayer

    06/21/2022, 3:45 PM
    Hello. I'm trying to use ShellTask locally on Windows using Prefect core, but I'm having issues. I found this old Issue on Github saying that ShellTask isn't windows-compatible. Is there a fix or any workaround?
    k
    13 replies · 2 participants
  • w

    Wei Mei

    06/21/2022, 4:27 PM
    Hi! I recently added a schedule to one of my flows and pushed it to cloud. I now have the latest version and -1 version both scheduled and am unable to find where to disable the -1 version.
    ✅ 1
    1 reply · 1 participant
  • a

    Alfred Martinez

    06/21/2022, 5:05 PM
    Hi everyone, I am having a hard time filtering graphQL flows by agent id or label name. any ideas on how to accomplish this?
    k
    1 reply · 2 participants
  • i

    ibrahem

    06/21/2022, 9:31 PM
    Hi.. quick question can I have a task that returns a list of strings then from that list I'll loop over it and do some logic. the issue I have is
    object is not iterable
    is there anyway to return the result of a task as a normal python object list, dict and not a task (please note i don't know the length of list and its very dynamic)
    m
    1 reply · 2 participants
  • j

    Josh

    06/21/2022, 10:16 PM
    I’m running a Prefect Flow on a Docker Agent with a DaskExecutor. The flow runs fine for awhile, then I get an error from the CloudFlowRunner
    Unexpected error: KilledWorker('{{TaskRunName}}', <WorkerState '<tcp://127.0.0.1:44295>', name: 1, status: closed, memory: 0, processing: 72>)
    Any idea what might be causing this?
    k
    21 replies · 2 participants
  • k

    Kyle McChesney

    06/21/2022, 10:42 PM
    is there a trigger like “at least one successful, and the rest skipped”. I have some logic like so
    @task
    def required():
        pass
    
    
    @task
    def optional():
        pass
    
    
    @task
    def final():
        pass
    
    
    with Flow('example'):
        res = required()
        with case(some_case, True):
            optional_res = optional()
    
        final(res, upstream_tasks=[optional_res])
    If optional is skipped, its okay to run final. If required fails, or optional fails, I dont want to run final
    k
    13 replies · 2 participants
  • k

    Kyle McChesney

    06/21/2022, 10:42 PM
    the example is simple, but final would have additional upstream dependencies, which would be treated like res
  • c

    Chris Goddard

    06/22/2022, 1:58 AM
    Hey there! I’m trying to figure out the easiest way to trigger a prefect cloud (1.0) job via the API - I know I can use the GraphQL API but I don’t have a ton of experience with GraphQL and I want to trigger a flow run based on knowing the flow name — not the id. I also need to trigger a flow on the latest version of the flow — and there will be several versions. This all needs to take place within the context of an automated build process (GCP cloud build) so I don’t want to have to install the cli or anything - I just want to drop in an API key via a secret and run a simple script to trigger the flow at the end of a build. Ideally it would be just a bash/curl script but I could also write a basic python script to do it.
    k
    1 reply · 2 participants
  • b

    Ben Muller

    06/22/2022, 2:01 AM
    Hey - I have a flow that has a few clocks eg
    au_harness_win_place = CronClock(
        cron="33 * * * *",
        parameter_defaults={
            "race_types": ["Harness"],
            "market_type_codes": ["WIN", "PLACE"],
            "event_type_ids": None,
            "market_countries": ["AU"],
            "table_name": "harness_market_catalogue_au"
        },
        start_date=datetime.now(tz=timezone("Australia/Brisbane")),
    )
    that is called like so
    with Flow(
        name="betfair_market_catalogues",
        storage=Storage.in_a_s3_bucket(),
        run_config=RunConfig().fargate_on_ecs(),
        schedule=Schedule(clocks=[au_nz_greys_win_place, au_harness_win_place]),
        executor=LocalDaskExecutor(scheduler="threads"),
    ) as flow:
    If I want to call this via the cli it doesnt like the None values and the python objects, do I have to accept stringified versions of this or does prefect handle this on its own? eg, how would I run this locally
    prefect run -p flows/market_catalogues/betfair_market_catalogues.py --param table_name="harness_market_catalogue_au" --param race_types=["Harness"] --param market_type_codes=["WIN", "PLACE"] --param event_type_ids=[] --param market_countries=["AU"]
    doesnt do the job
    k
    21 replies · 2 participants
  • r

    Russell Brooks

    06/22/2022, 7:08 AM
    We get a pickle thread lock error in a task when using paramiko. This is probably a problem with pickling ssh stuff. What is the alternative? How are people using paramiko with Prefect?
    k
    j
    5 replies · 3 participants
  • y

    yair friedman

    06/22/2022, 11:27 AM
    hi , i am executing my flow not from main flow. when i run it on the default executor everything is running file, when i use DaskExecutor it crashs with the following message 2022-06-22 14:06:40,649 prefect.TaskRunner INFO Task ‘train_model[0]’: Starting task run... 2022-06-22 14:06:40,650 prefect.TaskRunner INFO Task ‘train_model[1]’: Starting task run... The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. 2022-06-22 14:06:55.255 python[95140:4859256] * Terminating app due to uncaught exception ‘NSInternalInconsistencyException’, reason: ‘NSWindow drag regions should only be invalidated on the Main Thread!’ please advise.
    k
    13 replies · 2 participants
  • g

    George Shishorin

    06/22/2022, 12:20 PM
    Hello, community! I’m trying to build a flow with docker tasks. So below is some information: • prefect server • prefect docker and local agents are started • task with pulling image from hub.docker.com (it’s pulling ok) • task that create container • task that run container But when I run flow locally there is an error (screen). Could you please help with it? P.S. code in thread
    k
    17 replies · 2 participants
  • s

    Suresh R

    06/22/2022, 12:23 PM
    We are using Prefect cloud and We want to enable task level SLA. Whenever the task in the flow is delayed, We want to get notification. is that possible?
    k
    1 reply · 2 participants
  • m

    Michael Hadorn

    06/22/2022, 12:27 PM
    I have a question about: restarted flows with trigger:any_successful tasks.
    k
    m
    7 replies · 3 participants
  • t

    Toby Rahloff

    06/22/2022, 12:55 PM
    What is the best practice for implementing fan-out-to-fan-in patterns in Prefect 2.0? As a test scenario, I tried to create a flow that 1. Gets all AWS S3 bucket names 2. Spawns a task per bucket and counts the objects 3. Collects all count outputs and calculates the sum (Code is in the first comment) I get some errors (also in the comment) but I have the feeling that I'm using Prefect 2.0 in a way that might be wrong. How should I structure my flow to do it in the most Prefect-y way? I loosely based the idea on this amazing Prefect blog post.
    s
    8 replies · 2 participants
  • a

    Aditi Bhalawat

    06/22/2022, 1:16 PM
    Hey!I am facing issue while running prefect on VM instance of GCP.I am unable to open UI with external_IP:portNumber. Could anyone suggest solution to it?
    k
    5 replies · 2 participants
  • j

    Jason White

    06/22/2022, 1:34 PM
    Is there an updated timeline for the release of prefect 2.0? I see here
    The Prefect 2.0 beta period is expected to last for at least the second quarter of 2022.
    And that 2.0 is not recommended for production at this time. Is this still accurate? For context, we are looking to build an internal tool using prefect as the orchestrator and are debating whether to start with 2.0 or use 1.0 and migrate. We would be hoping to start using the tool Q3 this year.
    k
    1 reply · 2 participants
  • a

    Abhishek Mitra

    06/22/2022, 1:45 PM
    I am trying to run a flow through an ECS Agent on FARGATE. I have successfully deployed the prefect agent on cluster. But I'm unable to run a registered flow. I'm using prefect 1.2.2. Logs only say
    Submitted for execution: Task arn:
    Can anybody suggest a solution?
    k
    8 replies · 2 participants
  • z

    Zheng Cheng

    06/22/2022, 2:00 PM
  • z

    Zheng Cheng

    06/22/2022, 2:01 PM
    Hi maybe this should be more specific, not everyone knows how container works. here should be clear that it’s a Docker container. instead of just “container”
    k
    1 reply · 2 participants
Powered by Linen
Title
z

Zheng Cheng

06/22/2022, 2:01 PM
Hi maybe this should be more specific, not everyone knows how container works. here should be clear that it’s a Docker container. instead of just “container”
k

Kevin Kho

06/22/2022, 2:07 PM
Hi Zheng, this is the first time we got feedback around this. Will relay feedback to docs writer who is out right now. Or you’re welcome to make a pull request for it too.
👍 1
😁 1
View count: 3