https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • n

    Nitin Bansal

    02/28/2022, 1:58 AM
    Collecting prefect Could not find a version that satisfies the requirement prefect (from versions: ) No matching distribution found for prefect
    k
    m
    15 replies · 3 participants
  • n

    Nitin Bansal

    02/28/2022, 1:59 AM
    Getting this error while executing : pip install prefect (on Ubuntu)
  • n

    Nitin Bansal

    02/28/2022, 1:59 AM
    Any resolution?
  • o

    Ovo Ojameruaye

    02/28/2022, 6:40 AM
    Hi, my state of my flow is set as successful even though task fail. I am not sure why this happens. I have a parent flow which create and runs multiple child flows. A task in a child flow fails but the child flow is marked as successful so other downstream child flows run when they shouldn't. Each child flow (1-5) has a create_flow_run and wait_for_flow_run(). Child flow 2 and 5 wait_for_flow_run() have been set has reference task so the parent flow is successful if they both complete. A task fails in child flow 3 but the flow is marked as successful
    a
    4 replies · 2 participants
  • t

    Tomer Cagan

    02/28/2022, 7:38 AM
    Orion question (applies to 1.0 as well I guess): In the docs here, it is stated that tasks can only be called from a flow and not other tasks: "All tasks must be called from within a flow. Tasks may not be called from other tasks." So this means I can only do dynamic task spawning from within the flow function (Flow block in 1.0)? Also, I was working yesterday through the Orion's getting started guide and had an error and found a few typos - where should I report this?
    a
    m
    8 replies · 3 participants
  • s

    Stephen Lloyd

    02/28/2022, 9:06 AM
    Hi. I have a flow that I have successfully registered. When I do a quick run, nothing happens. What might I be missing? I followed this tutorial with my own logic. My run_config..
    RUN_CONFIG = ECSRun(labels=['s3-flow-storage'],
                        task_role_arn=TASK_ARN,
                        execution_role_arn=EXECUTION_ARN,
                        image='<http://1111111.dkr.ecr.us-east-1.amazonaws.com/prefectdemo:latest|1111111.dkr.ecr.us-east-1.amazonaws.com/prefectdemo:latest>',
                        memory=512, cpu=256)
    a
    h
    10 replies · 3 participants
  • e

    E Li

    02/28/2022, 3:34 PM
    Hello, does prefect save log files to certain local directory?
    k
    1 reply · 2 participants
  • g

    Greg Adams

    02/28/2022, 4:07 PM
    @Anna Geller @Kevin Kho (or anyone else) - Just curious, has there been anyone making a Pandas task library? I seem to be making my own as I go along, but just making sure I'm not doing it the hard way
    a
    k
    +1
    11 replies · 4 participants
  • d

    David Michael Carter

    02/28/2022, 5:03 PM
    I have a task which I am attempting to parallelize. The function I am mapping returns a small dataframe. I need to append all of these small dataframes into a final result which i will send to the next task. So far, I am attempting to append to a global DF within the flow steps (no luck):
    finalDF = finalDF.append(foo.map(aList))
    
    TypeError: cannot concatenate object of type '<class 'prefect.tasks.core.function.FunctionTask'>'; only Series and DataFrame objs are valid
    Am I on the right track here, or do I need a completely new approach?
    k
    2 replies · 2 participants
  • d

    Dexter Antonio

    02/28/2022, 5:27 PM
    I started using prefect cloud to manage my workflows. I like the UI but I would like to be able to perform prefect cloud tasks with python code. Specifically I am attempting to do the following three things: 1. Start a flow, which I registered on prefect cloud with Python code 2. Fetch the parameters used to run a flow from prefect cloud with Python code/GraphCL 3. Change the parameters and then rerun the flow The documentation on how to achieve these tasks is a little sparse, but I think it is possible. How can I go about achieving these goals?
    a
    k
    10 replies · 3 participants
  • e

    Evgeny Liskovets

    02/28/2022, 5:54 PM
    Sorry, wrong slack lol
    k
    5 replies · 2 participants
  • d

    Daniel Komisar

    02/28/2022, 6:01 PM
    Hi everyone, I have a question about how to correctly search for flows/tasks in a given state using the API. What I’d like to do is search for all flow runs that have finished. If I use
    flow_run(where: {state: {_eq: "Finished"}})
    that returns nothing, I think because it is using string matching. Is the set of strings that can appear in
    state
    listed somewhere? Thank you!
    k
    a
    6 replies · 3 participants
  • e

    Edvard Kristiansen

    02/28/2022, 7:26 PM
    Airbyte Task Operator Naming I am attempting to set up a flow which triggers a number of Airbyte pipelines using the pre-made Prefect Airbyte task. The run is working as expected using these docs as guide(https://docs.airbyte.com/operator-guides/using-prefect-task). However, all the tasks are called AirbyteConnectionTask which makes it harder to troubleshoot errors etc. This is the code:
    from prefect import Flow, task
    from prefect.tasks.airbyte.airbyte import AirbyteConnectionTask
    from datetime import timedelta
    from prefect.schedules import Schedule
    from prefect.schedules.clocks import CronClock
    from prefect.executors import LocalDaskExecutor
    
    
    schedule = Schedule(clocks=[CronClock("0 0 4 * * *")])
    
    
    pipeline1 = AirbyteConnectionTask(
        airbyte_server_host="localhost",
        airbyte_server_port=8000,
        airbyte_api_version="v1",
        connection_id="xx-xx-xx")
    
    pipeline2 = AirbyteConnectionTask(
        airbyte_server_host="localhost",
        airbyte_server_port=8000,
        airbyte_api_version="v1",
        connection_id="xx-xx-xx")
    
    pipeline3 = AirbyteConnectionTask(
        airbyte_server_host="localhost",
        airbyte_server_port=8000,
        airbyte_api_version="v1",
        connection_id="xx-xx-xx")
    
    pipeline4 = AirbyteConnectionTask(
        airbyte_server_host="localhost",
        airbyte_server_port=8000,
        airbyte_api_version="v1",
        connection_id="xx-xx-xx")
    
    
    
    with Flow("dwh Sync", schedule, executor=LocalDaskExecutor()) as flow:
        flow.add_task(pipeline1)
        flow.add_task(pipeline2)
        flow.add_task(pipeline3)
        flow.add_task(pipeline4)
    
    # Register the flow under the "airbyte" project
    flow.register(project_name="airbyte")
    I have already tried to use the @ task operator along with a function, but the task then finishes in a second without actually triggering the pipeline.
    a
    k
    5 replies · 3 participants
  • j

    Jacqueline Riley Garrahan

    02/28/2022, 7:38 PM
    I have saved a flow to a docker registry and now would like to load and register with my prefect server. How can I load from the deployed image?
    k
    2 replies · 2 participants
  • j

    Josh

    02/28/2022, 8:38 PM
    Is there any functionality for Prefect to notify me if an agent stops querying? I have notifications right now set up from tasks if a task fails, but if the agent never kicks off the flow because it stopped querying, I’ll never know.
    ➕ 1
    k
    8 replies · 2 participants
  • m

    Meet Patel

    02/28/2022, 8:40 PM
    I am having issue installing prefect. It throwing an error of “could not find a version that satisfies the requirement dask>=2021.06.0(from prefect) can anyone help me on this ?
    k
    2 replies · 2 participants
  • d

    Daniel Burkhardt

    02/28/2022, 9:22 PM
    Hi all, what's the best practice around using RDS postgres database authentication for Prefect Agents on AWS? Do folks use IAM database authentication or Postgres Native Authentication?
    a
    a
    2 replies · 3 participants
  • a

    Apoorva Desai

    02/28/2022, 10:08 PM
    Hello, I am trying to run a prefect flow triggered by when there is a change in a github repository that is NOT the repo that the prefect flow lives in. TLDR: pull request on repository A triggers prefect flow living in Repository B. This is the closest I've come to figuring this out: https://docs.prefect.io/core/pins/pin-08-listener-flows.html Has anyone done this before or have instructions?
    k
    2 replies · 2 participants
  • v

    Victor Xie

    02/28/2022, 10:11 PM
    Hi, I was wondering whether Prefect Cloud allows one to export all existing data and them import to an on-premise Prefect Server. This is desirable if one starts with Prefect Cloud but later find on-premise setup is a better choice for whatever reason.
    k
    2 replies · 2 participants
  • s

    Scott White

    02/28/2022, 10:59 PM
    Getting a bizarre error when creating a project with the CLI. The referenced project (
    veg_management
    ) was recently deleted with
    prefect delete project veg_management
    because I had some old flows in there I wanted to clean-up. But now when I attempt to recreate it I get the errors below. Note that if I change the name of the project (e.g.
    veg_management_
    ) then it works.
    k
    4 replies · 2 participants
  • c

    Chris Marchetti [Datateer]

    02/28/2022, 11:01 PM
    Hello! We were getting an error during a github action that deploys our pipeline:
    prefect.exceptions.AuthorizationError: Malformed response received from Cloud - please ensure that you are authenticated. See `prefect auth login --help`.
    We changed from using an auth token to api key as a result of the error and are our flows are hanging. Are there changes that need to be made to our agents or to using service accounts? Thanks for the help.
    k
    m
    5 replies · 3 participants
  • a

    Apoorva Desai

    03/01/2022, 9:08 AM
    Hello! How do I access the string value in a Parameter object? I'm passing a string Parameter to a flow throw the CLI as
    prefect run --watch --id flowId --param strparam="paramValue"
    . Then in the flow.py file, I am initializing outside the flow
    strparam = Parameter("strparam", default "hello")
    but I'm unable to access
    "paramValue"
    . Prefect only sees this
    <Parameter: strparam>
    a
    2 replies · 2 participants
  • y

    Yas Mah

    03/01/2022, 9:31 AM
    Hello :) is there any working example which demonstrate the usecase of packaging flows in Python packages and then importing them e.g. as a subflow or just import the flow and run it from an other script? Thank you for your support!
    a
    1 reply · 2 participants
  • a

    Adam Roderick

    03/01/2022, 1:31 PM
    After upgrading to v1, I am seeing an error when my ECS Agent attempts to start a new task. Any idea why?
    a
    k
    14 replies · 3 participants
  • d

    damien michelle

    03/01/2022, 3:05 PM
    Hi, I'm new to prefect and I am facing an issue. I have a flow A which takes a country_code as a paremeter. I would like to know how to map the flow A to a list of country_code ? Currently, I was thinking triggering the flow A by mapping the flow A to each country_code specified in a flow B (as shown in the images) Thanks by advance
    k
    7 replies · 2 participants
  • e

    Edvard Kristiansen

    03/01/2022, 3:05 PM
    Referencing Local Files in Code I am attempting to reference some key files that are stored locally on the VM where the flows were registered from, but prefect cant seem to find the files in the local file system. I have tried to change the storage to local with no luck. Sorry if I have missed something very basic here! It runs fine on my local machine with similar file references(have adapted the code to fit with the VMs file locations before registering the flow).
    @task
    def run():
        script....
    
    key_location = "~/prefect/Prefect/keys/key.json"
    
    with Flow("pipeline", executor=LocalDaskExecutor(), run_config=LocalRun(labels=["local"]), storage=Local()) as flow:
        run()
    I am getting error: FileNotFoundError: [Errno 2] No such file or directory: Any ideas?
    k
    8 replies · 2 participants
  • y

    YD

    03/01/2022, 6:14 PM
    Getting Error when trying to pass data from task What is the proper way to do this ? See code in Thread
    m
    k
    18 replies · 3 participants
  • t

    Tony Yun

    03/01/2022, 6:56 PM
    Hi, is there a way this
    --param a=2
    can accept a parameter name with a space in it? I only find this from the doc, where in my case the parameter is called “config list”:
    $ prefect run --id "d7bfb996-b8fe-4055-8d43-2c9f82a1e3c7" --param a=2
    k
    2 replies · 2 participants
  • m

    Max Lei

    03/01/2022, 7:23 PM
    I'm trying to test
    DaskExecutor
    , would
    DockerRun
    be a good method to see if the dockerfile works or not?
    a
    k
    +1
    37 replies · 4 participants
  • s

    Stephen Herron

    03/01/2022, 7:59 PM
    Hi - I’m having trouble implementing a “flow of flows” pattern on ECS. My flows, individually are running fine - but when using the pattern all my
    wait_for_flow_run
    fails. Not sure why that is - but the error is:
    "An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Role is not valid"
    Something extra (or missing) required for the ECS task definition/role maybe?
    k
    4 replies · 2 participants
Powered by Linen
Title
s

Stephen Herron

03/01/2022, 7:59 PM
Hi - I’m having trouble implementing a “flow of flows” pattern on ECS. My flows, individually are running fine - but when using the pattern all my
wait_for_flow_run
fails. Not sure why that is - but the error is:
"An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Role is not valid"
Something extra (or missing) required for the ECS task definition/role maybe?
Are those trying to register under a different role
k

Kevin Kho

03/01/2022, 8:01 PM
Hi @Stephen Herron, are the subflows actually running when you go to their Flow pages?
s

Stephen Herron

03/01/2022, 8:06 PM
No - but thanks I think I can see the issue now. I registered with a different label, so that agent’s infra is probably not quite correct
k

Kevin Kho

03/01/2022, 8:06 PM
Ah ok. Nice work! I was useless here haha
View count: 11