https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • d

    Daniel Manson

    10/05/2021, 10:29 PM
    Excited about Orion. I have a question/feature request, which I think may make a lot more sense in the context of Orion than original Prefect... If I want some tasks within a flow to operate with a shared filesystem, there's not easy way to do that in Prefect if the flow as a whole is running in a distributed fashion. However, perhaps in Orion you could have some sub part of the flow which runs as though that bit of the flow was "local", i.e. you could do
    [download from s3] => [unzip] => [do something interesting] => [upload to somewhere]
    as a series of tasks that run sequentially on a single filesystem, but as part of a larger flow where tasks are run on entirely separate filesystems. The advantage of allowing these steps to be full-blown tasks rather than just functions is that you can get dashboard level visibility, retries, and better use of library tasks etc. A related thought - could you compile a dbt graph into a nested part of a prefect flow, so you get full visibility into what dbt is doing at the level of Prefect. That would be cool.
    👀 2
    👍 1
    🤔 1
    c
    • 2
    • 2
  • j

    Jacob Blanco

    10/05/2021, 11:58 PM
    Just came to say HOLY s&*t Orion looks 🍌s! So excited to be a Prefect customer right now.
    :marvin: 12
    :party-parrot: 4
    ❤️ 10
    💯 4
  • g

    Gaylord Cherencey

    10/06/2021, 1:18 AM
    Also very exited about Orion 🎉, I am wondering if there is an example/documentation on generate new tasks or entire workflows in response to streaming events?
    ❤️ 3
    d
    c
    • 3
    • 4
  • b

    banditelol

    10/06/2021, 1:44 AM
    Hi Everyone, I'm currently trying to deploy prefect server on a server with running VPN (I'm using NordVPN). After quite a while I faced problem where I got the error
    ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network
    when executing
    prefect server start
    . One solution that's recommended is to add the subnet on docker yaml but I couldn't find a way to do that on prefect. Are there any recommendation on this case (deploy on machine with VPN) or any way to edit the
    docker-compose.yml
    ?
    k
    • 2
    • 1
  • a

    Adam Brusselback

    10/06/2021, 3:24 AM
    Wondering if anyone had ever implemented an SFTPResult or similar before?
    k
    • 2
    • 2
  • a

    Antti Tupamäki

    10/06/2021, 6:27 AM
    Hi is there easy way to configure https://www.psycopg.org/docs/connection.html#connection.autocommit in PostgresqlTask or do I need to write my own task to do that?
    a
    • 2
    • 1
  • j

    Jelle Vegter

    10/06/2021, 7:41 AM
    Hey guys, normally I register flows on the VM running the local agent. If I try to register a flow to the Prefect cloud from another machine, I get a connection error (see screenshot). Any ideas?
    a
    r
    k
    • 4
    • 23
  • w

    Will

    10/06/2021, 11:01 AM
    Hi, in the documentation for the ECS agent it's not clear what the configuration should be regarding security groups (or actually specific IAM permissions). I'm assuming you don't need to open the agent to the public internet and that it just polls Prefect Cloud, but it's not entirely clear; also, if anyone has a minimal set of IAM permissions I can apply to the task role, that would be great 🙂
    a
    k
    • 3
    • 4
  • a

    Andreas Tsangarides

    10/06/2021, 12:04 PM
    Hi all, How can one use a single Docker image to add multiple flows as scripts? I want the flows to be dynamic, as in be able to load env variables at task/flow execution, not at registration. I followed this but this registers a pickled flow.... If I define the storage at the flow level, I can use the
    stored_as_script
    argument in Docker() along with
    path
    , but cannot see how I can avoid building and pushing an image for each flow that way...
    a
    k
    • 3
    • 9
  • l

    Lucas Giger

    10/06/2021, 1:12 PM
    Hi all, I’m currently looking into data processing frameworks. One of the requirements my team has is to be able to integrate with an exterior service in a processing pipeline. For example if we have a tensorflow model being served with TFServe or something like that, how can a call to such a service be done within a prefect workflow? T1 -> TFServe -> T3
    k
    • 2
    • 3
  • b

    Bob Colner

    10/06/2021, 2:25 PM
    The new Orion architecture looks promising -interesting to see the evolution of data workflow tools away from the DAG model and converging with the generic massively modular workflow tools (e.g. Amazon Simple Workflow Service, cadenceworkflow.io, temporal.io etc.). Those tools are very powerful but always felt too low-level for simple workflows. I see prefect well positioned to bridge the gap -one tool that can span the entire workflow complexity spectrum. Keep the simple stuff simple but unlock increasing complex/dynamic needs 😉
    🚀 6
    :thank-you: 4
    :marvin: 5
  • b

    Bob Colner

    10/06/2021, 3:32 PM
    Orion feature request/idea: now that the tool is ‘free from the DAG’ I could image that supporting new ‘execution’ layers besides Dask will be much easier. Has the team thought about added native support for Ray.io clusters?
    m
    • 2
    • 3
  • a

    Andor TĂłth

    10/06/2021, 4:33 PM
    I had a flow that worked flawlessly for a while, and then it began to issue errors like the ones on the image. I have no clue, where to continue debugging. All tasks succeed, but the flow becomes failed because of these messages. Any hints are appreciated. I have already restarted Prefect server and the agent.
    k
    • 2
    • 6
  • a

    Andor TĂłth

    10/06/2021, 5:05 PM
    Let me ask one more. I have 2 set of tasks, and I'd like to execute the second set after the first set finished. There's nothing to be passed from the first set to the second, to setup dependence or order. I guessed that set_upstream() should be used, but still the second set starts before the first set finishes. How could I handle that?
    k
    • 2
    • 8
  • s

    Shaoyi Zhang

    10/06/2021, 5:38 PM
    Hey team - Is this expected behavior? I have the following setup: 1. a ECS agent with label “prod” 2. a Kubernetes agent also with label “prod” 3. run_config:
    flow.run_config = ECSRun(labels=['prod'],
            image='xxxxxxxx',
            env={"EXTRA_PIP_PACKAGES": "retry"}
        )
    I was expecting that although the Kubernetes agent has the label “prod”, prefect scheduler should still only route flow runs to the ECS agent. However, looks like that’s not the case. Error:
    Flow run ad4bce7b-29cb-4e42-a8c3-eeb9a75f7999 has a `run_config` of type `ECSRun`, only `KubernetesRun` is supported
    k
    m
    • 3
    • 5
  • k

    Kevin Kho

    10/06/2021, 7:30 PM
    message has been deleted
    :upvote: 3
  • a

    Abhas P

    10/06/2021, 8:38 PM
    Hey team - I was wondering what are the ways to auto register flows? I am thinking of something related to git hooks(that watch the folder inside the repository where the code for my prefect project sits) to trigger a register script? Are there better ways?
    g
    • 2
    • 1
  • m

    Matt Alhonte

    10/06/2021, 10:17 PM
    What's a good way to keep Automations version-controlled? Maybe a Flow that has GraphQL queries to create Automations? And just re-run it every time you add a new one to the file?
    k
    • 2
    • 2
  • m

    Marwan Sarieddine

    10/07/2021, 2:14 AM
    Hey folks, does someone know of a good task they use to copy relatively large files (~100s of MBS) from an sftp server ? Curious to know which python tool you are using
    ➕ 1
  • r

    Ryan Sattler

    10/07/2021, 3:42 AM
    Re Orion: is there a plan to announce a detailed roadmap about when features such as logging, mapping, or the task library will be added to Orion? I’m currently helping plan a new workflow system using Prefect that might be used in prod but not for some time yet, and we’re not sure if it would be better to start on Orion or start on 1.0 and migrate later.
    a
    k
    • 3
    • 3
  • m

    Matthias Roels

    10/07/2021, 8:51 AM
    About Orion: last night, I have read through the blog post and the documentation and it absolutely blew my mind. This is exactly what I wanted to have for a personal ML project of mine! One thing was I was confused about was the concept of deployments (see screenshot attached). So does that mean that flows should run on the same infra as the Orion server?
    j
    • 2
    • 3
  • s

    Sam Barger

    10/07/2021, 12:10 PM
    I'm upgrading from an OG developer legacy plan to standard. Other than losing some flow history (30 days to 14), I shouldn't expect any impact on my existing flows should I? (Sorry having trouble finding any documentation on this)
    k
    • 2
    • 2
  • z

    Zheng Xie

    10/07/2021, 2:24 PM
    Would like to check with you, is the Object Cloud Storage on IBM cloud an option to store the flow? I don’t see it in the docs, just wondering. Thanks.
    k
    • 2
    • 5
  • a

    Akiva Wolberg

    10/07/2021, 2:54 PM
    Hi all! Does anyone have an example of how to use prefect to connect to an existing, external PostgreSQL database? (Mine also happens to require an SSH connection.) Two main things I'm struggling with: • can I return the result of 
    conn = psycopg2.connect()
    from a task back to the flow, and then use it as an argument in another task? • if so, should I consider separating out the SSH and database connections into separate tasks? • using PrefectSecret appropriately --- I currently have a
    .env
    file and a task to get the environment variables and then use the results as a parameter in my other functions, but I suspect this isn't quite right.
    k
    • 2
    • 7
  • z

    Zheng Xie

    10/07/2021, 3:53 PM
    Prefect runs the tasks and flows in my own infrastructure. If my infrastructure is not my local Pycharm, but need to be an env that runs 24X7, like a kubernetes cluster on any cloud, I have seen example like this Deploy Code to AWS Kubernetes with Prefect: a Step-by-Step ... But it isn’t written in a way that I can adapt to another cloud, for example, a Kuberentes cluster on IBM cloud. Does someone here have an example for me to follow?
    k
    • 2
    • 2
  • a

    Adam Brusselback

    10/07/2021, 6:01 PM
    Hey again, just wondering if anyone had ever thought of / requested Parameters that could have a limited input set? Like being able to configure the parameter to accept a single value from a pre-defined, or runtime generated list?
    j
    • 2
    • 2
  • v

    Vamsi Reddy

    10/07/2021, 6:35 PM
    Hi everyone, I am trying to run tasks in parallel (multiple tasks at same time). I get the following error:
    Failed to load and execute Flow's environment: FlowStorageError('An error occurred while unpickling the flow:\n  AttributeError("Can\'t get attribute \'RandomNum\' on <module \'prefect.__main__\'
    k
    y
    • 3
    • 117
  • v

    Vamsi Reddy

    10/07/2021, 6:35 PM
    I am able to run the task locally
  • v

    Vamsi Reddy

    10/07/2021, 6:35 PM
    when i trigger my flow through prefect cloud it gives me the above error
  • b

    Brian Sorahan

    10/07/2021, 7:49 PM
    Hi all, not sure if this is the right channel for noob questions. I have an invite from someone in my org and it seems like it expects me to already have an account. I think I may have gotten a similar invite a while back but can’t find it in outlook. I tried going through the forgot password flow but never got an email to reset my password. I also tried just logging in through github to accept the invite but got this (see image below). Any help is greatly appreciated!
    ✅ 1
    n
    m
    • 3
    • 7
Powered by Linen
Title
b

Brian Sorahan

10/07/2021, 7:49 PM
Hi all, not sure if this is the right channel for noob questions. I have an invite from someone in my org and it seems like it expects me to already have an account. I think I may have gotten a similar invite a while back but can’t find it in outlook. I tried going through the forgot password flow but never got an email to reset my password. I also tried just logging in through github to accept the invite but got this (see image below). Any help is greatly appreciated!
✅ 1
n

nicholas

10/07/2021, 7:51 PM
Hi @Brian Sorahan - would you mind DMing me your team/company name so we can look into this?
b

Brian Sorahan

10/07/2021, 7:51 PM
Certainly
🙏 1
m

Mike Rose

06/15/2022, 2:22 PM
Hey there, I have this exact problem. What was the resolution?
n

nicholas

06/15/2022, 2:25 PM
@Mike Rose if you’re attempting to log in with GitHub, your profile will need to be public
m

Mike Rose

06/15/2022, 2:26 PM
Thanks @nicholas. Let me give that a try.
That did the trick! Thanks. Actually, just in case someone else hits this: My profile was public, but I had not selected a public email yet.
n

nicholas

06/15/2022, 2:30 PM
Great!
View count: 2