Excited about Orion. I have a question/feature request, which I think may make a lot more sense in the context of Orion than original Prefect... If I want some tasks within a flow to operate with a shared filesystem, there's not easy way to do that in Prefect if the flow as a whole is running in a distributed fashion. However, perhaps in Orion you could have some sub part of the flow which runs as though that bit of the flow was "local", i.e. you could do

[download from s3] => [unzip] => [do something interesting] => [upload to somewhere]

as a series of tasks that run sequentially on a single filesystem, but as part of a larger flow where tasks are run on entirely separate filesystems. The advantage of allowing these steps to be full-blown tasks rather than just functions is that you can get dashboard level visibility, retries, and better use of library tasks etc. A related thought - could you compile a dbt graph into a nested part of a prefect flow, so you get full visibility into what dbt is doing at the level of Prefect. That would be cool.

Just came to say HOLY s&*t Orion looks 🍌s! So excited to be a Prefect customer right now.

Also very exited about Orion 🎉, I am wondering if there is an example/documentation on generate new tasks or entire workflows in response to streaming events?

Hi Everyone, I'm currently trying to deploy prefect server on a server with running VPN (I'm using NordVPN). After quite a while I faced problem where I got the error

ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

when executing

prefect server start

. One solution that's recommended is to add the subnet on docker yaml but I couldn't find a way to do that on prefect. Are there any recommendation on this case (deploy on machine with VPN) or any way to edit the

docker-compose.yml

?

Wondering if anyone had ever implemented an SFTPResult or similar before?

Hi is there easy way to configure https://www.psycopg.org/docs/connection.html#connection.autocommit in PostgresqlTask or do I need to write my own task to do that?

Hey guys, normally I register flows on the VM running the local agent. If I try to register a flow to the Prefect cloud from another machine, I get a connection error (see screenshot). Any ideas?

Hi, in the documentation for the ECS agent it's not clear what the configuration should be regarding security groups (or actually specific IAM permissions). I'm assuming you don't need to open the agent to the public internet and that it just polls Prefect Cloud, but it's not entirely clear; also, if anyone has a minimal set of IAM permissions I can apply to the task role, that would be great 🙂

Hi all, How can one use a single Docker image to add multiple flows as scripts? I want the flows to be dynamic, as in be able to load env variables at task/flow execution, not at registration. I followed this but this registers a pickled flow.... If I define the storage at the flow level, I can use the

stored_as_script

argument in Docker() along with

path

, but cannot see how I can avoid building and pushing an image for each flow that way...

Hi all, I’m currently looking into data processing frameworks. One of the requirements my team has is to be able to integrate with an exterior service in a processing pipeline. For example if we have a tensorflow model being served with TFServe or something like that, how can a call to such a service be done within a prefect workflow? T1 -> TFServe -> T3

The new Orion architecture looks promising -interesting to see the evolution of data workflow tools away from the DAG model and converging with the generic massively modular workflow tools (e.g. Amazon Simple Workflow Service, cadenceworkflow.io, temporal.io etc.). Those tools are very powerful but always felt too low-level for simple workflows. I see prefect well positioned to bridge the gap -one tool that can span the entire workflow complexity spectrum. Keep the simple stuff simple but unlock increasing complex/dynamic needs 😉

Orion feature request/idea: now that the tool is ‘free from the DAG’ I could image that supporting new ‘execution’ layers besides Dask will be much easier. Has the team thought about added native support for Ray.io clusters?

I had a flow that worked flawlessly for a while, and then it began to issue errors like the ones on the image. I have no clue, where to continue debugging. All tasks succeed, but the flow becomes failed because of these messages. Any hints are appreciated. I have already restarted Prefect server and the agent.

Let me ask one more. I have 2 set of tasks, and I'd like to execute the second set after the first set finished. There's nothing to be passed from the first set to the second, to setup dependence or order. I guessed that set_upstream() should be used, but still the second set starts before the first set finishes. How could I handle that?

Hey team - Is this expected behavior? I have the following setup: 1. a ECS agent with label “prod” 2. a Kubernetes agent also with label “prod” 3. run_config:

flow.run_config = ECSRun(labels=['prod'],
        image='xxxxxxxx',
        env={"EXTRA_PIP_PACKAGES": "retry"}
    )

I was expecting that although the Kubernetes agent has the label “prod”, prefect scheduler should still only route flow runs to the ECS agent. However, looks like that’s not the case. Error:

Flow run ad4bce7b-29cb-4e42-a8c3-eeb9a75f7999 has a `run_config` of type `ECSRun`, only `KubernetesRun` is supported

message has been deleted

Hey team - I was wondering what are the ways to auto register flows? I am thinking of something related to git hooks(that watch the folder inside the repository where the code for my prefect project sits) to trigger a register script? Are there better ways?

What's a good way to keep Automations version-controlled? Maybe a Flow that has GraphQL queries to create Automations? And just re-run it every time you add a new one to the file?

Hey folks, does someone know of a good task they use to copy relatively large files (~100s of MBS) from an sftp server ? Curious to know which python tool you are using

Re Orion: is there a plan to announce a detailed roadmap about when features such as logging, mapping, or the task library will be added to Orion? I’m currently helping plan a new workflow system using Prefect that might be used in prod but not for some time yet, and we’re not sure if it would be better to start on Orion or start on 1.0 and migrate later.

About Orion: last night, I have read through the blog post and the documentation and it absolutely blew my mind. This is exactly what I wanted to have for a personal ML project of mine! One thing was I was confused about was the concept of deployments (see screenshot attached). So does that mean that flows should run on the same infra as the Orion server?

I'm upgrading from an OG developer legacy plan to standard. Other than losing some flow history (30 days to 14), I shouldn't expect any impact on my existing flows should I? (Sorry having trouble finding any documentation on this)

Would like to check with you, is the Object Cloud Storage on IBM cloud an option to store the flow? I don’t see it in the docs, just wondering. Thanks.

Hi all! Does anyone have an example of how to use prefect to connect to an existing, external PostgreSQL database? (Mine also happens to require an SSH connection.) Two main things I'm struggling with: • can I return the result of 

conn = psycopg2.connect()

from a task back to the flow, and then use it as an argument in another task? • if so, should I consider separating out the SSH and database connections into separate tasks? • using PrefectSecret appropriately --- I currently have a

.env

file and a task to get the environment variables and then use the results as a parameter in my other functions, but I suspect this isn't quite right.

Prefect runs the tasks and flows in my own infrastructure. If my infrastructure is not my local Pycharm, but need to be an env that runs 24X7, like a kubernetes cluster on any cloud, I have seen example like this Deploy Code to AWS Kubernetes with Prefect: a Step-by-Step ... But it isn’t written in a way that I can adapt to another cloud, for example, a Kuberentes cluster on IBM cloud. Does someone here have an example for me to follow?

Hey again, just wondering if anyone had ever thought of / requested Parameters that could have a limited input set? Like being able to configure the parameter to accept a single value from a pre-defined, or runtime generated list?

Hi everyone, I am trying to run tasks in parallel (multiple tasks at same time). I get the following error:

Failed to load and execute Flow's environment: FlowStorageError('An error occurred while unpickling the flow:\n  AttributeError("Can\'t get attribute \'RandomNum\' on <module \'prefect.__main__\'

I am able to run the task locally

when i trigger my flow through prefect cloud it gives me the above error