Wondering if anyone had ever implemented an SFTPResult or similar before?

Hi is there easy way to configure https://www.psycopg.org/docs/connection.html#connection.autocommit in PostgresqlTask or do I need to write my own task to do that?

Hey guys, normally I register flows on the VM running the local agent. If I try to register a flow to the Prefect cloud from another machine, I get a connection error (see screenshot). Any ideas?

Hi, in the documentation for the ECS agent it's not clear what the configuration should be regarding security groups (or actually specific IAM permissions). I'm assuming you don't need to open the agent to the public internet and that it just polls Prefect Cloud, but it's not entirely clear; also, if anyone has a minimal set of IAM permissions I can apply to the task role, that would be great 🙂

Hi all, How can one use a single Docker image to add multiple flows as scripts? I want the flows to be dynamic, as in be able to load env variables at task/flow execution, not at registration. I followed this but this registers a pickled flow.... If I define the storage at the flow level, I can use the

stored_as_script

argument in Docker() along with

path

, but cannot see how I can avoid building and pushing an image for each flow that way...

Hi all, I’m currently looking into data processing frameworks. One of the requirements my team has is to be able to integrate with an exterior service in a processing pipeline. For example if we have a tensorflow model being served with TFServe or something like that, how can a call to such a service be done within a prefect workflow? T1 -> TFServe -> T3

The new Orion architecture looks promising -interesting to see the evolution of data workflow tools away from the DAG model and converging with the generic massively modular workflow tools (e.g. Amazon Simple Workflow Service, cadenceworkflow.io, temporal.io etc.). Those tools are very powerful but always felt too low-level for simple workflows. I see prefect well positioned to bridge the gap -one tool that can span the entire workflow complexity spectrum. Keep the simple stuff simple but unlock increasing complex/dynamic needs 😉

Orion feature request/idea: now that the tool is ‘free from the DAG’ I could image that supporting new ‘execution’ layers besides Dask will be much easier. Has the team thought about added native support for Ray.io clusters?

I had a flow that worked flawlessly for a while, and then it began to issue errors like the ones on the image. I have no clue, where to continue debugging. All tasks succeed, but the flow becomes failed because of these messages. Any hints are appreciated. I have already restarted Prefect server and the agent.

Let me ask one more. I have 2 set of tasks, and I'd like to execute the second set after the first set finished. There's nothing to be passed from the first set to the second, to setup dependence or order. I guessed that set_upstream() should be used, but still the second set starts before the first set finishes. How could I handle that?

Hey team - Is this expected behavior? I have the following setup: 1. a ECS agent with label “prod” 2. a Kubernetes agent also with label “prod” 3. run_config:

flow.run_config = ECSRun(labels=['prod'],
        image='xxxxxxxx',
        env={"EXTRA_PIP_PACKAGES": "retry"}
    )

I was expecting that although the Kubernetes agent has the label “prod”, prefect scheduler should still only route flow runs to the ECS agent. However, looks like that’s not the case. Error:

Flow run ad4bce7b-29cb-4e42-a8c3-eeb9a75f7999 has a `run_config` of type `ECSRun`, only `KubernetesRun` is supported

Hey team - I was wondering what are the ways to auto register flows? I am thinking of something related to git hooks(that watch the folder inside the repository where the code for my prefect project sits) to trigger a register script? Are there better ways?

What's a good way to keep Automations version-controlled? Maybe a Flow that has GraphQL queries to create Automations? And just re-run it every time you add a new one to the file?

Hey folks, does someone know of a good task they use to copy relatively large files (~100s of MBS) from an sftp server ? Curious to know which python tool you are using

Re Orion: is there a plan to announce a detailed roadmap about when features such as logging, mapping, or the task library will be added to Orion? I’m currently helping plan a new workflow system using Prefect that might be used in prod but not for some time yet, and we’re not sure if it would be better to start on Orion or start on 1.0 and migrate later.

About Orion: last night, I have read through the blog post and the documentation and it absolutely blew my mind. This is exactly what I wanted to have for a personal ML project of mine! One thing was I was confused about was the concept of deployments (see screenshot attached). So does that mean that flows should run on the same infra as the Orion server?

I'm upgrading from an OG developer legacy plan to standard. Other than losing some flow history (30 days to 14), I shouldn't expect any impact on my existing flows should I? (Sorry having trouble finding any documentation on this)

Would like to check with you, is the Object Cloud Storage on IBM cloud an option to store the flow? I don’t see it in the docs, just wondering. Thanks.

Hi all! Does anyone have an example of how to use prefect to connect to an existing, external PostgreSQL database? (Mine also happens to require an SSH connection.) Two main things I'm struggling with: • can I return the result of 

conn = psycopg2.connect()

from a task back to the flow, and then use it as an argument in another task? • if so, should I consider separating out the SSH and database connections into separate tasks? • using PrefectSecret appropriately --- I currently have a

.env

file and a task to get the environment variables and then use the results as a parameter in my other functions, but I suspect this isn't quite right.

Prefect runs the tasks and flows in my own infrastructure. If my infrastructure is not my local Pycharm, but need to be an env that runs 24X7, like a kubernetes cluster on any cloud, I have seen example like this Deploy Code to AWS Kubernetes with Prefect: a Step-by-Step ... But it isn’t written in a way that I can adapt to another cloud, for example, a Kuberentes cluster on IBM cloud. Does someone here have an example for me to follow?

Hey again, just wondering if anyone had ever thought of / requested Parameters that could have a limited input set? Like being able to configure the parameter to accept a single value from a pre-defined, or runtime generated list?

Hi everyone, I am trying to run tasks in parallel (multiple tasks at same time). I get the following error:

Failed to load and execute Flow's environment: FlowStorageError('An error occurred while unpickling the flow:\n  AttributeError("Can\'t get attribute \'RandomNum\' on <module \'prefect.__main__\'

I am able to run the task locally

when i trigger my flow through prefect cloud it gives me the above error

Hi all, not sure if this is the right channel for noob questions. I have an invite from someone in my org and it seems like it expects me to already have an account. I think I may have gotten a similar invite a while back but can’t find it in outlook. I tried going through the forgot password flow but never got an email to reset my password. I also tried just logging in through github to accept the invite but got this (see image below). Any help is greatly appreciated!

In order to integrate slack (through the

state_handler

) with prefect cloud, does the webhook URL have to be stored in Prefect Secrets, or can it just be available as an environment variable (my use case would be loading an eternal secret to an eks cluster)

I'm trying to send out notifications when a flow fails by defining a callback setting the flow's

on_failure

to it. The callback has (according to the docs) the following signature

def f(obj: Union[Task, Flow], state: State) -> None:

My quest is this: how to identify the failed task from the (Flow) obj. I see I can iterate over the flows' tasks, I'm just not sure how to identify the task's state

Hey guys, I'm just curious if there's been any changes recently to authenticating and POSTing to Prefect Cloud? I've recently been getting some AuthenticationErrors indicating Forbidden and haven't recently changed auth tokens

Hello Community! We are getting this error quite often in the last weeks:

KeyError: 'The secret <some_secret> was not found.  Please ensure that it was set correctly in your tenant: <https://docs.prefect.io/orchestration/concepts/secrets.html>'

Now this happens in a mapped task, only for one or two of those mapped tasks, others work fine. The secret exists and is clearly usable. Re-running the flow usually helps, however we are not enjoying that re-running part so much. Any advice?

Hello, at 17:22, Oct 7 2021 (UTC-5) we have

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: <https://api.prefect.io/>

when flow try create other flow. Was it a common problem or a problem on our side?