What is the standard way to set up prefect code to handle promotion through various environments? (dev -> stage -> prod). I’m using Prefect Cloud and would like to use the Secrets manager.

Hi 🙂 I'm facing an error on an HPC cluster with my hello_world only with prefect 2:

Hi all 😎 I'm going through the Orion docs and noticed that the "states" link under the FAQ > Deployment > How is Orion organized? section leads to a 404 page (w/ URL = "https://orion-docs.prefect.io/faq/concepts/states") Not sure if this is the appropriate place to post this (please redirect me if not!), I believe the appropriate link is to the "states" page is "https://orion-docs.prefect.io/concepts/states/#overview"

Hi there, noob question here, there is an http api in prefect? I wan't to get the prefect task status on my zabbix, with an http request to prefect, so I can create trigger when something goes wrong, didnt find anything about in the docs

Hi everybody, It seems like our team is going to have tons of flows. Is there any ideas how to create a hierarchy of the flows or to arrange them somehow? Thank you.

Hi there, is it possible to use https://docs.prefect.io/api/latest/tasks/files.html file operators with files hosted on SFTP with user/pass? Or another to use prefect to fetch files from SFTP?

Hi guys. We're trying to set our log format to log everything JSON so we can consume it in Datadog and search the logs. We'd targetting the following format:

{
  "flow_name": "",
  "flow_id": "",
  "task_name": "",
  "task_id": "",
  "message": "",
  "customer_id": ""
}

The questions we have are: 1. Is using

PREFECT__LOGGING__FORMAT

the correct place to specify the JSON format? 2. How can we handle this additional attribute

customer_id

. Our intent here is for most tasks that are related to our customers to do something like

<http://logger.info|logger.info>('Something', customer_id)

3. To the previous point, how can we extend this further such that we can add arbitrary parameters or objects to the log,

<http://logger.info|logger.info>('Something', my_object,my_second_object)

and have them show up nsted in the json?

Hello all Is Prefect can be entirely installed on Kubernetes? If yes, is there a guide somewhere? To what I've understood, only some parts of the architecture can run in K8S. Am I wrong? Thank you

See you at 3:15p Eastern today! https://prefect-community.slack.com/archives/C036FRC4KMW/p1649259298561779

Hi everyone! I’m trying to use an S3List task https://docs.prefect.io/api/latest/tasks/aws.html#s3list but it seems like the bucket prefix is not an optional value even though it should be? 🤔 Boto3 docs lists it as optional (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.list_objects), but the task implementation doesn’t (see screenshot). This is what I try to do:

s3_list_task = S3List()

        flow.set_dependencies(
            task=s3_list_task,
            keyword_tasks=dict(
                bucket='my-bucket-name'
            )
        )

And this is the error message:

TypeError: run() missing 1 required positional argument: 'prefix'

Does this mean I can’t use buckets without prefixes together with the S3 prefect tasks?

Total rookie question here -- I'm attempting to install and play around with Prefect 2.0 in PyCharm, having some trouble verifying the install was successful. When I attempt to check the version in PyCharm's terminal I receive the following error messages. I receive similar error messages when attempting to run simple python scripts listed in the First Steps tutorial. I created a python project in PyCharm and installed Prefect 2.0b2 using their settings interface. Any idea where I went wrong? Ive attached screenshots of the error messages and my project's Prefect directory.

We need alerting when a flow is submitted for execution but is never executed. Can we create an automation for this? Basically for every flow, send an email if a flow is submitted but not started after 60 seconds. Is this possible?

hey, someone can help me with a query in graphql api? I wan't to collect all created flows in one query, but I wan't ONLY the LASTEST version of each, how can I do so?

Is possible to debug flows like celery? Example: I have my flows registered and i want to be the one that will pull the flow to run (with debug) in pycharm

PrefectLive is starting in 15 minutes! See you in chat! https://prefect-community.slack.com/archives/C036FRC4KMW/p1649259298561779

Hi. I'm just starting to look at Prefect 2.0. Is there a suggested way of handling a waiting period mid flow? For example if I start a flow, and after an initial task I want to wait for 5 minutes before continuing, do I just sleep for 5 minutes in a task? What about if I wanted to wait for days?

is there a way programatically to get a list of deployments in prefect? i know we can do

prefect deployment ls

but i want to get all my created deployments...1000 of them and run them all. so im assuming i have to programatically execute prefect deployment run deploymenet_name for each deployment in my deployments

Hi, I have two projects in Prefect Cloud: one for dev & one for prod. Each project has its own a Docker agent using DockerRun for running flows in containers (in EC2) built based on images from Dockerhub. These images are being re-built with a CI/CD flow (using a Dockerfile) whenever we push code to our Git repository either in dev or prod (one image tag for dev and one for prod) The flow source code is stored in Github and this is configured as the storage for our flows. For some reason my flow runs in the

prod

project suddenly fail with

Failed to load and execute flow run: ModuleNotFoundError("No module named 'project_name'")

however, all the flows with exactly the same source code in Prefect

dev

project run just fine. When I pull these images and start containers locally the project structure and packages are the same for both dev and prod. What could be the issue here?

Strictly speaking, do Tasks need to run in an environment that has the original "Task" function source code? Cloudpickle/pickle would lead me to think that the "bare minimum" necessary would be just an environment with task's py dependencies, but not the task source code themselves.

What is the recipe for configuring a scheduled Flow's parameters based on the "scheduled time"? I found

parameter_defaults

for Clocks, but they seem to be static w.r.t each clock. For example, I want the parameter be "day of the week",i.e. Mon, Tue....

Hey team - having some troubles running docker tasks in a docker agent on windows (dind) ..

What is the difference between lazarus and zombie killer?

What's the best approach to using a for loop that would normally be within the

main

section of a standard python script?

Is "dependent flows" a common used scenario in the real world in industry?

Does Prefect Enterprise support SAML SSO?

Hi there, I'm looking at some of the new stuff in Orion beta and I'm not sure I follow how storage works in Orion. I can use the helper to create new storage options, but I don't see how to specify which option is used beyond the "server default". Can I specify this in my deployment spec? Inside the flow runner?

I’m migrating my flow code from prefect 0.15 to orion

2.0.b2

. do you know where the

resource_manager

is located now? Before I was importing

from prefect import resource_manager

https://docs.prefect.io/core/idioms/resource-manager.html CC @Anna Geller @Kevin Kho

Hello Everyone! New user of Prefect here 🙂 I am trying to understand if using map makes sense here. Currently, I am looping through a dataframe to get values and assign them as variables to pass to tasks. The code below works fine but wanted to know is mapping a viable solution?

def prefect_flow():
    with Flow('cloud_reporting_etl') as flow:
        for column, value in jobs_df.iterrows():
            job_name = value['Job Name']
            query = value['Query']
            filename = value['Filename']
            extracted = extract_data(f"""{query}""")
            load_data_to_s3(extracted, filename)
    return flow

Hello, Is there a way (without Prefect Cloud) to setup security around prefect UI user capabilities to run/delete/read flows?

I’m running orion

2.0b2

and starting the server (on an EC2 machine) with an IP on a different port. (self-hosted). I’m able to connect to it via browser (I’ve set all the security groups and routes to allow me to connect to it on a browser). However it seems that internally Prefect Orion is still interacting with the default host and port. So the UI doesn’t render all the info (see screenshot in the thread) Is there something extra I need to change?