What options do we have for decommissioning a flow we no longer use? Can we archive it without deleting it? Can we do any of this via the API?

Hi everybody, Does anybody know how to make a one-time artifact for a flow run? Love this feature but it is a little pain to delete them)) I would really appreciate to hear your experience. Thank you in advance

Hello everyone! I am very new to Prefect and have already started to build my new Flow.🙌 When trying to work with Prefect Secrets locally, using the following type of Secrets syntax in my local flow should run as well :

from prefect.tasks.secrets import PrefectSecret

@task
def my_task(secret_credentials):
    return secret_credentials

with Flow('test') as flow:
    credentials = PrefectSecret('NAME_OF_SECRET')
    my_task(credentials)

According to this documentation https://docs.prefect.io/orchestration/concepts/secrets.html#setting-local-secrets to be able to set these local secrets I have to 1. Create a custom config file by running:

cd ~/.prefect/ && touch config.toml

2. Add the local secrets into this file But the “.prefect” file doesnt exist on my machine. I have tried to uninstall and install prefect again and it doesnt generate. Any idea why this is happening and what I should do to create this file? Thanks in advance for the help! //fyi @Henrietta Salonen @Maximilian Lutz

Hi, if I have 2 sub-flows in a main flow like below, can the flow_b get the output of flow_a as its input?

with Flow(...) as flow:
    data = extract_some_data()
    flow_a = create_flow_run(…, parameters={'param-key':data})
    flow_b = create_flow_run(…,parameters={'param-key':output_from_flow_a})

Join me in 5 minutes for a Meetup with the MLOps community where I will demo using Prefect with Pandera, Tune, and Evidently AI. Link

is there a way i can avoid the superfluous

List

and

Dict

tasks in my DAG? this is taken from a barely modified example on prefect’s blog (code in thread)

Hello,

Quick question is there a way to remove/delete a flow from a project?

guys need to evaluate airflow vs prefect vs dagster which one should we go for?

Hey all! Our team is going to try and setup prefect to be more "proper". We are going to use

DockerStorage

, if we can, so that registering flows brings along dependencies. We also want to use

KubernetesRun

so that prefect can schedule jobs on k8s. For the latter, however, is

KubernetesRun

still the right thing to do? Someone outside this community mentioned to us about dask for scheduling instead of the "kubernetes scheduler" (his words; I'm still new to this and figuring it out). Basically, which way is the "right way"? Adding dask isn't something we're opposed to doing, but we have a lot of stuff going on and are aiming for the lowest effort path, haha. I promise we're not lazy, just overworked.

Hello, for SQLServerFetch I'm receiving a login failed error due to invalid connection string attribute (0). Compared code with a teammate who can successfully fetch so not entirely sure the issue.

mssql_fetch = SQLServerFetch(db_name='db', user='user', host='host', fetch='all')

Join us tomorrow on February 24, 2022 at 11AM PST (2PM EST) to learn how to generate end-to-end lineage graphs of your data pipelines using Monte Carlo and Prefect so you can reduce the time to detection and resolution for critical data incidents: https://www.montecarlodata.com/how-to-build-more-reliable-data-pipelines-with-monte-carlo-and-prefect/

Hey yall, I’m looking for some help with sending emails with child/parent flows. I have a setup like this: https://discourse.prefect.io/t/how-to-pass-data-from-a-child-flow-to-another-task-in-a-parent-flow/126, but with an

EmailTask

instead of the transform_and_show task. I want to send the data result in the body of the email. However, I am getting the message

AttributeError: 'FunctionTask' object has no attribute 'encode'

. If I make it a string, I get an empty email. I’d like to send the actual result and not the functiontask.

Hey everyone - piling on with another livestream! Join @Dylan and myself on PrefectLive 📺 at 3p Eastern (5 minutes from now) as he writes some Prefect flows with a Twilio integration. We definitely won't be trolling his roommates! 🙂 See you there.

Hello everyone, is there any code to help with monitoring the state of a flow from code running within the flow? I know how to do it if you have a flow run id already, but I’m not sure how to get this from within that flow. Thanks.

i am trying to register flow with server deployed on gke and docker for flow storage, and getting belw mentioned error for Dockerfile

Hi All, if I want to build an image for dask cluster on fargate, would extending

prefecthq/prefect

and pip install my source code in the image be enough?

Can someone point me to an article on why prefect 2.0 is doing away with DAGs? I’m struggling to see how DAGs are restrictive.

Hi Community, How do I get the actual value of a parameter?

data_update = Parameter("data_update", required=True)

    put_dateformatted_data_to_s3.map(
        df=data,
        bucket_name=unmapped(get_key_value("bucket")),
        key_name=unmapped("key"),
        suffix=unmapped(f"_{data_update}"),
    )

This writes a file with a name of

19_<Parameter: data_update>.parquet

I want it to be the actual param provided in the flow

Hi there,

Hi there, I'm trying to use mapping in a flow that has a merge function in the middle of the flow. How could I use map in this flow.

What's the best way to set up a flow to run for N minutes (hours) maximum?

If I want to have an external python file (e.g. in a

src/

directory) what is the best way to import it? I tried following similar logic to this: https://docs.prefect.io/orchestration/flow_config/storage.html#loading-additional-files-with-git-storage but adding to the import path:

import pathlib, sys
file_path = pathlib.Path(__file__).resolve().parent
sys.path.append(file_path)

But keep getting this error:

[23 February 2022 4:22pm]: Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'src'")

Is there a best practice for importing external python code into a flow?

Hi, How could I prevent the downstream mapped task from skipping when an upstream mapped task fails for some of the inputs? It’s mentioned here:

If the output of one mapped task is used as input to another mapped task, any failed or skipped task will make the subsequent task fail/skip by default.

How can I change this default behaviour so that the downstream mapped task runs even if the upstream mapped one fails?

I’m trying to run a Prefect Flow on a Docker Agent. The agent is able to pull the image and complete the deployment of the flow run. The flow progresses from

Scheduled

to

Submitted

state in the UI. But there is no docker container that is ever spun up and the agent doesn’t do anything. Is there a checklist similar to this, but for submitted state? https://discourse.prefect.io/t/why-is-my-flow-stuck-in-a-scheduled-state/73

I am running Prefect docker agent in supervisor. But I am getting following error: ModuleNotFoundError: No Module named prefect at line "from prefect.cli import cli". Anyone faced similar issue?

Hello everyone, I have my prefect flow py file inside a virtual environment in a remote ec2 instance how do I run the file such that when I log out it will remember and pick up the schedule indicated

Hi community! I am wondering if I can create a flow run with a specific task states configuration. My use case is to run a subset of the flow tasks, so I could set the state of some tasks to

Success

and the flow run just runs the rest of the tasks (obviously with no data dependencies between those tasks). Thank you in advance!

Hi community! I need your help!! I tell you, when we are trying to run a process from the website (by clicking on the run button), we first have to start the agent in our command line. Is there a way or a link to some documentation to avoid this human intervention in order to be able to execute the process automatically?

Hello, When I'm trying to label a flow as per the documentation ,I'm getting an error for prefect version 1.0.0

ModuleNotFoundError: No module named 'prefect.environments'

.. It is there in

0.15.13