Hi! I'm trying to create a custom VM image that launches a prefect agent on spawn, is there any records of such work already done here ? I haven't found anything but still worth to ask My goal is to simulate an elastic cluster with prefect: have a flow that launches a instance with this image, start a new sub-flow with this new agent, and deprovision the instance Almost everystep is validated, I just need to build the VM image in

raw

format and I got lost here, if anyone has experience with it I'd gladly take some advice!

Hi! I'm trying to find option similar to

depends_on_past

in airflow. We wanted next run of flow to start only when previous run is in Success state. Can anyone help?

Has anyone got spurious CancelledError:s when running with Dask? Flow id:

e2631f0e-2987-405f-bf3b-3fbb46acf90c

Hi, is there a way get the data that is used in the Dashboard UI via api calls? I’m mainly interested in recreating the Run History graph. For context, we’re an agency that implements/manages a bunch of Prefect instances and I would like to build a way for us to get daily summaries of our Client’s flow runs without needing to log in and out of the UI for each instance.

Hey guys, i'm struggling for a bit here. I need to define a custom

context

value inside the flow run (So i can retrieve later via graphql query), the issue is (not all my flow will have this custom context value), for what i understand the context values is passed in

run()

call, is not possible to add/modify and retrieve this values later? If so, there a alternative i can use to get custom values from a flow?

I already ask this question in thread, but didnt get answer. I get this error, when trying to set the api with the --api argument at prefect agent start:

line 831, in _verify_token
  raise AuthorizationError("No agent API token provided.")
prefect.exceptions.AuthorizationError: No agent API token provided.

hi - our security folks would like to understand the oauth grants for the slack app, before installing it into our workspace. Is this information readily available?

Hello, in which cases do I need to redeploy my flow on prefect? For example, if I have a task

import app.logic
@task
def do_something():
     app.logic.dosomething()

If I am using a Local Agent and modify

app.logic.dosomething()

without modifying the task or the general flow structure, would I still need to redeploy my flow?

Hey all! Just learning about Prefect, really cool stuff so far! I have primarily been looking into what the current solutions are for event-driven flows, and looking through the comments and documentation seems like in Prefect Core there isn't anything built in like Airflow Sensors (I have seen some of the AWS Lambda recommendations and blog posts). Looking at the Orion site however looks like this is being added with the newest engine? Is there any documentation regarding that or is that more so a feature coming soon to Orion and not yet hashed out?

Join our livestream!

Hey team, I assume Prefect has nothing to do with Java, but double checking: Is prefect affected by this log4j vulnerability? https://www.bleepingcomputer.com/news/security/new-zero-day-exploit-for-log4j-java-library-is-an-enterprise-nightmare/

question about manually setting a task from e.g. failed to success, is it possible to provide the output of the task?

Happy Friday! Please is there an example I can follow on using great expectations validation in prefect? I have it set up but when I run the flow, I get the error "ConfigNotFoundError('Error: No great_expectations directory was found here!" not sure what config I'm missing and where to add it. Thank you

👋 Hey community! Yesterday, we shipped some exciting improvements to Orion in the 2.0a6 release. Here are a couple that we're especially excited about: Subflow Navigation In the previous release, we introduced the ability to navigate seamlessly between subflows and parent flows via Radar :radar: . With this release, we made subflow navigation even easier by: - Enabling the dedicated subflow runs tab on the Flow Run page - Tracking of upstream inputs to subflow runs - Adding a flow and task run count to all subflow run cards in the Radar view - Adding a mini Radar view on the Flow run page Executors are now TaskRunners Previous versions of Prefect could only trigger execution of code defined within tasks. Orion can trigger execution of significant code that can be run outside of tasks. In order to make the role previously played by Prefect's

Executor

more explicit, we have renamed

Executor

to

TaskRunner

🏃. Check out the release notes to learn more about the other enhancements and fixes.

Hi all - pricing question - do child tasks in mapped task count as task runs? or does this apply to only the parent task?

Hey everyone I have a flow that had required parameters. And I have a problem... I want to be able to programatically add schedule clocks with default parameters. I am able to do it through UI, however when it comes to the graphql solution ,[https://docs.prefect.io/orchestration/flow-runs/scheduling.html#creating-flow-schedules ] the biggest obstruction is that Setting flow group schedules will remove any existing schedules. I would like to have

add_flow_group_schedule

instead of just

set_flow_group_schedule

(Just like we have in the UI ~ The New Schedule button) -> How does it manage to do it ?

How do you recommend approaching CI pipelines with docker storage? So I think we might end up having N flows in python scripts, using docker as storage deploying to kubernetes. I guess we will run all flows on every release, that means creating a new docker image for each of the flows and registering them again. Can be ok, I mean, I not going to do it manually... but seems a bit overkill to rebuild and upgrade all flows if only 1 has changed. Any recommendation on how to manage flows through CI pipelines if we have all flows on the same repo?

hey all - new prefect user here! i'm trying to figure out how to run a task across all combinations of three parameters. here's a toy example (in practice, some lists are returned from previous tasks). this runs but doesn't do what i want -- what i'd like to get is a run of

add(1, 10, 100)

,

add(1, 10, 200)

,

add(1, 10, 300)

,

add(1, 20, 100)

, etc, with combinations similar to the results of

itertools.product

in standard python

@task
def add(x, y, z):
    return x + y + z

@task
def print_all(results):
    print(results)

with Flow('Run combinations') as flow:
    a = [1, 2, 3]
    b = [10, 20, 30]
    c = [100, 200, 300]

    results = add.map(a, b, c)
    print_all(results)

flow.run()

Is there a way for to override the

target

keyword argument for all tasks at flow level? I have some tasks with

target

caching and for testing I would like all the result files to be created again without caching. If we can selectively decide which one to override for a particular run that would be even better

Is there a repo showcasing variant pipelines? I'd like to be able to scan through the full range of capabilities.

Also, is there a use-case example leveraging "Great Expectations" for testing and validating?https://greatexpectations.io/expectations

Hey everyone I have a weird situation that I like to hear what you think I have a parent flow that triggers child flow when I get a failed status in the parent flow with error that the new flow didn't get the params But the new flow is actually triggered with the right params What I can tell on the child flow is that he waits for a minute to scale-up it's resources before starting

Hi all, I am trying to run flow using ECS Fargate (Following the wonderful guide at https://towardsdatascience.com/how-to-cut-your-aws-ecs-costs-with-fargate-spot-and-prefect-1a1ba5d2e2df#5b72) However, I am running into an issue with python versioning, as I am getting some syntax errors for f-string using f"{var=}" formatting that was introduced in python 3.8 - Although in the task definition it seems that I AM using python 3.8 as I'm using the prefecthq/prefect:latest-python3.8 image - What am I missing?

Hi is there a way to inspect if/where the targets of a task was ultimately saved after running

flow_result = flow.run(...)

? I'm missing a couple of output files and I have no idea where they are saved so trying to hunt them down to prevent memory leak (not in my expected folder and also not in

~/.prefect/results

)

Is it possible to start a run just using curl?

I'm playing around a bit with Orion (2.0a6) and I noticed that the uvicorn server is leaking memory quite badly. Is this something worth reporting?

Hey, another question regarding using ECS Fargate and S3 for deploying flows 🙂 I have a common python file with many helper methods that are used by all of my different flows, however, they are obviously not deployed together with the flow themselves when using

flow.register(project_name="test-project")

What is the best practice to importing common .py files? Since at the moment the simple

from utils import util

fails with an error:

Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'utils'")

Hi. I just started getting the following error when registering flows in our CI/CD pipeline. Is this happening for anyone else? We are running 0.15.10:

Traceback (most recent call last):

File "register_flows.py", line 20, in <module>

register_flows(all_flows)

File "register_flows.py", line 14, in register_flows

prefect_client.register(

File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 1242, in register

self.graphql(

File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 569, in graphql

raise ClientError(result["errors"])

prefect.exceptions.ClientError: [{'path': ['set_schedule_active'], 'message': "Required parameters were not supplied: {'version1', 'version2'}", 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

Hi, how can I kick off an agentless flow run on cloud backend via python API? The equivalent of

prefect run --execute

. I was going to do

ShellTask("prefect run --execute").run()

but thought there might be another way. Maybe

create_flow_run

?