I have a question related to Prefect Cloud. We have our jobs running in AKS - with our Flows using a custom Image. Our flows are all built into the image using Docker Storage. We've been trying to mitigate the possibility of adverse effects if we were somehow to leak or Prefect API keys. To that end we've locked down our cluster a bit with a security policy to disallow any flows trying to run on images that we have not authored (e.g. in our registry). Is there any possibility that someone (with Prefect Cloud API Keys) could register a flow that uses our image, but an alternate flow storage mechanism? We were considering subclassing the FlowRunner class to handle some checking/validation there - validating some of the metadata set at registration time against some of the metadata provided at Runtime. Is this possible, does it make sense? Is there a way we can ensure that our Custom FlowRunner is the default (and only) FlowRunner permitted? To be more concise our goal is to ensure that the only flows running in our infra (AKS) run with our images (this we have solved), and only the flows that are built into those images can be run (this we have not solved).

👋 Hello, team! I just signed on and hope to find answers for accessing task results.

When I used

FilterTask

with my own filter func, does the task still filter out the default

NoResult

None

and exceptions?

Hey, does anyone in the community did incremental etl with Prefect ? We are considering doing it and maybe there are mistakes that can be avoided ^^ Thanks !

Hi there! Another question. We recently did an upgrade to our prefect instance, and post upgrade some logs are no longer getting pulled in from the running task. We use a very lightweight wrapper around the DbtShellTask: https://docs.prefect.io/api/latest/tasks/dbt.html#dbtshelltask And we have the log_stdout=True variable set which according this this should let the stdout show up: https://docs.prefect.io/core/idioms/logging.html#logging-stdout I’ve ssh’d into the running task and verified that its emitting the stdout we are expecting, is there a step I should be looking at somewhere? Thank you!

Is there a way to use Prefect Client to get a latest flow_id within a project?

Does anyone have Slack alerts set up with dbt tests on Prefect? Is it possible to send an alert with dbt source tests that are warning or failed?

hey folks - any mods out there who could help me change the email associated with this workspace? i’m losing access to this email here

Is there a way to force a task to rerun even if it's cached?

hi. I recently came across prefect and it looks amazing. I am trying it out with a few of our jobs, but I'm stuck on something! (Sorry for the simple question!) When registering a flow with Bitbucket storage, how do I ensure that shared functions (from a project utilities module) are available to the flow when it runs on an agent (in this case a local agent)? In the project folder flows are in a flows folder, utilities in another folder, etc. I guess the flow is not being run from within the repo project directory because it can't find the utils and libs modules?

Is there any way to specify a custom FlowRunner (which has subclassed Prefect's FlowRunner) as the default FlowRunner with PrefectCloud? I would like to enforce some checks on the Flow Run configuration/settings for all flows. I was thinking of subclassing the FlowRunner and adding some validation in its

__init__

method

I have logger class which i am using for my code. But the logs printed using that is not coming in prefect UI. Only the ones used with

prefect.context.get("logger")

is coming in the UI. How to surface my custom logger method logs also into the UI?

I have the code structure as follows

main-project
  | 
  src
     |
     | __init__.py
     | module1.py
     | module2.py
  |
  configs
     | 
     | config.yaml
  |
  flows
     |
     | sample_flow.py

module1,2 uses configs from config.yaml file and sample_flow import the code from modules, declares them as tasks and wraps them in a flow. I have some other custom inbuilt dependencies as well. Some of the modules needs to run on GPU. I want to run this on AWS EC2. • What is the best way to package the code? • What is the way to run agent that uses GPU on EC2?

Is there an easy way to rerun all the flows in a project for a cloud backend?

Hi all, I am currently getting a `urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=4200): Read timed out. (read timeout=15)`on my EC2 Prefect Server setup. I stumbled upon this: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/config.toml#L64 Although it seems a Cloud configuration. Can someone please point me out on the right direction?

Hi All 👋, Relatively new to Prefect and looking into it for running my team's data operations. We have previous experience with Celery for running our distributed tasks, and are looking to upgrade to either Airflow or Prefect. With Airflow, we found several undesirable aspects & complexity which has caused us to investigate Prefect. Our workload: • 1000s of tasks • Lowest granularity is every 5 mins • Primarily python-based code • Homogenous dependencies, but may change in the future as we move to more ML based workflows. I have read through the prefect docs, and have some questions about how Prefect actually schedules tasks (in a non-prefect cloud context). From the docs, one criticism of Airflow is:

• the centralized nature of the Airflow scheduler provides a single point of failure for the system

In a typical master/worker deployment, you have a single-point-of-failure schedule (yes you can run it highly available with locking / other mechanisms) responsible for reading from a schedule DB, to invoke tasks into a queue to be processed by workers. Can someone clarify how Prefect server solves this issue? From the docs, it seems Prefect server is also a SPOF in this architecture. Based on code here: https://github.com/PrefectHQ/server/blob/master/src/prefect_server/services/towel/scheduler.py#L21, we would not be able to run several instances of the server simultaneously, as there is no locking taking place against the DB, and would cause double execution

I am currently running my jobs on Fargate but have bumped into their (weirdly low) limits on the number of cores I can use. What would be the easiest upgrade pathway for running jobs provided I wanted to keep using serverless? My guess would be it’s time to switch to a

DaskExecutor

and then use a temporary cluster as described here: https://docs.prefect.io/orchestration/flow_config/executors.html#using-a-local-cluster Just want to make sure I’m not missing a more obvious solution before I move to a cluster.

Hi, I am trying to configure a flow that will run in a docker image and will be executed by a docker agent. The flow was configured successfully. I started a docker image from my machine and triggered a run via the ui in the prefect cloud. I got the following error message: “`run_config` of type

DockerRun

, only

LocalRun

is supported ” what does it mean?

with Flow("iris_batch", storage=Local(path="/app/batch/flow.py",stored_as_script=True),
          run_config=DockerRun(image="test:latest")
          ) as batch_prediction_flow:

What i am missing? Amy help will be appreciated:) Thank you!

Just out of curiosity, is there an official release date for prefect 2.0? 🙂 I’m aware of the public beta, but when is stable estimated to be released?

hi community! i'm trying to deploy prefect server on EC2 but can't get past "Couldn't connect to Prefect Server at <address>:4200/graphql", however i can access

<address>:4200/graphql

in the browser - i've followed the troubleshooting guide, but no success

Hi all, we are seeing all our flows on prefect 1.0 fail with error : cannot allocate memory. Can u name a scenario/reason why this could happen? This happens often when the flows are long time running and there is no way for us to debug using the logs on prefect. Please help.

Hi, is there a chance that an ECS Flow runner will be added in Prefect 2.0? Thanks!!

I don’t see task

state_handlers

functionality in Orion/Prefect2.0. Is is on the roadmap? (or I’m I missing something..) thanks

@Kevin Kho great to see you on stage at PyCon!

Hi! What’s the correct way to set the

PREFECT__CONTEXT__SECRETS__GITHUB_ACCESS_TOKEN

variable in the k8s manifest created by

prefect agent kubernetes install

? Do I have to use the

--env

switch or can I provide the value by editing the generated deployment container’s

env:

list directly?

Hi! is there a possibility to manually retry a failed task on prefect 2.0 UI?

This message was deleted.

See you all this Wednesday on PrefectLive! https://prefect-community.slack.com/archives/C036FRC4KMW/p1651510465396669

How long will a flow stay in Paused status while it waits for approval? And is the best way to decline just to cancell the flow run?