Is there a way to force a task to rerun even if it's cached?

hi. I recently came across prefect and it looks amazing. I am trying it out with a few of our jobs, but I'm stuck on something! (Sorry for the simple question!) When registering a flow with Bitbucket storage, how do I ensure that shared functions (from a project utilities module) are available to the flow when it runs on an agent (in this case a local agent)? In the project folder flows are in a flows folder, utilities in another folder, etc. I guess the flow is not being run from within the repo project directory because it can't find the utils and libs modules?

Is there any way to specify a custom FlowRunner (which has subclassed Prefect's FlowRunner) as the default FlowRunner with PrefectCloud? I would like to enforce some checks on the Flow Run configuration/settings for all flows. I was thinking of subclassing the FlowRunner and adding some validation in its

__init__

method

I have logger class which i am using for my code. But the logs printed using that is not coming in prefect UI. Only the ones used with

prefect.context.get("logger")

is coming in the UI. How to surface my custom logger method logs also into the UI?

I have the code structure as follows

main-project
  | 
  src
     |
     | __init__.py
     | module1.py
     | module2.py
  |
  configs
     | 
     | config.yaml
  |
  flows
     |
     | sample_flow.py

module1,2 uses configs from config.yaml file and sample_flow import the code from modules, declares them as tasks and wraps them in a flow. I have some other custom inbuilt dependencies as well. Some of the modules needs to run on GPU. I want to run this on AWS EC2. • What is the best way to package the code? • What is the way to run agent that uses GPU on EC2?

Is there an easy way to rerun all the flows in a project for a cloud backend?

Hi all, I am currently getting a `urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=4200): Read timed out. (read timeout=15)`on my EC2 Prefect Server setup. I stumbled upon this: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/config.toml#L64 Although it seems a Cloud configuration. Can someone please point me out on the right direction?

Hi All 👋, Relatively new to Prefect and looking into it for running my team's data operations. We have previous experience with Celery for running our distributed tasks, and are looking to upgrade to either Airflow or Prefect. With Airflow, we found several undesirable aspects & complexity which has caused us to investigate Prefect. Our workload: • 1000s of tasks • Lowest granularity is every 5 mins • Primarily python-based code • Homogenous dependencies, but may change in the future as we move to more ML based workflows. I have read through the prefect docs, and have some questions about how Prefect actually schedules tasks (in a non-prefect cloud context). From the docs, one criticism of Airflow is:

• the centralized nature of the Airflow scheduler provides a single point of failure for the system

In a typical master/worker deployment, you have a single-point-of-failure schedule (yes you can run it highly available with locking / other mechanisms) responsible for reading from a schedule DB, to invoke tasks into a queue to be processed by workers. Can someone clarify how Prefect server solves this issue? From the docs, it seems Prefect server is also a SPOF in this architecture. Based on code here: https://github.com/PrefectHQ/server/blob/master/src/prefect_server/services/towel/scheduler.py#L21, we would not be able to run several instances of the server simultaneously, as there is no locking taking place against the DB, and would cause double execution

I am currently running my jobs on Fargate but have bumped into their (weirdly low) limits on the number of cores I can use. What would be the easiest upgrade pathway for running jobs provided I wanted to keep using serverless? My guess would be it’s time to switch to a

DaskExecutor

and then use a temporary cluster as described here: https://docs.prefect.io/orchestration/flow_config/executors.html#using-a-local-cluster Just want to make sure I’m not missing a more obvious solution before I move to a cluster.

Hi, I am trying to configure a flow that will run in a docker image and will be executed by a docker agent. The flow was configured successfully. I started a docker image from my machine and triggered a run via the ui in the prefect cloud. I got the following error message: “`run_config` of type

DockerRun

, only

LocalRun

is supported ” what does it mean?

with Flow("iris_batch", storage=Local(path="/app/batch/flow.py",stored_as_script=True),
          run_config=DockerRun(image="test:latest")
          ) as batch_prediction_flow:

What i am missing? Amy help will be appreciated:) Thank you!

Just out of curiosity, is there an official release date for prefect 2.0? 🙂 I’m aware of the public beta, but when is stable estimated to be released?

hi community! i'm trying to deploy prefect server on EC2 but can't get past "Couldn't connect to Prefect Server at <address>:4200/graphql", however i can access

<address>:4200/graphql

in the browser - i've followed the troubleshooting guide, but no success

Hi all, we are seeing all our flows on prefect 1.0 fail with error : cannot allocate memory. Can u name a scenario/reason why this could happen? This happens often when the flows are long time running and there is no way for us to debug using the logs on prefect. Please help.

Hi, is there a chance that an ECS Flow runner will be added in Prefect 2.0? Thanks!!

I don’t see task

state_handlers

functionality in Orion/Prefect2.0. Is is on the roadmap? (or I’m I missing something..) thanks

@Kevin Kho great to see you on stage at PyCon!

Hi! What’s the correct way to set the

PREFECT__CONTEXT__SECRETS__GITHUB_ACCESS_TOKEN

variable in the k8s manifest created by

prefect agent kubernetes install

? Do I have to use the

--env

switch or can I provide the value by editing the generated deployment container’s

env:

list directly?

Hi! is there a possibility to manually retry a failed task on prefect 2.0 UI?

This message was deleted.

See you all this Wednesday on PrefectLive! https://prefect-community.slack.com/archives/C036FRC4KMW/p1651510465396669

How long will a flow stay in Paused status while it waits for approval? And is the best way to decline just to cancell the flow run?

Hi everyone! I am trying to execute a flow on two different docker containers with different number of CPU cores. Is it possible to dynamically set

num_workers

for LocalDaskExecutor based on the number of cores available on the container?

question! what has higher precedence? PREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS or Items 3-8 from https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials

Quick question here - we're looking at doing a livestream with Hex. Anyone here using Hex in combination with Prefect in some way (maybe it's just part of your stack)?

hi - we started seeing a new log from prefect manifesting in some failed k8s jobs. Any idea what could be going on here?

Where in

ECSRun

would I put Dask security args? Like

tls_ca_file

and stuff?

Hello team! I'm looking use Prefect at my company. While testing, I noticed that

random

generates the same values between runs. I'm using Prefect Cloud 1.0 and running a local agent (with Prefect 1.2.1 library installed). Here's my flow:

import prefect
from prefect import task, Flow
from random import random

@task
def task1():
    logger = prefect.context.get("logger")
    random_val = random()
    <http://logger.info|logger.info>(f"{random_val=}")
    if random_val < 0.5:
        raise Exception()


with Flow("random-bug-flow") as flow:
    task1()

flow.register(project_name="ant-flows")

I manually scheduled a run of this flow in Prefect Cloud and I see the random value in the logs. However, when I schedule another run (and any run thereafter), it generates the same "random" value. Is this expected?

Hello Prefect team! Hope you all are doing well 🙂 We are currently using Prefect 1.0 to launch tasks on AWS ECS using a local agent installed on an EC2 machine. The reason we are using a local agent is because we plan to do the same on other clouds (namely Azure and GCP) as well, and since they dont have their specific native agents (like ECSAgent for ECS), the local agent seemed more universal/generic approach for a multi cloud environment. We had some questions regarding the above however: 1. Using the local agent with the Prefect Cloud, is there a hard limitation or some throttling/delays in starting flows that can occur if we schedule a lot of tasks concurrently. For example running 250 flows (and the tasks therein) concurrently. Can a local agent support this? 2. I noticed Prefect 2.0 coming up soon, however upon reading the docs there, things like ECSAgent etc do not seem to exist there, and I was wondering what the migration process from 1.0 to 2.0 would look like. Basically - we would like to run our flows in a multi cloud setup (1 agent per cloud, tagged with respective labels). Thanks!

Hi community, I’m trying to find a documentation that will show an example (if possible) of programmatic cron job scheduling. I.e. I would imagine, that there will be a way to create a parametrized schedule via server api. Use case: Imagine a user interface where user could create a recurring job, specify job schedule and some additional parameters for the job to be executed (i.e. an ETL pipeline with sources and destinations configured via params)

Following up on https://prefect-community.slack.com/archives/CL09KU1K7/p1651543946765889 I wanted to confirm this was not an issue in Orion. In Orion, we've decoupled the one to many relationship between a Flow and Clocks (technically Schedule has the clocks) that was present with the introduction of a DeploymentSpec, which has a 1-1 with a schedule. Rather than iterating clocks here: https://github.com/PrefectHQ/server/blob/master/src/prefect_server/api/flows.py#L688 https://github.com/PrefectHQ/prefect/blob/cb0f80c6fc743cdae9f2af5b493ed454bc53d07c/src/prefect/schedules/schedules.py#L108 We are iterating deployments here: https://github.com/PrefectHQ/prefect/blob/orion/src/prefect/orion/services/scheduler.py#L83, which have a 1-1 between schedules. To solve the similar case of one flow, parameterized N many times & different schedules, we should be able to create N many DeploymentSpecs for the given flow without scaling issues, correct? cc: @Kevin Kho