Hi All... import os from datetime import timedelta from prefect import flow, task from prefect.server.schemas.schedules import IntervalSchedule from prefect.deployments import Deployment @task def hello(server): print('hello', server @flow() def transformation_one(): hello('one') @flow() def transformation_two(): hello('two') @flow() def transformation_three(): hello('three') transformation_one_deployment = Deployment.build_from_flow( flow=transformation_one, name="reporting_server-flow_one", schedule=IntervalSchedule(interval=timedelta(seconds=30)) ) transformation_two_deployment = Deployment.build_from_flow( flow=transformation_two, name="reporting_server-flow_two", schedule=IntervalSchedule(interval=timedelta(seconds=30)), ) transformation_three_deployment = Deployment.build_from_flow( flow=transformation_three, name="reporting_server-flow_three", schedule=IntervalSchedule(interval=timedelta(seconds=30)) ) if name == "__main__": transformation_one_deployment.apply() transformation_two_deployment.apply() transformation_three_deployment.apply() os.system('prefect agent start -q default') I have a script similar, this is saved in a file named scheduler.py.....When I'm trying to run , I see only this , none of my flows are getting triggered.......I checked by calling the flows without deployment , at that time they were working.....when used with Deployment....the flows are not running..........Can someone please help

I have now run the medium's tutorial code passing conda image as base image to dockerfile and it crashes. So the Dockerfile becames:

FROM prefecthq/prefect:2-python3.10-conda

COPY requirements.txt .
COPY setup.py .
COPY prefect_utils .

RUN pip install --upgrade pip setuptools --no-cache-dir
RUN pip install --trusted-host <http://pypi.python.org|pypi.python.org> --no-cache-dir .

ARG PREFECT_API_KEY
ENV PREFECT_API_KEY=$PREFECT_API_KEY

ARG PREFECT_API_URL
ENV PREFECT_API_URL=$PREFECT_API_URL

ENV PYTHONUNBUFFERED True

COPY flows/ /opt/prefect/flows/

ENTRYPOINT ["/bin/bash", "--login", "-c", "prefect agent start -q default"]

Am I doing something wrong? From a thread in #prefect-community

Hi I prefer not to use helm in my k8s cluster I would like to be able to explicitly apply manifests to setup an agent - are there any examples?

Hi everyone, I’ve recently upgraded my prefect version from 2.8.6 to 2.10.12. Now prefect no longer captures the logs from my tasks that run prefect-shell. Does anyone know a fix for this?

Hi, I'm having issues using the Pager Duty webhook under "Notifications" - I've set the "Notify Type" to "failure", but in the Prefect log file it complains: WARNING | apprise - Failed to send Pager Duty notification: Bad Request - Unsupported Parameters., error=400 and also: unexpected value; permitted: 'info', 'success', 'warning', 'failure' (type=value_error.const; given=prefect_default; permitted=('info', 'success', 'warning', 'failure')) - any idea what I'm doing wrong?

Hi, has anyone else seen “NotReady” tasks filling up the slots in the task concurrency limit?

Hi everyone! QQ about deployment webpage. Whenever I visit localhost:4200/deployments/deployment/<deployment_id> I can see only last week flow runs (until tomorrow). How can I select a different time frame here?

I can't find any info on the docs on how to create work pools via code, I'm currently using subprocesses to invoke a cli command in order to create the workers but this is not acceptable IMO. Any ideas?

I'm currently utilizing the most recent version of Prefect - specifically, FROM prefecthq/prefect:2.10.12-python3.10. However, I'm experiencing some difficulties installing the GDAL dependencies within it. I attempted to install them using the following commands:

RUN apt-get install -y gdal-bin libgdal-dev

and

RUN apt-get install -y software-properties-common && add-apt-repository ppa:ubuntugis/ppa && apt-get update && apt-get install -y gdal-bin python3-gdal libgdal-dev

, but unfortunately, I haven't achieved the desired outcome. Has anyone encountered and managed to resolve this issue? Any insights would be greatly appreciated!

Hi Guys, I am not able to create work pool due to below ssl error in prefect server. The error only occurs in workpool creation probably happens when prefect server is connecting to raw.githubusercontent.com 163917.540 | ERROR | prefect.server - Encountered exception in request:

Hello, Is there a reason that some flows end up stuck in the running state? I’m assuming this has to do with some error handling somewhere but is this a known issue? Most of our runs are launched from laptops talking to a remote prefect server. Is there something that we should be doing to prevent this from happening? This results in a lot of confusion for our users so any ideas/suggestions would be helpful here.

I there a way to set up an automation that sends an notification when a flow and it's multiple subflows fail. I have it currently setup so that it I have one automation sends out 45 emails , but I would like it to send out one email with the status of each of the subflows. My preference is to do this from the prefect ui but I can also do it in my code as well.

On concurrency limits - I have set a concurrency limit, and am running flows with the proper tag. However, I can see that the number of flows actively running are greatly exceeding my limit, and some of them are crashing, likely due to OOM. Why isn't the # of flows being throttled? I can see that the "Active Task Runs" is None next to the limit

Hi @Prefect, I have an issue with the flows just "disconnecting" in the task run. It looks like the tasks are being kicked off via VisualStudioCode, but in the middle of the task it looks like it exits out for some reason. But the flow keeps running in the UI, which is weird... (see console and UI screenshot for details). Any help would be appreciated, we've been stuck for days

(\\v-tm-qeq-05\Prefect\) PS Microsoft.PowerShell.Core\FileSystem::\\v-tm-qeq-05\PrefectScripts\cleung> & //v-tm-qeq-05/Prefect/python.exe //v-tm-qeq-05/PrefectScripts/cleung/Script/ibes_pykx.py
13:27:13.699 | DEBUG   | prefect.client - Connecting to API at <http://v-tm-qeq-05:4200/api/>
13:27:13.999 | INFO    | prefect.engine - Created flow run 'awesome-tarsier' for flow 'ibes-pykx'
13:27:14.001 | DEBUG   | Flow run 'awesome-tarsier' - Starting 'ConcurrentTaskRunner'; submitted tasks will be run concurrently...
13:27:14.004 | DEBUG   | prefect.task_runner.concurrent - Starting task runner...
13:27:14.245 | DEBUG   | Flow run 'awesome-tarsier' - Executing flow 'ibes-pykx' for flow run 'awesome-tarsier'...
13:27:14.247 | DEBUG   | Flow run 'awesome-tarsier' - Beginning execution...
13:27:14.250 | INFO    | Flow run 'awesome-tarsier' - Initializing initial arguments
13:27:14.311 | INFO    | Flow run 'awesome-tarsier' - Created task run 'saveRetRawNaive-0' for task 'saveRetRawNaive'
13:27:14.315 | INFO    | Flow run 'awesome-tarsier' - Submitted task run 'saveRetRawNaive-0' for execution.
13:27:14.360 | INFO    | Flow run 'awesome-tarsier' - Created task run 'saveRecRevAI260-0' for task 'saveRecRevAI260'
13:27:14.363 | INFO    | Flow run 'awesome-tarsier' - Submitted task run 'saveRecRevAI260-0' for execution.
13:27:14.452 | DEBUG   | Task run 'saveRetRawNaive-0' - Beginning execution...
13:27:14.456 | INFO    | Task run 'saveRetRawNaive-0' - Executing saveRetRawNaive
13:27:14.501 | DEBUG   | Task run 'saveRecRevAI260-0' - Beginning execution...
13:27:14.510 | INFO    | Task run 'saveRecRevAI260-0' - Executing saveRecRevAI260
13:27:16.039 | DEBUG   | prefect.client - Connecting to API at <http://v-tm-qeq-05:4200/api/>
(\\v-tm-qeq-05\Prefect\) PS Microsoft.PowerShell.Core\FileSystem::\\v-tm-qeq-05\PrefectScripts\cleung>

In theory should it be possible to execute a run that submits ~15,000 tasks on the concurrent task runner? I'm using a semaphore implementation to limit the execution to only 3 tasks at a time. Prefect cloud seems to be crashing when I try and open the flow run logs for this flow as its running, although it's still running ok in the execution infrastructure so far.

Has anyone experieced an internal server error when starting the agent recently? Nothing on our end has changed but we are now getting a 500 when the endpoint

<https://api.prefect.cloud/api/accounts/23ec087c-89c2-47de-a959-4ba9654879c0/workspaces/d31c992f-ff7d-48c1-9490-f321dfdba460/flow_runs/filter>

is called on startup. I also just tried updating to a later version (2.10.12)

Hey guys, anyone facing issues while trying to run a flow from terminal using CLI? I have used this command: prefect deployment run <flow-name>/<deployment-name> is there any other option ?

Hi! I don't know if this happened to any of you. I am running a prefect agent in AWS ECS (fargate) as a service. Today the agent raised an error, and after that it is still alive but it does not work (work queue is healthy but agent does not run any flow). The error seems to be like a network error or something like that.

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/h2/connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
                         ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
During handling of the above exception, another exception occurred:
...
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state 
ConnectionState.CLOSED
Backing off due to consecutive errors, using increased interval of  120.0s.
Resetting backoff due to successful run.

out of curiosity, why deny underscores in a block name string?

Hello Prefect Community 👋! Our team is migrating all the flows from prefect 1 to prefect 2. We are almost done and only remaining with copying secrets from Prefect 1 cloud. Currently we cannot view the secrets in the prefect UI. What is the best way to copy secrets from prefect 1 to prefect 2 blocks?

Hey is there a way to create a WorkPool in code? rather than using the CLI?

Hi, I just deployed prefect-server on our EKS cluster, and when I browse through the ingress the UI works but I see this kind of error message:

Can't connect to Server API at <http://127.0.0.1:4200/api>. Check that it's accessible from your machine.

should I change some configuration in prefect maybe an environment variable to make it work?

Hello. quick question: is there a way to iterate through all the tasks associated with a flow? Thanks in advance

if I delete a block in the prefect cloud gui, I cannot create another block of same type with same name. cli usage

prefect block ls

does show that the block is no-longer there. Any idea what is happening there?

I know I am being a complete PIA - it will subside I promise!! I have the following code in thread - a k8s deployment and a docker deployment - both pointing at exactly the same image - the docker one runs perfectly fine the k8s one has the following error;

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 318, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.9/site-packages/prefect/client/utilities.py", line 40, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/prefect/deployments.py", line 197, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=deployment.path, local_path=".")
  File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 147, in get_directory
    copytree(from_path, local_path, dirs_exist_ok=True)
  File "/usr/local/lib/python3.9/shutil.py", line 566, in copytree
    with os.scandir(src) as itr:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/flows'

Hey team - what is the replacement for the email task in prefect 2?

from prefect.tasks.notifications import EmailTask

Hey everyone, I'm trying to understand how Prefect Projects work but can't find many resources. Are there any online resources (other then the prefect documentation) that can help me get started?

A V1 issue - I have suddenly started to get;

prefect.exceptions.ClientError: [{'path': ['flow'], 'message': 'Missing Authorization header in JWT authentication mode', 'extensions': {'path': '$', 'code': 'invalid-headers', 'exception': {'message': 'Missing Authorization header in JWT authentication mode'}}}]

When trying to trigger a flow using

create_backfill_flow

locally - any ideas?

Getting the same error when I try to do

prefect run

cli command!?

Related to the end of life of v1?