Hi, does the Prefect server need restarting every time a new schedule is created? I've created a deployment, added a schedule to it and can see the queue it's allocated to is healthy. But nothing is appearing as "Pending" in the flows. If I restart the UI it seems to then show them?

Hello, I'm trying to go through the ECS guide here - https://prefecthq.github.io/prefect-aws/#getting-started , but my Flow Runs always get stuck in

Pending

before timing out after 120 seconds. I can see the Flow Run start and hit

Pending

and at the same time I see the ECS Task appear in the AWS console (shown in screenshot) - My Agent logs the following:

17:59:41.311 | INFO    | prefect.agent - Submitting flow run 'efc6b670-c269-4a87-ba2f-467e4a2c6e85'
17:59:42.349 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Retrieving task definition 'prefect__ecs-task-flow__ecs-task-deployment'...
17:59:43.619 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Creating task run...
17:59:44.216 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Waiting for task run to start...
17:59:44.299 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Status is PROVISIONING.
17:59:54.508 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Status is PENDING.
18:01:46.542 | ERROR   | prefect.agent - Failed to submit flow run 'efc6b670-c269-4a87-ba2f-467e4a2c6e85' to infrastructure.
Traceback (most recent call last):
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect\agent.py", line 490, in _submit_run_and_capture_errors
    result = await infrastructure.run(task_status=task_status)
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect_aws\ecs.py", line 619, in run
    ) = await run_sync_in_worker_thread(
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect\utilities\asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect_aws\ecs.py", line 814, in _create_task_and_wait_for_start
    self._wait_for_task_start(
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect_aws\ecs.py", line 1057, in _wait_for_task_start
    for task in self._watch_task_run(
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect_aws\ecs.py", line 1042, in _watch_task_run
    raise RuntimeError(
RuntimeError: Timed out after 122.3249180316925s while watching task for status {until_status or 'STOPPED'}
18:01:46.561 | INFO    | prefect.agent - Completed submission of flow run 'efc6b670-c269-4a87-ba2f-467e4a2c6e85'

Does anyone know what the issue could be? I'm fairly new to ECS, so I'm trying to figure out if the fix I need is in the ECS or Prefect side of things. My ECS Cluster is brand new, I haven't made any changes - basically its just what you'd get if you hit next at every screen during the creation process.

Hi all, I was trying to run a Prefect agent on my local machine with a deployment using a docker container for the infrastructure block. I am able to build the docker image with my dockerfile set up and deploy a flow using the infrastructure block, along with a storage block. However, after deploying, when I try to run the flow with the agent running on my machine, it is giving this error saying: 'charmap' codec can't encode characters in position 5-44: character maps to <undefined> Based on the error text, it sounds like there might be something with one of the variables it is pulling in (like it is trying to read a special character and not able to or something). However, I can't tell where the issue is occurring based on the error message. Does anyone have more insight on how this error might occur? See screenshots for the error and deployment code that I have. Let me know if you need any more details. Was working on this with @Joseph Loss

I am new to prefect. My company is using it with success. But our CI (local flow testing) fails if there is high IO load. It is not failing with high CPU load. We are not able to find out what is happening. Is this channel the correct one. If not where should I post my question? Hoping to find information and the format of the question by reading others posts first...

Good morning can we have multiple flow per deployment?

Hello! I'm also new to Prefect. How can I create an API Key if I'm using the open source version?

Hi all! Thanks for the help. I’m coming from using pydoit and make, and looking for more of a lightweight, single-machine build system with caching than a full, cloud-based task orchestrator. I’m trying to determine if Prefect would be a good fit. [continued in thread]

Hi! Please ELI5: what’s the difference between

S3

and

S3 Bucket

blocks?

What happens in Prefect when you restart a failed run? e.g. let’s say I had retry=3 but all retries failed because the database was down and the last task in the flow needed to ingest data into the database. One hour later the database comes up and I manually restart a failed run. Does the restarted run know it’s yet another an incarnation of a previously failed run, or is it stateless and think it’s a different run? I guess my question is whether Prefect re-runs are stateful or stateless and what are the re-try semantics? Also what are the best practices around idempotency? I want an all-or-nothing semantics with the final results written to database, no partial leakage. So the question also becomes whether or not re-runs that wrote no data are considered successful. Update: I saw this post from @Anna M Geller https://discourse.prefect.io/t/how-can-i-restart-a-flow-in-prefect-2-0/1332/7 I think there is a hidden “Data Engineering 101” assumption that: 1. Tasks and DAGs as a whole should be idempotent 2. Use timestamps as means to achieving idempotent tasks - e.g. max(LAST_UPDATED), this is the flag that will indicate whether the task already succeeded, so no side effects 3. Implement atomic all-or-nothing semantics within each task a. For example - no partial writes to database tables

There are probably recommended complementary books / courses to Prefect / Airflow, as there are data engineering principles that stick with you regardless of the framework you use to implement your data pipelines. I would love to learn good principles and approaches towards designing and implementing DAGs.

How do I run Prefect behind nginx? I have: https://www.my-site.com/app https://www.my-site.com/prefect And I need Prefect to rewrite the URL such that all assets (e.g. css, javascript, etc.) are rewritten with the

/prefect

prefix. Otherwise users will get a blank page when visiting /prefect, because the assets won’t be pointing to the right location. Does anyone have any experience with that? Should I set

PREFECT_UI_SERVE_BASE

env var? How should the Dockerfile for the agent change?

Hey all! Just getting started with Prefect and I'm trying to understand how to see the values of my parameters from within the UI. I have two methods decorated as a flow, one of which calls the other. The code works, but the parameters used don't show up in the UI. Is there some better way to see dynamic parameters in the UI? Edit: Apparently the answer was just to upgrade my version of prefect. 2.8.7 > 2.10.6

Hey all! Just getting started with Prefect and I have a question Is there a way to use Prefect to create a base flow class that can execute tasks based on configuration? I couldn't find a way to create such a flow class and am currently only using the flow decorator on a flow function.

Hey, is there any MongoDB block?

Hey all! New to Prefect, love the product, and I have (what I hope is) an easy question about setup. Many thanks in advance! I have Perfect Server and a Prefect Agent running in a k8s cluster. I have a single git repo that houses all of my Flow code. The git repo has a pipeline that bakes all Flows/dependencies into a docker image then pushes that image to a registry. So far so good. After I run my build pipeline and create a new image with all updated Flows bundled in it, I think I need to update the Agent to tell it which image to pull when running a

KubernetesJob

- does that sound accurate? This is my first foray into Prefect and kubernetes so I may be conflating several technologies 😂

Hi all - have a billing question. How would I best reach out to support?

Hi guys, Do you know if it is possible to use Github storage block with git submodules and pull recursively: git parent project | | git subproject I've tried adding multiple --sb, using get_directory(), having sb and try to load from one workflows the files for the other repository. Tried everything but didn't find a solution. I just want to know if it is possible and it someone did something similar. Thanks a lot!

What is the difference between a work queue and work pool? The documentation seems to refer to them very interchangeably? Is it the same thing..?

Hey all, is there a way to return an object with no type validation from a flow \ task?

Hey all, Perhaps this is a silly question, but with the depreciation of /deployments, our flows have become ... messy to look at under /flows. I am wondering if there is a way to group flows/deployments together -- see thread for sloppy markup image, but would like the ability to put flows/deployments into 'folders'

Good morning, everyone. So, I am planning on pitching Prefect for internal use at my current company, but I would like some guidance on the best way to implement it. I know you can install prefect on windows, but I am having issues with the Deployment not liking the python install being C:\Program Files because of the space. Is it best to run prefect on Linux? If so, how can I deploy my scripts from a windows VSCode IDE to that Linux server? This may be a novice question, but I'm looking for the best practice to help my pitch. Thank you!

Hello folks, Quick question: I understand the Prefect agent lives on my machine when developing locally (or does it?). When using ECS for instance it lives in its own container in a cluster. However when using the Docker block, where does the agent live? Related: when does it make sense to use ECS vs simply the Docker block? What am I missing? Thanks!

Hi there. I just have one quick question. Is there anyway to run automation in parallel. Lets say I want to run Flow B and C after completion of Flow A. But I want to both in parallel is that do able?

Hi! I would like to ask if this page: GCP GCS tutorial can be reinstated on the prefect discource site? https://discourse.prefect.io/t/how-to-deploy-prefect-2-0-flows-to-gcp/1251

Hi there. I've posted a github discussion here, but the discussion section seems not to be that active. I'm fairly confused about what actually

deployment.yaml

achieves in a prefect project, when it lacks many of the infra/resource specific information available when you build a deployment via

prefect deployment build

. Link to the discussion: https://github.com/PrefectHQ/prefect/discussions/9494 The docs are fairly confusing wrt what exactly a prefect project is and how it relates to lower level definitions.

Hi Everyone, can anyone please tell why i am getting this error.

Crash detected! Execution was interrupted by an unexpected exception: httpx.HTTPStatusError: Client error '404 Not Found' for url '<http://orion-platformv2.services.carpl.ai/api/task_runs/24fd3ed1-78e5-49b2-84ab-ea42c385c942/set_state>'
For more information check: <https://httpstatuses.com/404>

I stated inference in 500 inputs, and after sometime it gets crashed. some of the ques i have is. 1. Why it gets crashed, is it because I gave 500 inputs and prefect server fail to handle it. 2. If it gets crashed, is there any mechanism to handle this. 3. And I am using Kubernetes as infrastructure Can anyone help me please

Hi all, I have been trying to do something as described here using either prefect 2.0 OR 1.0 -> You can see my comment there. In summary, I just want to run tasks where I can specify one or multiple targets (depending on how many objects are returned by the task) so that the first time I run a task it saves the object it returns as a local file and in subsequent runs it just loads it so that following-up tasks can use that information.

hi, this is in regards to projects deployment. looking at example , where is infra-block given. for example, if i were running a deployment via CLI, i would put

--infra-block ecs-task/prefect-project

?

Hello prefect community.... I am having issue importing schedules ........ImportError: cannot import name 'schedules' from 'prefect'.... I am using prefect Version: 2.10.9

Hi all, I have a question regarding serializers. I use the following in a task to save the result to a

.pkl

file. The result is a dictionary with two elements (two arrays).

@task(cache_key_fn=cache_file_based_fn,
      persist_result=True,         result_storage=LocalFileSystem(basepath=settings.main_dir+'/results'), 
      result_storage_key=settings.main_task_file,
      result_serializer="pickle",
      cache_result_in_memory=False)

Json is not supported because narrays cant be encoded with JSON. Pickle is fine with me but when I try to open the pickle file that was generated I get the following error:

UnpicklingError: invalid load key, '{'.

Any clue why this is happening? just FYI:

settings.main_task_file = 'results.pkl'