@Marvin in prefect 3, how can I set up to run the UI from docker ?

Hi, I am evaluating prefect cloud 2.0 and ran into the following

PrefectHTTPStatusError

for one of my subflow runs. The cloud UI also is in an inconsistent state since it says flow/subflow is still running. It works fine locally. Any help would be appreciated!

Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 559, in orchestrate_flow_run
    result = await run_sync(flow_call)
  File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 56, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/usr/local/lib/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/root/deployer/pipeline.py", line 21, in run_ro
    etl.run_ro()
  File "/usr/local/lib/python3.9/site-packages/prefect/flows.py", line 367, in __call__
    return enter_flow_run_engine_from_flow_call(
  File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 154, in enter_flow_run_engine_from_flow_call
    return run_async_from_worker_thread(begin_run)
  File "/usr/local/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 136, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/usr/local/lib/python3.9/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 445, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/prefect/client.py", line 104, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 454, in create_and_begin_subflow_run
    terminal_state = await orchestrate_flow_run(
  File "/usr/local/lib/python3.9/contextlib.py", line 670, in __aexit__
    raise exc_details[1]
  File "/usr/local/lib/python3.9/contextlib.py", line 653, in __aexit__
    cb_suppress = await cb(*exc_details)
  File "/usr/local/lib/python3.9/contextlib.py", line 199, in __aexit__
    await self.gen.athrow(typ, value, traceback)
  File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 1181, in report_flow_run_crashes
    await client.set_flow_run_state(
  File "/usr/local/lib/python3.9/site-packages/prefect/client.py", line 1505, in set_flow_run_state
    response = await <http://self._client.post|self._client.post>(
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1842, in post
    return await self.request(
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.9/site-packages/prefect/client.py", line 277, in send
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/prefect/client.py", line 223, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Server error '503 Service Unavailable' for url '<https://api-beta.prefect.io/api/accounts/cf175afc-09c0-4bc3-bdba-2ad01653e8ea/workspaces/7eb80c34-5efe-4f01-9ee9-64715100b7e3/flow_runs/29563420-7c6f-4680-97bf-14d148d06ee7/set_state>'
For more information check: <https://httpstatuses.com/503>
02:01:18 PM
Crash detected! Execution was interrupted by an unexpected exception.

tl;dr How scalable is Prefect Server for scheduling concurrent runs of 10s of thousands of flows? I'm evaluating Prefect Server as a solution for managing a data pipeline. How I envisioned this working was using the Prefect framework as a computationally minimal "glue." Prefect would be the central pipe through which the state of the data flows (but not the data itself). It would hand off all computation to microservices via (a)sync requests and its only concern would be controlling how the state is pushed from service to service. Prefect seems to be a great tool for this job; I really like the API and how quickly a developer can get up and running. However, there are a couple of things that I don't yet have total clarity on. This documentation describes Server as a "highly scalable scheduler" but also says that it may start degrading at "~10-20 tasks running concurrently against a typical Server." Does the above degradation occur because the

typical Server

is deployed to a single node via docker-compose with a single agent, instead of being deployed via something like Helm, which horizontally scales scheduled flows across a cluster of agents? How quickly does the size of the PostgreSQL database generally grow, relative to the number of flows run? Is there any reason that regularly deleting old runs would be any more complicated than as suggested here? Are there any case studies with Prefect Server being used to run 10s of thousands of concurrent flows? I see that there used to be a Nomad Agent, are there any helpful resources for running Prefect on a nomad cluster? Thanks for any advice you are able to share. I have been diving into Prefect for only the past couple of days so greatly appreciate any referential pointers here.

Hi - we are running into a problem with running a flow on a docker container. We get the following error:

Failed to load and execute flow run: ModuleNotFoundError("No module named '/root/'")

Our setup: • a GitHub agent packages and deploys a docker container to ECR • ECS pulls the ECR image and deploys a fargate instance • (For our proof of concept) When the machine runs, we register a flow that lives in the image/fargate instance • After that we run

prefect local agent start

(both flow registration and starting the agent are run from a script that our ECS task definition runs on boot) • We do a “quick run” and get the module not found error. Other things to note: • all of our flows and supporting modules are on this container. • I’ve read some of the existing discussion surrounding this error and was not sure if the solution applied to our situation - given that we are running everything on the same Fargate instance

Hey all, i'm struggling to connect to our Azure Devops Repo using the GIT storage class. I keep getting an HTTP Unauthorized error. I've tried connecting with git credentials and with a PAT and neither work. Has anyone else managed to connect to AZ Devops?

@Marvin, when I try to change my API key to a custom one that never expires, I get the following error when I try to login to the key in my terminal window "Cannot log in with a key when a different PREFECT_API_KEY is present as an environment variable that will override it.". How can I fix this

Hi folks. I'm trying to get my tasks to log to Cloudwatch through the ECSRun configuration. I'm testing out specifying a yaml config in

task_definition_path

, and running into an issue. My flows run on different ECR images, so I don't want to specify the image in the Task Definition file. However, when I leave it out, and specify it in the constructor:

run_config = ECS_RUN(image=ECR_IMAGE, task_definition_path='s3://<path>'

I get an error that container.image should not be empty. My task definition file that I'd like to use is:

networkMode: awsvpc
cpu: 1024
memory: 2048
containerDefinitions:
  - name: flow
    logConfiguration:
      logDriver: awslogs
      options:
        awslogs-group: test_prefect_log_group
        awslogs-region: us-east-1
        awslogs-stream-prefix: ecs-prefect
        awslogs-create-group: "True"

Is there any way to use a Task Definition file without specifying an image? Or is there another/better way to make sure my ECS tasks get logged? Thanks for any help.

@Marvin how can I automatically retry Crashed flow

Is it possible to stop a task map if any of the calls fail?

Hi there. I recently started getting some errors on VS Code using pylance. For example, the following line

from prefect import Flow, Parameter, task, unmapped

produces

"Flow" is not exported from module "prefect"
Import from "prefect.core" instead Pylance(reportPrivateImportUsage)

"Parameter" is not exported from module "prefect"
Import from "prefect.core" instead Pylance(reportPrivateImportUsage)

"task" is not exported from module "prefect"
Import from "prefect.utilities.tasks" instead Pylance(reportPrivateImportUsage)

is not exported from module "prefect"
Import from "prefect.utilities.edges" instead Pylance(reportPrivateImportUsage)

Also, I am getting pylance errors when passing Parameters to tasks that expect

str

arguments. I don't believe this was happening before. Do you know if something changed recently in pylance or in how we are supposed to import modules? I am using prefect 0.15.4 Thanks!