When using

LocalDaskExecutor(scheduler="processes")

, logs from

prefect.context.get("logger")

don't make it to Prefect server, but using

LocalDaskExecutor(scheduler="threads")

works just fine. Am I missing something? I've opened an issue https://github.com/PrefectHQ/prefect/issues/5895 - but just thought I'd check if anyone has dealt with the same problem here

Hello! We're trying to figure out the conditional logic for a piece of our flow and struggling to get it right. The problem is that I want

unlock_hosts

to wait for the entire

with case(did_deletes_fail, True):

block (starts on line 16) to finish. But since I can't name that

case

task, I can't set that state dependency. If we get to the

update_state()

task, we need to immediately fail the flow and NOT run any other task, especially

unlock_hosts()

and

daemon_start().

I thought maybe I should use a

switch

instead, but my "b branch" is "do nothing". What conditional logic and dependencies should I be using here? Thanks!

is there a way to get additional insight to what api calls are being made under the hood of the FivetranSyncTask? i’m running it like so and for some reason it’s running all of my syncs as manual syncs even though i have manual set to

False

. Code in thread

Hi, we are experiencing random errors from prefect cloud while running a flows. We are reading the secrets and have the following error:

File "/usr/local/lib/python3.9/site-packages/prefect/client/secrets.py", line 148, in get
    result = self.client.graphql(
  File "/usr/local/lib/python3.9/site-packages/prefect/client/client.py", line 473, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['secret_value'], 'message': 'An unknown error occurred.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

Server is randomly returning 500 error. It's working 9/10 times, however, i would expect this to work 10/10 so we don't need to restart flows manually. Has anybody experienced the same problem?

Hi, I've got a problem running flows from Prefect Server in Kubernetes. It was working at one point but now the flows gets stuck in a

Submitted for execution

state. I found these docs and tried debugging/restarting services with them but no luck:

<https://docs.prefect.io/orchestration/faq/debug.html#my-flow-is-stuck-in-a-submitted-state>
<https://discourse.prefect.io/t/why-is-my-flow-stuck-in-a-submitted-state/201>

I also added these flags to the kubernetes prefect agent:

--log-level DEBUG --disable-job-deletion

That gave me more detail but still no indication of what the problem is. Any help would be appreciated, thanks!

I’m running Prefect Server in an airgapped environment with certain restrictions on ports. I wasn’t able to see the UI at all until I used

--expose

and

--ui-port <UNRESTRICTED_PORT>

. So now I can see the UI framework but it isn’t pulling actual data and things remain looking like in the attached screenshot. What do I need to know about how data is pulled from the other containers in order to troubleshoot this? I tried setting other ports to numbers in the unrestricted range but those caused various problems (server startup not completing, or commands like

prefect create project

hanging). In order to troubleshoot these two issues, what information could help me? How does the UI get data from other containers (and which ones), and what could be preventing the communication? BTW Prefect server seems to work just fine, I can run flows successfully.

Hello everyone! I'm running flows using 1 prefect agent. However, as the number of flowruns increases, we want to increase the number of agents to more than two. But there is a problem. When two agents are set to the same label, if the concurrency of the label is 6, I would like to take 3 flowruns per agent. However, contrary to my expectation, the situation where 1 agent takes 5 and the remaining agent takes 1 continues. How do we address this imbalance? please help me!!!

I am getting an error “Flow could not be retrieved from deployment.botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden” when I am running the flow from the orion server set up in EKS. I am using S3 storage and Kubernetes Flow runner and a custom image in ECR. The deployment spec and flow is on my local machine and I deploy it using the prefect cli command (prefect deployment create <path of python file>) Any help on this is really appreciated… This is the sequence of steps that led me to the error: 1. I Modified the config on my local machine to point to our orion server in DEV.

prefect config set PREFECT_API_URL=http://<orion-dev-server>:4200/api

2. I then wrote the code of the flow and deployment spec on my local machine. Code of the deployment spec:

DeploymentSpec(
    flow_location="<flow location>",
    flow_name="flow_one",
    name="deployment-one",
    flow_runner=KubernetesFlowRunner(
        image="<image is on AWS ECR>",
        namespace="<kubernetes namespace>",
        image_pull_policy="Always"
    )
)

I then run the deployment by prefect deployment create <deployment python file> This runs fine. The file gets copied to S3 storage and I can see the deployment and flow on the orion server.

Hi any update about this issue regarding airbyte/Prefect error https://github.com/PrefectHQ/prefect/issues/5936 ?

Hi everyone I am getting this error when trying to register a flow in the cloud

prefect.exceptions.AuthorizationError: Malformed response received from Cloud - please ensure that you are authenticated. See `prefect auth login --help

It also says it is a json error , after inspecting, the response gotten from the requests library is a huge html and not a json. My current config.toml file has worked for previous registrations in other projects. Does this sound familiar?

Hello everyone, We have been using prefect 1.0 for quite some time in lower and higher environments. Recently, there was some activity ran by the security team and the server where prefect db is hosted was moved to private subnet. Right after that, several issues start popping up. • Unable to run the jobs from UI • Flows can't be scheduled because scheduler option is not working • Agent is not picking up the scheduled jobs Whole stack is deployed on AWS ECS Fargate. Issues still persist even after bringing back the subnets to public.

Hi, I’m trying to deploy Prefect 2.0 on k8s (EKS) using the default manifest provided by

prefect kubernetes manifest orion

and I’m facing some issues with the UI which seems to not be able to connect to the database (it’s empty). It’s a similar issue to this one but on k8s, not locally. The agent container communicates with the server container (API) just fine and is able to pick up flows. I’m also able to connect to the API via prefect CLI on my local machine, create work queues, deployments, trigger flows etc (which are then picked up by the k8s agent). But UI is empty and I’m not able to see/create e.g. worker queues. Any idea what I might be missing?

Hi, I am Raviraj and have been using prefect for 2 years, but recently I face a new error

for i in range(0, len(list_dict_data)):
TypeError: object of type 'FunctionTask' has no len()

Can someone tell me how can I get return data from the

FunctionTask

object?

Kind of a very weird case, but is it possible to run a flow locally, having previously registered the flow in the cloud, and have the local run reflected in the UI in prefect cloud?

Hello 👋, I am running Prefect-1 on a distant cluster. My Flows are working well and I can monitor them from the Prefect Cloud. Though, I would like to try doing it using a Prefect server. The issue is that I do not have docker on my cluster (and I cannot install it) so I cannot start a server there. I was wondering if it would be possible to run a prefect server on my local machine and connect to a running agent on my cluster through ssh tunnel or similar? If anyone has ever done it I would really appreciate some pointers or an example on how to setup something like that. Thanks 🙂

Hi, I was just going through the Prefect Orion release notes 2.0b7, it says Ray and Dask are now part of collection - “_Split Ray and Dask task runners into their own collections”_ Though if I look at the requirements.txt, I could see the reference of the Ray package - The reason I am raising is because I am using slightly older version of Red Hat that does not work with the Ray and so I had to pip install prefect without dependency and if the Ray is removed from the requirements, that would make my life easy, thanks. https://github.com/PrefectHQ/prefect/blob/orion/requirements.txt#L29

Hello Everyone, I am running prefect server on K8s. I am checkpointing certain tasks but not all. Seems like when the flow breaks and i restart it, the flow is running from the very beginning rather than the point where code broke. How can make sure the flow restarts from the task where it failed #prefect-server

I am running prefect 2.0 on aws lambda (runtime python3.9) but i get an error

"Orion requires sqlite >= 3.24.0 but we found version 3.7.17"

Does anyone know a workaround? i

This message was deleted.

Hey all, I'm getting an error upgrading from prefect

1.2.2

to

1.2.3

Installation appears to work as expected but I get the following when I run

prefect server start --postgres-port 5433

Has anyone else encountered this or did I miss some migration notes somewhere?

Hi all, Using prefect version: 1.2.3 Whenever I set the

performance_report_path

argument in

DaskExecutor

it fails with the following error:

Unexpected error: ValueError('No global client found and no address provided')

Hi, Using prefect 1.X and able to trigger a flow via a GraphQl endpoint (see sample response below). I was wondering if it's possible for a prefect flow to return a response (e.g. computed values or a generated file). { "data": { "create_flow_run": { "id": "b7f44e94-d899-4f1b-a854-98dff845a0e2" } } } Thank you

Hey all. I have a flow that is scheduled to run every day but I keep getting this error which I don't comprehend where to trace it to: [{'message': "can't start new thread", 'locations': [{'line': 2, 'column': 5}], 'path': ['set_flow_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': "can't start new thread"}}}] Any idea why? I'm using Prefect Core Version: 0.13.18

Hi all, is it possible to have different executors per step? This is useful when resource requirements are different for steps

[v1.2.1] Hello, where can I find the IP configuration for containers ? Edit: I may be the docker proxy that I need to configure to do this, no necessarily via Prefect containers , idk When I do

docker port tmp_ui_1 8080/tcp

I get

127.0.0.1:8080

but I would like to change it to

0.0.0.0:8080

:::8080

Is it possible to control how Prefect traverse the DAG? e.g. depth first? An example use case would be, I would like to prioritise all child mapped tasks for a give parent to complete first, e.g. if I had 10 tasks each map out to 10 child tasks, and I had 10workers, I would like it to complete one parent task and all 10 child mapped tasks at a time, before moving onto next, rather than start a child of each parent task, if that make sense?

How do I update the prefect server url in the Slack Notification ? At present it points to http://localhost:8080/default/flow-run/687d73b7-112b-4fca-b8c1-73c7fb12e6ae, how do I update the server ip address ? is there some config parameter ?

hey all! question! we have 3 flows kick off at the same time, all 3 flows are slightly different and take different times to execute. once all 3 flows are finished, we want to kick off ONE flow that would process data that the other 3 produced. ideally that one flow would be allowed to run once the other 3 flows are finished

hey, I triggered these ad-hoc by clicking the play button runs and they are in scheduled state since forever.

Hello. Is it possible to use

S3List

task to get list of bucket specifying full path? Whenever I try to provide full path there is an error about bucket not matching some Regex. It works only when I provide plain name of the root bucket like this

test-bucket

but I need content of the directory inside that bucket. How this can be done?