👋 My work pool becomes unhealthy but I have no clue of why ... The work pool is deployed in GCP Compute Engine and the jobs run in GCP Cloud Run. So my GCP engine has very limited resources (eg: 1CPU, 1G of MEM) but it does not seem to be a problem since the consumption of CPU and MEM remain under 50%. So how can I understand why my work queue keeps dying after a couple of hours?

Hey has anyone had any experience setting up a DaskTaskRunner for fargate on a subflow? I’m currently trying to use the default task runner for my parent flow but dask for a subflow and am experiencing some weird behavior once running parallel tasks. I’m not sure if this is related to dask or how I am setting up the task runner. I can confirm that behavior is normal when configuring the dask task runner for the parent flow and not the subflow.

hey all, I'm trying to register a flow in v1 but I keep getting timeouts. I'm trying to set my timeout to 60 per this PR https://github.com/PrefectHQ/prefect/pull/4119 using

PREFECT__CLOUD__REQUEST_TIMEOUT=60

but don't think it's quite working. does anyone have any experience with situations like this? full stack trace in thread

How does prefect cache? • I’m looking at Luigi, which just saves out artifacts as I’ve determined (pickle, joblib, csv, etc.). I want similar behavior in Prefect because I have computationally intensive intermediate tasks that i’d like use for analysis. • In Luigi, i can delete the artifact to re-start the task. In prefect can I clear the cache of a particular task?

Hello

prefect deployment build

Is it possible to set the cron timezone with this command?

Hi, need some help on troubleshooting an issue I’m facing. I’ve been trying to deploy my flow using S3 as my storage and have an EC2 instance (Linux) run my agent. I followed this tutorial btw. The agent seems to be running cause it triggers at the scheduled time but I’ve been repeatedly getting this error though:

Downloading flow code from storage at '/deployment/'

Flow could not be retrieved from deployment.

And the traceback ends with (full traceback in thread):

AttributeError: module 'lib' has no attribute 'OpenSSL_add_all_algorithms'

Running the agent locally is perfectly fine so it’s only when deployed to EC2 that I’m getting the error. I’ve checked a number of things already including referencing this thread from Brian: • EC2 is able to connect to S3 and download files (used AWS CLI here) as well as run

head_object()

(Python - boto3 here) • Reinstalled

s3fs

. My error isn’t quite the same as Brian’s but tried it anyway. • Changed my versions of

pyOpenSSL

and

cryptography

following the first answer here. • made sure

path

and

entrypoint

are present and correct (side note that my flow is quite deep into my directory) Some additional notes: • Deployed using Python

Hey all, I need help so if I use this combination [ cache_key_fn=task_input_hash, cache_expiration=timedelta(seconds=20), retries=1, retry_delay_seconds=20 ] for a task then what will priortize i.e. if task crashes or fails and retries with same input will cache ignore that and run the task again or it will respond the same response from failed state without running the task again ? Also as I have kept cache_expiration _time and retry delay time equal so would it allow to run task again without any cached response ?

Hi team, any idea why when I run a parent flow in prefect that has 12 subflows which each upload a CSV to Google Cloud Storage, why it says

killed

after running for a while fetching my csv from online? Memory shouldn't be an issue here... very confused

Hi all! Since yesterday my flows have been failing with the error message: Submission failed. KeyError: “No class found for dispatch key ‘azure-container-instance-job’ in registry for type ‘Block’.“. I suspect this error is popping up due to the recent updates but am unsure where it came from. I was using docker image prefecthq/prefect:2-python3.9, which updated recently, so I since updated my project to Prefect 2.8.1 and now run the flow on a docker image using prefecthq/prefect:2.8.1-python3.9. When I inspect the AzureContainerInstanceJob from my own laptop (the flows run in Azure Container Instances) I can fetch the block without issue. ” AzureContainerInstanceJob(type=‘container-instance-job’, env={}, labels={}, name=None, command=None, aci_credentials=AzureContainerInstanceCredentials(client_id=‘xxxxxxx’, tenant_id=‘xxxxx’, …) ” Any have a sense of direction for me? Thanks!

2.8.1 broke ecs task submission, keep getting "No class found for dispatch key 'ecs-task' in registry for type 'Block'." thought it could be prefect_aws wasn't installed but that wasn't the case. Reverted back to 2.8.0 and working again now.

Hi, 2.8.1 is breaking the

from prefect.orion

import path and is therefore, unfortunately, not backwards compatible 😢

Hi, running the following (taken from https://docs.prefect.io/concepts/database/#in-memory-databases):

prefect config set PREFECT_API_DATABASE_CONNECTION_URL="sqlite+aiosqlite:///file::memory:?cache=shared&uri=true&check_same_thread=false"

Outputs:

Unknown setting name 'PREFECT_API_DATABASE_CONNECTION_URL'.

What am I missing?

Hi Team, It may be the silly question but i would like to know howl ong prefect 1.x available with support. Do you have any plan to stop supporting for prefect 1.x?

Hi everyone. I hope you're doing good. I have a question. Suppose I want to run my flows with my custom image in k8s. I can specify a custom image in infrastructure blocks, what should I update it manually whenever I build a new image? Is there an automated way to do it? Thanks.

Hello, when delete a deployment I assumed that all related flow runs will also deleted. But flows keep running state, no deployment connected to them, I also cannot delete it in the UI. It throws error. How can I gracefully delete a deployment?

Hi, I'm having an issue running a flow with a large number of mapped tasks (1k+, DaskTaskRunner, LocalCluster, multiprocess, async tasks, prefect 2.7.12, prefect cloud). When running my flow everything seems be executing well at first, however eventually I start seeing the below error on a small portion of tasks (10-25/1k), those tasks seem to stall, and the flow exits despite the tasks still running. Has anyone seen this before?

Task run '0f3019f5-5551-4404-9f88-05274da187d8' received abort during orchestration: This run cannot transition to the RUNNING state from the RUNNING state. Task run is in RUNNING state.

My ECSTasks are timing out after they reach PENDING status, even after I tried adding ECS full access. My setup guide was the Prefect 2 Fargate guide on Medium (using Prefect 2.7.12). What might be the cause of this?

I tried to run a simple local flow and it failed with this error.

RuntimeError: There is no current event loop in thread 'AnyIO worker thread'.

Inside one of my tasks, I am using asynio and aiohttp to request some data from a RestAPI.

Hi all. I'm trying to connect to a MSSQL database from within a Prefect flow using pyodbc. When I deploy my flow using a Process infrastructure block, it works fine, but when I deploy it using a Docker infrastructure block, I get a "login timeout error". I get the same result when I try to invoke a telnet shell command to validate my connection to the MSSQL server. I've deployed a huge variety of connections to other kinds of servers and databases using Docker infrastructures and never seen this issue. Any ideas why my connection would timeout from inside a container but not outside of it? I added a GET request to https://ipinfo.io/json to verify that my IP address shows the same regardless of whether I deploy to a Process or a Docker container.

Hi there. I've been trying to migrate all of the flows that use

shell_run_command

from

prefect-shell

into the new

ShellOperation

class, however, I'm always running into the following error when I try to run ANY command. This is using WINDOWS. Examples:

ShellOperation(commands=["echo Hello"]).run()

does

read_logs()

with the python client work for everyone else? The following code used to return flow run logs:

async def get_logs(start_ts, end_ts):
  async with get_client() as client:
    logs = await client.read_logs(
      log_filter=LogFilter(
        timestamp={ "after_": start_ts, "before_": end_ts }
      ),
    )
    return logs

but if I query anytime after

2023-01-18

for

start_ts

it returns en empty array. I have flow runs after that date and I see stuff has been logged when accessing prefect cloud, so it’s strange to me I would see different results from the client vs the cloud UI

When creating a

DockerContainer

block, I see that I can pass in environment variables using the

env

argument. Is it possible to point to an

env_file

instead? I have some password information that I store in my .env file that I would prefer not to be shown in the UI.

I'm getting an error in my flows, that I can't figure out.

ERROR   | Flow run 'jovial-okapi' - Crash detected! Execution was interrupted by an unexpected exception: RuntimeError: dictionary changed size during iteration

Will prefect cli code like

prefect agent start -q queue1 -q queue2

still work as is after upgrading to prefect

v2.8

given the change to worker pools and queues? That is, do you have to specify work pools with

v2.8+

? I will use work pools but haven’t put time in yet to sort out what fits our use case best.

Question regarding result persistence and restarting failed flows: I have a flow that resembles this that I am running locally and on kubernetes

@flow
def demo_flow(x=1):
    val = task_1(x)
    val = task_2(val)
    val = task_3(val)
    val = task_4(val)
    val = prone_to_fail_task(val)
    return val

If expensive_prone_to_fail_task fails, I want to be able to restart my flow and have it retry just

prone_to_fail_task

again. What do I need to achieve that? Will adding

cache_key_fn=task_input_hash,

to just

task_4

be sufficient?

We are running orion behind a simple https proxy service, which is fronted by an AWS ALB. We updated our dev instance to 2.8.1 and are puzzled that when we access the UI through proxy service, we see Work Queues in the navigation bar, but when we access it directly we see the Work Pools in the navigation bar. There is no caching in the proxy and it is a simple pass through https Proxy, with an authentication step. http://orion-dv.dev.<domain>:<port>/ui-settings returns

{
  "api_url": "<http://orion-datavinci.dev.immersa.io:4200/api>",
  "flags": [
    "work_pools"
  ]
}

and https://orion.dev.<domain>/ui-settings

{
  "api_url": "<https://orion.dev.immersa.io/api>",
  "flags": []
}

FYI

/settings

endpoint using both URLs returns the same settings. Have looked at the code in

server.py

that handles

/ui-settings

and

experimental.py

but that does not explain why the response differs for

/ui-settings

when going via proxy vs direct ? Any ideas ?

Hello, does AWS S3 block storage work with S3-compatible service providers (such as Digital Ocean - Spaces)?

Hi, I'm trying to get

orion

running locally in Kubernetes using K3D. Setup: • OS:

macOS Ventura 13.2.1

• Docker: Docker Desktop • K3D:

k3d version v5.4.7, k3s version v1.25.6-k3s1 (default)

(the output from

k3d --version

) I'll attach the config files below for: • K3D Cluster Setup • Prefect Orion Deployment and NodePort service Issue: Since K3D is running K8s inside of a docker container, I need to open a port. I can set up the cluster to have an open port (I've chosen 30001) and then I can set up a NodePort in the cluster that listens externally on 30001 and forwards to the Orion pod on 4200. If I set the startup parameters for Orion to be the following ...

command: ["prefect", "orion", "start", "--host", "0.0.0.0", "--port", "4200", "--log-level", "DEBUG"]

... then I can connect to the Orion UI using the following URL ...

<http://127.0.0.1:30001>

... and this connects through to Orion and shows me the Flow Runs screen as the default. However, I receive the message ...

Can't connect to Orion API at <http://127.0.0.1:4200/api>. Check that it's accessible from your machine.

... which I guess means that the UI is using AJAX to communicate back to the server, but it can't get to it using port 4200, as it needs to be 30001. If I set the startup parameters for Orion to be the following (note the change in port) ...

command: ["prefect", "orion", "start", "--host", "0.0.0.0", "--port", "30001", "--log-level", "DEBUG"]

... then when I connect to the Orion UI using the following URL ...

<http://127.0.0.1:30001>

... the page is completely black. The URL changes to be forwarded to ...

<http://127.0.0.1:30001/flow-runs>

... but the page is simply black. If I try to refresh the page I receive the message

the server unexpectedly dropped the connection

in Safari, and

127.0.0.1 didn't send any data. ERR_EMPTY_RESPONSE

in Chrome. I'm sure that this is a silly config issue, but I can't work out for the life of me what it is and I've spent a silly amount of hours on this. Any help is much appreciated. Thanks. Files: •

k3d-config.yaml

- config to create the cluster. •

orion.yaml

- the Kubernetes Deployment and Service config.

Hey, while using prefect cloud I'm having trouble for users in my organization seeing our workspace - we defined the SSO but when people are logged in they don't see our workspace

Did anyone encounter this problem?