Hello team! I am Joseph and new to prefect, which material can I use to get start please?

Hi Team, Sometimes we are seeing task failures in ECS with this error

Failed to start task for flow run ad860ca0-61df-4055-b3f6-7f48c2d9ead4. Failures: [{'arn': 'arn:aws:ecs:us-east-1:XXXXXXXXXXXX:container-instance/XXXXXXXXXXXX', 'reason': 'RESOURCE:CPU'}, {'arn': 'arn:aws:ecs:us-east-1:XXXXXXXXXXXX:container-instance/XXXXXXXXXXXX', 'reason': 'RESOURCE:MEMORY'}, {'arn': 'arn:aws:ecs:us-east-1:XXXXXXXXXXXX:container-instance/XXXXXXXXXXXX', 'reason': 'RESOURCE:MEMORY'}]

Understandably, its because enough resources are not available on ECS nodes to schedule the tasks when launch type is EC2. But as per my understanding, in such cases the ECS task should get scheduled with "Provisioning" state, and ECS cluster should scale out and add more instances after which the task would run. However in our case we see above error and the task just fails to schedule. Is there a specific parameter we are missing which will provide the ECS feature that i mentioned above ? or this is the expected pattern ?

Hi there, did anyone get a Docker infrastructure with a private Docker registry in 2.3.0 to work? I defined a "Docker Registry" and a "Docker Container" block and used the Docker Container block as infrastructure block in the deployment.

prefect deployment build myflow.py:myflow \
--path /prefect/flows/myflow \
-ib docker-container/dev-image \
--name local_docker \
--skip-upload \
--apply

After running the deployment it breaks with "no basic auth credentials". Inspecting the "Docker Container" block shows that it should use the defined registry block. Double checked the credentials multiple times and successfully logged in with

docker login...

I tried the same on 2.2.0 with the same error after executing the deployment :-(

set the channel topic: Welcome! 👋 -Please put code & tracebacks in 🧵 -✅ means addressed, not always resolved -:github: bugs & feature requests : https://github.com/PrefectHQ/prefect/issues -Discourse: https://discourse.prefect.io

Is there planned to be any concept of "agents" in the api/UI that shows the different agents that have been connected over time, when they were last active, etc?

Hi There, I just started with Prefect v2, and trying to figure out the following: 1. Let’s assume we deployed a new pipeline to Prefect Server, and this new pipeline has new dependencies (external python libraries), How we can automate the installation of these dependencies on the agent execution environment while guaranteeing high availability (assuming that pipelines are running 24/7 on our servers)? 2. In the case of an Agent deployed on Kubernetes, How does the agent mange the workload across the Pods, is it automatically handled by the Agent, or do we need to handle this from our side?

Is there any way to change the timezone on the UI in 2.0?

Is it possible to run tasks in docker environment? Usecase: I have a Flow composed of 3 tasks. First two are pretty simple, and can be done without using any external python library. The third one requires some external python libraries and a few system dependencies to be installed. Is there a way to run the first two tasks on the host environment, and the third in docker?

Hi everyone - we're still on Prefect V1 for Q3. We had a failing heartbeat with a long-running API backfill. I did switch our configuration to 'threads' as documented here, but two backfills failed again last night (while a third smaller one succeeded). What's interesting is that the ECS Cluster is still active but the agent has stopped responding to new incoming requests, and there's likely a zombie job in Prefect due to this. The specific question I have is "Has anyone else had a problem of a new FARGATE job stopping the agent itself from responding?" I thought, based on the configuration, that those would have been on separate FARGATE clusters on ECS.

Hey all, we’re hosting @Will Raphaelson, a senior tech product manager at Prefect, on PrefectLive today at 3p Eastern where he’s going to show the latest improvements to deployments, in particular using Docker and Github! Come join us live on Twitch!

hey team! I'm working with prefect 1.0 and curious about triggering a flow in one tenant, when a flow in another tenant finishes successfully. I think it may be possible with a graphql query similar to this post https://stackoverflow.com/questions/68981899/create-flow-run-on-localhost-with-http-request-how-to-pass-parameters-graphql. I'm curious if anyone has any alternatives that they think are better. I'm also trying to find my graphql endpoint from these instructions https://docs-v1.prefect.io/orchestration/server/deploy-local.html#ui-configuration, but I don't see a prefect server tab, anyone know what's going on?

Hi I'm deploying prefect v1 on a VM with Ansible. I actually have a playbook to relaunch my prefect agent when needed with a kill and a launch with nohup. If my VM crash it doesn't relaunch the prefect agent as I use nohup Does someone here have ever done a linux service or a daemon to handle prefect agent ?

Hi Everyone, When I create a deployment, the deployment should be uploading all my files in my current directory to my storage (in this case, s3), however, this isn't happening on prefect 2.3.0 I've triple checked my

.prefectignore

and no, it is not ignoring anything critical. Is this a bug?

Hello, is there a documentation on how to set up prefect orion server? e.g. other than

PREFECT_API_URL

what other things need to be configured? I keep getting

[Errno -2] Name or service not known

I have prefect orion set up with helm in EKS

I am using prefect2.0. I have the following flow that I run in a docker infrastructure.

@flow
def my_flow():
    return "hola"

Using the OrionClient I am able to read the states of a flow run using

orion_client.read_flow_run_states(flow_run_id)

. The last state is

COMPLETED

and has a

data

attribute that is NOT

None

. For example the data attribute value is:

state.data = _Result(key='b1fd11041ae54cfcbb9b41ec938ff0cf', filesystem_document_id=UUID('bd1ed851-82d2-4f87-b11c-be04a80634af'))

What does this represents? Why don't I have

state.data = "hola"

instead? How do I get my flow return data from the OrionClient? Thanks a lot!

Hi everyone, I tried to set a pipenv environment in my deployment but when I ran the pipeline the environment didn't start up. Does anyone know how to fix this? Thanks!

Hey all, Prefect Live will start in 5 minutes (today at 3p Eastern). We’re hosting @Will Raphaelson, a senior tech product manager at Prefect, on PrefectLive where he’s going to show the latest improvements to deployments, in particular using Docker and Github! Come join us live on Twitch!

Hello. in Prefect1, I have a task in a flow, and I want to reuse this task in another flow 2 times with different parameters, with the second run being dependent on the fist one. Something like this:

../flow_1.py
    @task(my_task)
    def my_task(param)
.................................
from flow_1.py import my_task
../flow_2.py
    @Flow(some_flow):
        my_task(param_1)

        my_task(param_2, upstream_tasks=[my_task])

Since the 2 tasks in my Flow in flow_2 have the same name, would

upstream_tasks

work? Is there a better way to do this? Thank you

hi there! i'm trying to understand what is happening with subflows, infrastructure-wise. in particular, where they're going to be executing. more detail in thread

Hi, I’ve got a question about the REST API. The docs for the REST API say that I should be able to issue

GET /collection

to list all members of a collection, such as

GET /api/flow_runs/

. However, the server, OpenAPI spec, and example at http://localhost:4200/docs#/ say that I must supply an id to the collection, such as

GET /api/flow_runs/1234

. What the correct way to fetch the full list of collection members?

Hello, 2.0 question. For my kubernetes deployment, I've found a way to modify the flow yaml so that it will automatically delete the job pod shortly after completion. That said, it's not really scalable to manually change these every time. Is there a way to add the spec.ttlSecondsAfterFinished value via prefect cli or is that coming #soon?

In Prefect 2.0, does GitHub storage work with private Git repos? I don't really see an option to pass in creds or an api key to git

is there any change to the personal prefect UI? I cannot no longer open it:

Question, I have a simple flow that runs a single Postgres function through the SqlAlchemy module. I have orion running on a remote node, and I am running an agent locally and connecting to a test queue to try things out... I am getting multiple min delays between the flow starting and the task starting though

Hi all 👋 Quick Prefect 1 version (potentially 2.0 too, dunno yet 😄) Self-hosting with Docker, with the URL of Slack webhooks, is it possible to configure to a custom domain, rather than the default

localhost/4200

that we can’t reach without connecting to the VM etc..

Hi everybody! I’m having trouble getting the

DaskTaskRunner

working in this tutorial: https://docs.prefect.io/tutorials/execution/#parallel-execution I get a bunch of errors, e.g. “no active connection” , or “TypeError: DaskTaskRunner.submit() got an unexpected keyword argument ‘key’”

Also, I am having other (file permissions?) failures when trying to run on my deployed infrastructure rather than locally for the exact same flows mentioned above (see log in thread) I wasn't able to find anything pointing me to a resolution here... the files exist in the specified deploy location, they work in my local docker container, they don't work on this VM.

Hi All. My flow fails because a task returns "No heartbeat detected from the remote task; marking the run as failed". I used some log statements to debug, seems that the flow doesn't even start. Please suggest what I can do. Thank you.

Hi, we are randomly experiencing an issue with our prefect scheduler. Some days, some flows get submitted late. We don't see any issue with resources on the hosting servers and we don't think it is a clock issue, as other days, same flows run on schedule fine. Perhaps someone here has experienced this and has a pointer. Note: both server and agents are deployed on the same server, run out of simple python virtual environment.