Hi, I am installing prefect on-premise, unable to set database URL to postgres, any ideas?

root@dataeng-prefect:~# prefect version
Version:             2.7.12
API version:         0.8.4
Python version:      3.8.16
Git commit:          524c25cd
Built:               Mon, Feb 6, 2023 4:31 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         <client error>
root@dataeng-prefect:~# prefect config set PREFECT_API_DATABASE_CONNECTION_URL="<postgresql+asyncpg://postgres:yourTopSecretPassword@localhost:5432/prefect>"

Unknown setting name 'PREFECT_API_DATABASE_CONNECTION_URL'.

Hi everyone, I try to publish notification to mattermost channel, The channel is within a specific team group. How can i publish a notification to different channel on different team group. example to publish webhook from jira: "mattermost.com/webhook?secret="TOKEN"&team="TEAMGROUP"&channel="prefectchannel" "

HI. I have a flow in Prefect 1 that has tasks that maps to send COPY INTO statements to snowflake using the SnowflakeQuery task. It only runs 2 sql statements at a time, I’d expect this to scale as our snowflake warehouse can scale up to 5 clusters. We don’t have any concurrency limits set in the UI. Does anyone know what would limit this?

Hi Prefect Community ! I have an issue that happens time to time with my flows A task keeps running indefinitely (even if I set a timeout) while my flow is crashed And the flow does not retry after it crashs. It happens randomly so I couldn't reproduce the error I am using Prefect

2.7.5

. Any idea ?

Hi! Are there any beginner tutorials on how to set up CI/CD on Azure for prefect deployments? After pushing to master branch, I want the deployments to be created automatically + building docker image and running a few tests.

Hey everyone! What's a simple way to send an alert / slack message when one long running flow has had many failed tasks? Our flows can have hundreds - thousands of tasks and we'd like to know when, say, 30% (after maybe 1000) of them have failed.

Hi again, unable to set API and UI to external IP

root@dataeng-prefect:~# prefect version
Version:             2.8.2
API version:         0.8.4
Python version:      3.8.16
Git commit:          afbed19d
Built:               Fri, Feb 17, 2023 10:02 AM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         server

set the API and UI to external IP, start the server but no luck

root@dataeng-prefect:~# prefect config set PREFECT_API_URL=<http://172.27.24.162:4200/api>
Set 'PREFECT_API_URL' to '<http://172.27.24.162:4200/api>'.
Updated profile 'default'.
root@dataeng-prefect:~# prefect config set PREFECT_UI_API_URL=<http://172.27.24.162:4200>
Set 'PREFECT_UI_API_URL' to '<http://172.27.24.162:4200>'.
Updated profile 'default'.
root@dataeng-prefect:~# prefect server start

 ___ ___ ___ ___ ___ ___ _____ 
| _ \ _ \ __| __| __/ __|_   _| 
|  _/   / _|| _|| _| (__  | |  
|_| |_|_\___|_| |___\___| |_|  

Configure Prefect to communicate with the server with:

    prefect config set PREFECT_API_URL=<http://127.0.0.1:4200/api>

View the API reference documentation at <http://127.0.0.1:4200/docs>

Check out the dashboard at <http://127.0.0.1:4200>

Hi! Is there any way to create an automation in Prefect Cloud based on late flows, but only flows that are late by a specific duration?

Hey, Prefect community ! Does a hosted docker image with the package

prefect-aws

pre-installed exist? Thank you 🙂

Hello. Is it possible to use AWS session tokens with S3 storage blocks? There isn't a field for them in the UI.

In Prefect 1, the context object had a

flow_id

property that we were accessing the following way using

prefect.context.get("flow_id")

. In Prefect 2, I am looking for the equivalent property in the context object, but have only managed to find a

flow_run_id

. Am I missing something here?

Hi , how can I create a S3-compliant storage block in prefect 2.0, to access that storage in boto3 I do the following

s3 = boto3.resource('s3',
                    aws_access_key_id=access_key,
                    aws_secret_access_key=secret_key,
                    endpoint_url=endpoint_url)

Hello Everyone!! I wanted to know what is a recommended configuration to run Prefect Server? Is it single threaded ( since it runs on python? ) will it make use of multiple cores? ( ex: 2, 4, 8, 16 ). What is the recommended RAM for server? ( 4GB, 8GB, 16GB )? etc. Are there any benchmarks from prefect that can be viewed? I tried to find the recommended system requirements in the docs, issues, Q&A, discourse etc but couldn’t find anything. Hoping to find some help here. Thanks.

Why does prefect keep adding

Downloading flow code from storage at ''

into all my logs. Is there any way to disable it? It gets added multiple times and is not helpful, the system never use to do this in the past.

I have been trying to deploy an async flow, but I am pretty sure I'm missing something. Using this snippet from Anna:

import asyncio
import random
import time

from prefect import flow, task, get_run_logger


async def async_func():
    if random.random() > 0.3:
        time.sleep(1)
    await asyncio.sleep(random.randint(1, 5))
    return 1


@task
async def my_task():
    result = await async_func()
    return result


@flow()
async def my_flow():
    tasks = []
    for _ in range(500):
        t = my_task()
        tasks.append(t)

    await asyncio.gather(*tasks, return_exceptions=True)


if __name__ == "__main__":
    asyncio.run(my_flow())

How can one create a deployment for this? Would you still be able to use

Deployment.build_from_flow()

with this? And if so, would this still work with deployment parameters?

Hi guys, I am running a prefect server (

prefect server start

)on a oracle free tier. I tried to connect to the server, but I figured out, that I first had to add some ingress rules in my oracle VNC settings and open my firewall. I´ve s opend 80 and 4200. I can connect to a running apache server (http://myip:80), but I can not reach the prefect UI (http://myip:4200). Do I have to open some other ports too?

Hi all, I am trying to remove logs in Prefect Cloud like Created task run..., and have not had any success. I followed the suggestion here but for some reason cannot override the default Prefect logging configuration with my own. I am deploying through GitHub actions and hosting infra on GCP Cloud Run. Anybody have a similar experience? This is worsened by the fact that Prefect does not throw an error if the custom configuration is not found, but instead moves forward with the defaults

I am getting a weird behavior when deploying flows with prefect 2.0 I was able to deploy it locally (running Deployment.build_from_flow). I deployed the flow to a server but for some reason it errors saying that i can't find the path (path from my local computer).

What would cause this error to appear suddenly when running flows locally in Anaconda? I've been able to run them for 1+ year just fine and never had this issue.

Does anyone have experience deploying Agent on render.com webservices? I seems to work but the deployment never finished. Thank you.

Hello, I get

Data integrity conflict

when I delete a running flow run. How can I solve this? I am using prefect 2.7.12 BTW it works when using a sqllite db, but fails in postgresql. The error message is:

<class 'asyncpg.exceptions.ForeignKeyViolationError'>: update or delete on table "task_run" violates foreign key constraint "fk_artifact__task_run_id__task_run" on table "artifact"
DETAIL:  Key (id)=(51c7df11-9d79-4370-ad77-ba98d8a7989f) is still referenced from table "artifact".

Is there any way for the code to know which environment is it running in? I would like to write some conditionals to separate local and cloud logic.

Hi asked similar question on another channel, but thought this channel might be better. I am playing around with Prefect 2 and I ran into 3 questions about running deployments: 1. is it possible to retrieve a flows result when ran with prefect.deployments.run_deployment? 2. when you persist data from a flow/task, is there a way to get the file name? or set up the location to be determined by a function so the file can otherwise be found? 3. Is it possible to manually read the data from a PersistedResult file? questions 2 and 3 are admittedly related to each-other. Any help that can be provided on the above will be appreciated

Hi guys. 👋 How can I specify an image pull secret for a Prefect job? I tried to passing a job manifest in infra. block's Base Job Manifest configuration, but it didn't work. In what format it expects custom template? Thanks.

Hi all, I'm trying to run a flow in a docker image, I managed to deploy and apply the flow, while running I get the following error

+ echo 'Non-spark-on-k8s command provided, proceeding in pass-through mode...'
Non-spark-on-k8s command provided, proceeding in pass-through mode...
+ CMD=("$@")
+ exec /usr/bin/tini -s -- python -m prefect.engine
[FATAL tini (14)] exec python failed: No such file or directory

looks like the image should have some mandatory software ? do I need to install prefect inside this image? are the pre-requisites documented anywhere ?

How does prefect store external credentials? Say I'm running workflows on the prefect cloud that interact with my AWS stack like querying from Athena. How are my AWS credentials stored on the prefect cloud?

Hello! I'm working with flows and "sub-flows" with Kubernetes as our infrastructure. Is it possible to have the child "sub-flows" run in the same pods dispatched (affinity) for the parent flow? It seems that since run_deployment talks to the Prefect server, this may not be possible. Thx! rb

Hi 😄 I was wondering why we can only write or read bytes on prefect filesystems. Why aren't there any method to write more conventional files like images or videos ?

Hi, I have an agent running on GCP compute engine. Over the weekend the work queue became unhealthy and flow runs are now late. I have tried restarting the agent but it doesn't pick up that there are any scheduled or late flow runs. If I try to run a flow through the UI by quick run the flow goes into a scheduled state and never runs. When I cancel that flow run the agent picks up that the flow run should be canceled like this

19:01:29.664 | INFO    | prefect.agent - Found 1 flow runs awaiting cancellation.
19:01:29.665 | ERROR   | prefect.agent - Flow run '63ee09b8-7bc9-4df9-a690-e1d6f77fbf98' does not have an infrastructure pid attached. Cancellation cannot be guaranteed.

Im not sure whats going on with the agent

Hi Prefect team 👋 I’m new to Prefect and am evaluating a couple of orchestration tools, am hoping to get a scheduled deployment running so I can experiment with Prefect. I cloned the prefect-cloud-gcp repo (https://github.com/anna-geller/prefect-cloud-gcp) to my local environment and followed the instructions from @Anna Geller’s post step by step, but the Github Actions run continued to fail at the below step, after I double-checked my authentication steps on Google Cloud Shell & blocks secrets. Would it be possible for me to work with someone on the team to troubleshoot - I can share more of the error messages in a separate thread as it contains my GCP account info. Would appreciate pointers 🙏

GitHub Actions: Plan of Action

The “Getting Started” GitHub action does the following:

1. It creates an Artifact Registry repository if one doesn’t exist yet.

2. It builds a Docker image and pushes it to that Artifact Registry repository based on the

Dockerfile