Hello I have started using prefect for airbyte orchestration task. Below I have attached my code

import prefect
from prefect import task, Flow

@task
def claims_func():
    airbyte_server_host = "localhost",
    airbyte_server_port = 8000,
    airbyte_api_version = "v1",
    connection_id = conn_id,



with Flow("fhir-flow") as flow:
    claims_func()


 
flow.run()

As per my understanding when when I run this flow it should trigger the connection and automatically sync data in from gcp->snowflake. I don't get any error but no data is synced.

@Marvin how to refactor a code that uses asyn..await syntax to work with Prefect?

@Marvin what is the simplest way to setup slack notifications on job failures?

@Marvin why would a server fail to start with this error:

sqlalchemy.exc.DBAPIError: (sqlalchemy.dialects.postgresql.asyncpg.Error) <class 'asyncpg.exceptions.DeadlockDetectedError'>: deadlock detected

Hi, i have a problem every time i run my flow with docker show de message before starting the flow and the following error showing: DockerException( docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied')) The above exception was the direct cause of the following exception: RuntimeError: Could not connect to Docker. The flow build with sucess. prefect deployment build HOTSPOTS/hotspots.py:flow_function --name 'hotspot-docker' --work-queue 'hotspots' --storage-block 'local-file-system/local-storage' --path 'HOTSPOTS' --infra-block docker-container/hotspot-container Does anybody know how to solve this?

Hey folks. Is there a way to have Prefect treat a file on disk that was created during a task’s execution as a Result, that can be persisted using S3Result (for example) without loading that file into memory / returning it from the task function? I am trying to see if my organization can use Prefect, but our core need is to run 3rd party programs that produce large output files that may not fit in memory. Thanks and I apologize if this is already in the docs.

What is the correct way to set the

imagePullSecrets

value in the

prefecthq/prefect-server

helm chart? Tried multiple ways, but none seem to work (examples in comments)

@Marvin How to trigger a flow instantly via REST api in Prefect 3

Hi, new to Prefect and Python so apologies if what I’m about to describe is a pathetically trivial/obvious problem. Here’s my environment: 1. Prefect Cloud for orchestration. 2. Azure Kubernetes Prefect Agent (note I don’t have an executor defined in Azure as I’m not sure how to do this yet, or whether I even need to, but it’s important that the entire Flow runs in Azure - ideally on the existing Agent if possible). Have imported KubernetesRun from

prefect.run_configs

in my Flow, and can see the Agent from Prefect Cloud, and was able to run ‘hello world’ successfully via it also. 3. Azure PostgreSQL database I need to run SQL against as part of a Prefect Task (have used a Secret in Prefect Cloud to create a dictionary with all the database login parameters) 4. Private API I need to call as part of a Prefect Task 5. GitHub storage for the whole Flow script (created a GitHub Access Token Secret to enable PrefectCloud to access the registered script) 6. Have run the

prefect backend cloud

CLI command on my local Mac, to hopefully force everything to execute in Azure rather than locally Here’s the journey I’ve been on: 1. I got the basic “hello world” flow executing on a local agent (yay!), with the Flow registered in the Prefect Cloud, and the code being pulled from GitHub 2. I then got it working against the Kubernetes agent, but for some reason the ‘hello world’ message didn’t appear in the logs. Is that because I need to set

prefect.config.cloud.send_flow_run_logs = True

somewhere/how, or something else? 3. Tried importing psycopg2 and creating a connection to the PostgreSQL database to retrieve a very simple count result. Am not sure if I need to be using the PostgresExecute API call (which itself uses Psycopg2) rather than importing psycopg2 into my Flow. Thoughts/recommendations welcome! 4. Also tried importing the requests Python module to call the API. 5. When I try running the script to register the Flow it appears to work (finished with exit code 0), but when I look in my Prefect Cloud Flows tab it’s not there. Any idea why, please? Sorry to bother you all about this, but am kind of at a loss on how to move forwards with this if I can’t even see the Flow I want to execute. Suspect it may be something to do with importing those libraries/modules and them not being available in the Execution environment, but have no idea from what I’ve read so far what I need to do to get that working. Any advice (even RTFM, if you can point me to the right topic) would be gratefully received. Thanks in advance! * UPDATE * Have posted the script to the below thread as requested by @Kevin Kho Huw

@Marvin after upgrading prefect3, I cant see flow task diagram at flow runs widget. It only say message "This flow run did not generate any task or subflow runs". Why?