I’d like to trigger some flows whenever a new file is added to a Google Drive folder. My current plan is to run a small python server that receives webhook calls from the Drive API, then uses the Prefect API to trigger the flows. Questions: 1. Is the right Prefect API to use the GraphQL API mutations API

createFlowRun

? 2. Is there anything like this around already? Would love to not reinvent the wheel and deploy a new service.

@Marvin Hello Can you help on how we can connect to MS SQL Server Database..

I've been trying to run prefect based on a custom image. I have a rather complicated image build on a separate dockerfile. I wanted to just add prefect on top of that and cobbled up the following Dockerfile

FROM [original-image]

RUN pip install prefect smbprotocol

# I keep the flow definition in the "prefect" directory
COPY prefect prefect

# The image wouldn't work without that
ENV TZ=Etc/UTC
ENV MKL_THREADING_LAYER=GNU
ENV MKL_SERVICE_FORCE_INTEL=1

# Run our flow script when the container starts
CMD ["python", "prefect/flow.py"]

So far I've been testing it in a local docker pool and all was well. I now moved to ACI and am getting entrypoint-related errors:

Error: Failed to start container zuge6yxkhm5gm, Error response: to create containerd task: failed to create shim task: failed to create container 4d9adc93761405465a82af86e363832c0e7a043a584588e48a8ef0bb55faf532: guest RPC failure: failed to create container: failed to run runc create/exec call for container 4d9adc93761405465a82af86e363832c0e7a043a584588e48a8ef0bb55faf532 with exit status 1: container_linux.go:380: starting container process caused: exec: "/opt/prefect/entrypoint.sh": permission denied: unknown

I found that the ACI pool has a default entrypoint

/opt/prefect/entrypoint.sh

, so I cleared the entrypoint field, but no dice, I'm still getting the same error. I also tried to copy the entrypoint from prefect repo into the image (this one: https://github.com/PrefectHQ/prefect/blob/main/scripts/entrypoint.sh) but that still didn't work. Any suggestions? What's the intended way to run prefect based on a custom docker image? I want to avoid altering the original image as it's pretty difficult to build already.

how do i get a list of all prefect deployments in python

hi all, I’ve upgraded to Prefect 2.0.0, now how can I define a block for Azure blob storage? I just see S3 and GCP blocks…

@Marvin how do i set the name of a flow_run?

Hi! I might be missing something silly, hope you can help. I have an image pushed to my Google Cloud Artifacts Registry (

europe-docker.pkg.dev/project/folder-staging/image_name:latest

), previously this image had an issue as it was missing a dependency (

pandas

lib) - I fixed this and the image was re-deployed via our CI, I've verified that the image now does have the dependency installed correctly. The CI was also triggered to re-deploy the flow by calling the following function:

def deploy_projects_observability_export_flow():
    projects_observability_export_flow.deploy(
        name="projects_observability_export_flow_k8s_deployment",
        work_pool_name="default-worker-pool",
        image="europe-docker.pkg.dev/***/prefect-runners-staging/default:latest",
        tags=["staging"],
        schedule=CronSchedule(cron="0 0 * * *", timezone="UTC"),
        job_variables={
            "cpu_request": "110m",
            "mem_request": "128Mi",
            "cpu_limit": "1200m",
            "mem_limit": "8Gi",
        },
    )

However, the flow runs that are set to leverage this image keep showing the

ModuleNotFound

error. Do you happen to know what I'm missing?

Hi, I'm just getting started. When I run prefect I get an error, but I'm not sure if it's a bug or just my mistake

# prefect
Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 5, in <module>
    from prefect.cli import app
  File "/usr/local/lib/python3.12/site-packages/prefect/cli/__init__.py", line 22, in <module>
    import prefect.cli.kubernetes
  File "/usr/local/lib/python3.12/site-packages/prefect/cli/kubernetes.py", line 36, in <module>
    get_prefect_image_name(),
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prefect/utilities/dockerutils.py", line 56, in get_prefect_image_name
    else "sha-" + prefect.__version_info__["full-revisionid"][:7]
                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^
TypeError: 'NoneType' object is not subscriptable

Hey folks. Is there a way to have Prefect treat a file on disk that was created during a task’s execution as a Result, that can be persisted using S3Result (for example) without loading that file into memory / returning it from the task function? I am trying to see if my organization can use Prefect, but our core need is to run 3rd party programs that produce large output files that may not fit in memory. Thanks and I apologize if this is already in the docs.

Hi, new to Prefect and Python so apologies if what I’m about to describe is a pathetically trivial/obvious problem. Here’s my environment: 1. Prefect Cloud for orchestration. 2. Azure Kubernetes Prefect Agent (note I don’t have an executor defined in Azure as I’m not sure how to do this yet, or whether I even need to, but it’s important that the entire Flow runs in Azure - ideally on the existing Agent if possible). Have imported KubernetesRun from

prefect.run_configs

in my Flow, and can see the Agent from Prefect Cloud, and was able to run ‘hello world’ successfully via it also. 3. Azure PostgreSQL database I need to run SQL against as part of a Prefect Task (have used a Secret in Prefect Cloud to create a dictionary with all the database login parameters) 4. Private API I need to call as part of a Prefect Task 5. GitHub storage for the whole Flow script (created a GitHub Access Token Secret to enable PrefectCloud to access the registered script) 6. Have run the

prefect backend cloud

CLI command on my local Mac, to hopefully force everything to execute in Azure rather than locally Here’s the journey I’ve been on: 1. I got the basic “hello world” flow executing on a local agent (yay!), with the Flow registered in the Prefect Cloud, and the code being pulled from GitHub 2. I then got it working against the Kubernetes agent, but for some reason the ‘hello world’ message didn’t appear in the logs. Is that because I need to set

prefect.config.cloud.send_flow_run_logs = True

somewhere/how, or something else? 3. Tried importing psycopg2 and creating a connection to the PostgreSQL database to retrieve a very simple count result. Am not sure if I need to be using the PostgresExecute API call (which itself uses Psycopg2) rather than importing psycopg2 into my Flow. Thoughts/recommendations welcome! 4. Also tried importing the requests Python module to call the API. 5. When I try running the script to register the Flow it appears to work (finished with exit code 0), but when I look in my Prefect Cloud Flows tab it’s not there. Any idea why, please? Sorry to bother you all about this, but am kind of at a loss on how to move forwards with this if I can’t even see the Flow I want to execute. Suspect it may be something to do with importing those libraries/modules and them not being available in the Execution environment, but have no idea from what I’ve read so far what I need to do to get that working. Any advice (even RTFM, if you can point me to the right topic) would be gratefully received. Thanks in advance! * UPDATE * Have posted the script to the below thread as requested by @Kevin Kho Huw