@Marvin I'm setting up a push worker pool using azure container instances. I'm following the tutorial and I created the role and identity, but when I configure the ACI work pool on prefect, it asks me for a credential block with a password. Do I need to create a new principal service id with a password ?

@Marvin how do i call an async flow as a fastapi background task?

@Marvin how do I add a timezone to a cron expression when deploying a flow with the flow.serve method?

@Marvin describe the prefect architecture

Hi @Marvin I am getting the following error while running prefect flows.

aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host <http://hooks.slack.com:443|hooks.slack.com:443> ssl:default [Temporary failure in name resolution]

@Marvin I am using this yaml to do prefect

deploy --name my_pipeline

. How can I do the same with a python script and not via the CLI?

# File for configuring project / deployment build, push and pull steps

# Generic metadata about this project
name: my-pipeline
prefect-version: 2.12.1

# build section allows you to manage and build docker images
build:
  - prefect_docker.projects.steps.build_docker_image:
      image_name: <http://mycompany.azurecr.io/dir/prefect-image|mycompany.azurecr.io/dir/prefect-image>
      tag: latest
      dockerfile: Dockerfile
      push: true
      credentials: "{{ prefect.blocks.docker-registry-credentials.my-cred }}"


# push section allows you to manage if and how this project is uploaded to remote locations
push: null

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
  - prefect.projects.steps.git_clone_project:
      repository: <https://bitbucket.org/mycompany/repo.git>
      branch: master
      access_token: "{{ prefect.blocks.secret.bb-credentials }}"
  - prefect.projects.steps.set_working_directory:
      directory: /opt/prefect/repo

deployments:
  - name: my_pipeline
    description: TODO
    entrypoint: my_pipeline/my_pipeline.py:my_flow
    parameters: {}
    work_pool:
      job_variables:
        image: "{{ image_name }}"
      name: worker_pool
      work_queue_name: default

@Marvin How do I set up an s3 storage block for a self hosted prefect server using MinIO?

Something I am struggling to get my head around is security in regards to

FargateCluster

. It seems like the cluster is being assigned a public IP address. Ideally I wouldn’t want that to be the case as I don’t want people snooping on my cluster / submitting jobs. However, when I pass

"fargate_private_ip": True

to

cluster_kwargs

my cluster fails to start with the error:

Cluster failed to start: Timed out trying to connect to <tcp://10.0.1.111:8786> after 30 s

That makes sense. Someone somewhere failed to connect to a local IP address, presumably from outside the subnet. What I don’t understand is how I can prevent people from arbitrarily accessing my cluster from the internet whilst allowing all the ‘right’ traffic through

What’s the best practice for Data Retention Policy on Prefect deployment runs? Just as a reference, here is how it is implemented for Apache Airflow, as yet another garbage collector DAG: https://stackoverflow.com/questions/66580751/configure-logging-retention-policy-for-apache-airflow I’m sure that Prefect has either a built-in mechanism for that, or encourages a common idiom for rotating / archiving / deleting artifacts from old runs. Context: We have a persistent storage on Azure Blob Storage (the S3 equivalent) where we store artifacts (e.g. output files and images) from a Machine Learning (Kedro) run. The space can pile up pretty quickly across runs and we would run out of storage, rendering our Prefect deployments not operational. What kind of policies are recommended to evict data from old runs? I don’t want to run out of space and I want the Prefect pipelines to remain operational. I know that some of you would say: “_It depends_”, so for the sake of this example let’s imagine that I have a dedicated 256GB of storage. Should I set a threshold (e.g. 70% of full) that will be as a trigger for evicting (removing) artifacts from old runs? Also, when should this run? as the first (prerequisite) subflow in my bigger flow, or as yet another deployment in Prefect on a recurring schedule? Thanks!

Hey all! I've been playing around with setting up a Kubernetes agent in Google Kubernetes Engine which can spin up a ephemeral Dask cluster on demand. This all seems to work rather smoothly (thanks to the amazing work done by the Prefect team and others), but I'm having some issue getting the autoscaler to remove the nodes after the flow has run. I get the following error messages on my Kubernetes cluster after my flow has run: "Pod is blocking scale down because it’s not backed by a controller" "Pod is blocking scale down because it doesn’t have enough Pod Disruption Budget (PDB)" I'm pretty inexperienced with Kubernetes so I was wondering if anyone has any pointers to how I might configure the KubeCluster so that it works with autoscaling? We're thinking of using the cluster to hyperparameter tune a model. We do not use Kubernetes for anything else and have no need for the resources in between training runs so getting the node pool to autoscale down to zero (the agent will stay in a different node pool) would save us some money. My run code is below: