@Marvin how can i debug the prefect code in my local machine

@Marvin I am using this yaml to do prefect

deploy --name my_pipeline

. How can I do the same with a python script and not via the CLI?

# File for configuring project / deployment build, push and pull steps

# Generic metadata about this project
name: my-pipeline
prefect-version: 2.12.1

# build section allows you to manage and build docker images
build:
  - prefect_docker.projects.steps.build_docker_image:
      image_name: <http://mycompany.azurecr.io/dir/prefect-image|mycompany.azurecr.io/dir/prefect-image>
      tag: latest
      dockerfile: Dockerfile
      push: true
      credentials: "{{ prefect.blocks.docker-registry-credentials.my-cred }}"


# push section allows you to manage if and how this project is uploaded to remote locations
push: null

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
  - prefect.projects.steps.git_clone_project:
      repository: <https://bitbucket.org/mycompany/repo.git>
      branch: master
      access_token: "{{ prefect.blocks.secret.bb-credentials }}"
  - prefect.projects.steps.set_working_directory:
      directory: /opt/prefect/repo

deployments:
  - name: my_pipeline
    description: TODO
    entrypoint: my_pipeline/my_pipeline.py:my_flow
    parameters: {}
    work_pool:
      job_variables:
        image: "{{ image_name }}"
      name: worker_pool
      work_queue_name: default

Has anyone encountered this error in the prefect-agent run output? I get usually 2 lines of expected log and then this error message. The agent runs, however the agent log is super polluted and hard to read with all the error messages.

Once I deploy a prefect-worker via helm chart, using value.yaml file, I have only one work-pool mapped to the newly generated worker. Is there any way to assign multiple workpools to a single worker? Here is the values.yaml file I have been using, but this one only activates gke-cpu-5000m-4Gi workpool.

namespaceOverride: prefect
worker:
  cloudApiConfig:
    accountId: HIDE
    workspaceId: HIDE
  config:
    workPool: gke
    workPool: gke-cpu-5000m-4Gi
serviceAccount:
  create: false
  name: "HIDE"

I am running prefect 2.0 on aws lambda (runtime python3.9) but i get an error

"Orion requires sqlite >= 3.24.0 but we found version 3.7.17"

Does anyone know a workaround? i

Hi all! Shouldn't the

config.toml

file be in the path where prefect is installed? There is only

backend.toml

file in the path where my prefect is installed, are those two the same? Also i can't read

backend.toml

. I got the error message like below:

TomlDecodeError: Found invalid character in key name: ':'. Try quoting the key name. (line 1 column 2 char 1)

What's the backend.toml file?

@Marvin what's the best way to run child flows in parallel?

@Marvin How do i set up Prefect with Postgresql without using Docker?

Hi guys, did anyone manage to mount volume (kubernetes) to every prefect deployment that is created by specific workpool? how can I mount to specific prefect deployment a volume, and to use the content of it for environment variable

Something I am struggling to get my head around is security in regards to

FargateCluster

. It seems like the cluster is being assigned a public IP address. Ideally I wouldn’t want that to be the case as I don’t want people snooping on my cluster / submitting jobs. However, when I pass

"fargate_private_ip": True

to

cluster_kwargs

my cluster fails to start with the error:

Cluster failed to start: Timed out trying to connect to <tcp://10.0.1.111:8786> after 30 s

That makes sense. Someone somewhere failed to connect to a local IP address, presumably from outside the subnet. What I don’t understand is how I can prevent people from arbitrarily accessing my cluster from the internet whilst allowing all the ‘right’ traffic through