Hi there. I recently started getting some errors on VS Code using pylance. For example, the following line

from prefect import Flow, Parameter, task, unmapped

produces

"Flow" is not exported from module "prefect"
Import from "prefect.core" instead Pylance(reportPrivateImportUsage)

"Parameter" is not exported from module "prefect"
Import from "prefect.core" instead Pylance(reportPrivateImportUsage)

"task" is not exported from module "prefect"
Import from "prefect.utilities.tasks" instead Pylance(reportPrivateImportUsage)

is not exported from module "prefect"
Import from "prefect.utilities.edges" instead Pylance(reportPrivateImportUsage)

Also, I am getting pylance errors when passing Parameters to tasks that expect

str

arguments. I don't believe this was happening before. Do you know if something changed recently in pylance or in how we are supposed to import modules? I am using prefect 0.15.4 Thanks!

@Marvin how can I use a progress bar with prefect 3.X to check up on the progress of multiple tasks being run in parallel?

Hi Prefect Community! We are running our infrastructure on GKE Autopilot and have been seeing an increase in the number of

Crashed

jobs recently. I am trying to do root cause analysis on it so I start by digging through the logs in Prefect Cloud. What I see is that at some point (it is random) the logs stop and nothing further is output to the UI. Digging through the logs in Google Logs Explorer I see the same behavior, Prefect container logs stop at the same specific point in time. Inside Google Logs I am also able to see a lot of Kubernetes related logs and am starting to see a pattern but not clear how to fix it. • Roughly 5-10 seconds after the last log this shows up: ◦

INFO 2023-02-03T19:18:11Z [resource.labels.nodeName: gk3-prefect-autopilot-cl-nap-ji2s72nv-db29cac6-hxzc] marked the node as toBeDeleted/unschedulable

• Quickly followed by: ◦

INFO 2023-02-03T19:18:11Z [resource.labels.clusterName: prefect-autopilot-cluster-1] Scale-down: removing node gk3-prefect-autopilot-cl-nap-ji2s72nv-db29cac6-hxzc, utilization: {0.5538631957906397 0.1841863664058054 0 cpu 0.5538631957906397}, pods to reschedule: adorable-axolotl-d8k8c-6dx5c

◦

INFO 2023-02-03T19:18:38Z [resource.labels.clusterName: prefect-autopilot-cluster-1] Scale-down: node gk3-prefect-autopilot-cl-nap-ji2s72nv-db29cac6-hxzc removed with drain

• GKE tries to reschedule the job but it fails with the following, which is when Prefect alerts for the

Crashed

state: ◦

INFO 2023-02-03T19:18:11Z [resource.labels.podName: adorable-axolotl-d8k8c-6dx5c] deleting pod for node scale down

◦

ERROR 2023-02-03T19:18:19.215934101Z [resource.labels.containerName: prefect-job] 19:18:19.214 | INFO | prefect.engine - Engine execution of flow run '8ca83100-dcc3-46d5-91be-f342b19b45a9' aborted by orchestrator: This run cannot transition to the RUNNING state from the RUNNING state.

This appears to be happening on jobs randomly and leads me to believe that GKE believes the cluster is overprovisioned so it is trying to reduce the cluster size and move jobs around, but jobs can't be moved in the middle of execution and Crash/Fail. I am also curious if this is due to resource sizing, but I am not seeing any issues with the jobs I have been troubleshooting with

insufficient resource

problems. They all typically state the following in the

containerStatuses

leaf of the JSON element with the following message:

state: {
terminated: {
containerID: "<containerd://aac705>"
exitCode: 143
finishedAt: "2023-02-03T19:18:19Z"
reason: "Error"
startedAt: "2023-02-03T19:16:52Z"
}}

Any incite would be greatly appreciated!

Hi! I want to debug a flow-of-flows on my local machine, but the flow runs created using create_flow_run seem to be executing on my production Prefect Cloud environment. How do I ensure that these flow runs are all executed on my local machine?

Hi all, I’m having trouble diagnosing a GitHub storage problem. I’ve created a trivial testing flow similar to the example script-based workflow for GitHub. I’ve deployed a k8s agent using

prefect k8s agent install

. I’ve uploaded my flow to a private GitHub repo and registered it with Prefect. And I’ve added a Cloud Secret called

GITHUB_ACCESS_TOKEN

that holds a valid GitHub personal access token. When I run my flow the agent’s GitHub storage gives me an

UnknownObjectException(404, 'Not Found')

error. If I change the flow to use a different Cloud Secret key for the PAT, such as

access_token_secret='MYKEY'

, then the agent tells me

ValueError('Local Secret "MYKEY" was not found.')

. How can I introspect the kubernetes agent to verify that the GitHub PAT secret is being loaded from Prefect Cloud correctly?

Does prefect have a drag-and-drop-based UI,? I want to use it for network operators. They can’t code.

hi @Marvin. how to tag a flow? is it something that should be done in the code?

Hi all, Quick question about flow.from_source(). Can I use this from within the same repo? As in, can I do something like this from within the same repo defined by the GitRepo url? @flow async def my_flow(name: str = "world"): print(f"Hello {name}!") if name == "__main__": DEFAULT_GH_SECRET_BLOCK_KEY = "github-secret" access_token = Secret.load(DEFAULT_GH_SECRET_BLOCK_KEY).value flow.from_source( source=GitRepository( url="https://github.com/company/my-repo.git", branch="main", credentials={"access_token": access_token}, ), entrypoint="/builtin_worklflows/paige-test.py:my_flow", ).deploy( name="my-deployment", work_pool_name="my-k8s-job", job_variables=dict(pull_policy="Never"), schedule=(CronSchedule(cron="0 0 * * *", timezone="America/Chicago")), tags=["test1","test2"], paramaters=dict(name='Paige') )

Error grabbing logs: unexpected EOF

<- Has anyone seen this in their (docker) agent output. It happened after two frames of stacktrace for me, on a mapped task (149 of 876), and the agent stopped processing anything (or sending anything to stdout anyway, and the rest of the tasks didn't run). Grepping for "grabbing logs" turns up nothing in the prefect repo, our own code, or any of the dependencies in my virtualenv. It's a cloud-backed flow run; is this coming from cloud, maybe?

@Marvin I want your help to fix and properly enhance my stack, here's my compose file:

name: prefect-wizard

services:
  nginx:
    image: nginx:latest
    ports:
      - "80:80" # Expose NGINX on port 80
    volumes:
      - ./proxies/nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - server
      - api
      - keycloak
      - oauth2-proxy

  # VNC browser for local development to use for keycloak and oauth2-proxy
  browser:
    image: jlesage/firefox
    ports:
      - "5800:5800"
    environment:
      # Optional: Set a password for the VNC web interface
      - VNC_PASSWORD=my_password
    depends_on:
      - keycloak
      - oauth2-proxy
      - nginx

  keycloak:
    image: <http://quay.io/keycloak/keycloak:24.0|quay.io/keycloak/keycloak:24.0>
    command: [ "start-dev", "--http-port=8080" ]
    environment:
      KC_BOOTSTRAP_ADMIN_USERNAME: admin
      KC_BOOTSTRAP_ADMIN_PASSWORD: admin
      #KC_HOSTNAME: "keycloak"          # no path, just host (and optional port)
      KC_HOSTNAME_STRICT: "false" # dev convenience
      KC_HOSTNAME_STRICT_HTTPS: "false" # dev convenience
      KC_PROXY: "edge" # recommended when behind a proxy
    ports:
      - "8080:8080"
    volumes:
      - keycloak_data:/opt/keycloak/data
      - ./keycloak/local/config.json:/opt/keycloak/conf/realm-config.json

  oauth2-proxy:
    image: <http://quay.io/oauth2-proxy/oauth2-proxy:v7.5.1|quay.io/oauth2-proxy/oauth2-proxy:v7.5.1>
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - "4180:4180"
    command: [ "--config=/etc/oauth2-proxy.cfg" ]
    volumes:
      - "./proxies/oauth2-proxy.cfg:/etc/oauth2-proxy.cfg"
    depends_on:
      - keycloak


  api:
    build:
      context: .
      dockerfile: docker/api/nonprod/Dockerfile
    ports:
      - "9832:9832"
    environment:
      PREFECT_API_URL: <http://server:4200/api>
    volumes:
      - ./api:/app
      - ./autoupdate:/app/autoupdate
      - ./core:/app/core
    extra_hosts:
      - "host.docker.internal:host-gateway"
    env_file:
      - .env
    restart: always
    healthcheck:
      test: [ "CMD-SHELL", "curl -f <http://localhost:9832/health> || exit 1" ]
      interval: 5s
      timeout: 5s
      retries: 5

  db:
    ...

  redis:
    ...

  server:
    image: prefecthq/prefect:3.4.13-python3.11
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@db:5432/prefect>
      PREFECT_SERVER_API_HOST: 0.0.0.0
      PREFECT_UI_API_URL: "<http://localhost:4200/api>"
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: 6379
      PREFECT_REDIS_MESSAGING_DB: 0
      PREFECT_LOGGING_LEVEL: DEBUG
      PREFECT_LOGGING_LOG_PRINTS: True
      PREFECT_SERVER_LOGGING_LEVEL: DEBUG
      PREFECT_SERVER_LOGGING_LOG_PRINTS: True
      #PREFECT_SERVER_API_AUTH_STRING: "admin:admin"
    command: prefect server start --no-services
    ports:
      - "4200:4200"
    healthcheck:
      test: [ "CMD-SHELL", "python -c 'import json,urllib.request,sys; sys.exit(0 if json.loads(urllib.request.urlopen(\"<http://localhost:4200/api/health>\", timeout=1).read()) is True else 1)' " ]
      interval: 5s
      timeout: 3s
      retries: 30

  services:
    ...

  create-deployment:
    ...

  autoupdate-worker:
    image: prefect-workpool
    depends_on:
      server:
        condition: service_healthy
      services:
        condition: service_started
      create-deployment:
        condition: service_completed_successfully
    environment:
      PREFECT_API_URL: <http://server:4200/api>
      DOCKER_HOST: <tcp://host.docker.internal:2375>
      PREFECT_LOCAL_STORAGE_PATH: /logs/storage
      # DOCKER_TLS_VERIFY: ${DOCKER_TLS_VERIFY:-}
      # DOCKER_CERT_PATH: ${DOCKER_CERT_PATH:-}
    restart: always
    volumes:
      - ./autoupdate:/opt/prefect/autoupdate
      - ./core:/opt/prefect/autoupdate/core
      - ${HOME}/.wizard:/logs
    extra_hosts:
      - "host.docker.internal:host-gateway"
    env_file:
      - .env

volumes:
  postgres_data:
  redis_data:
  keycloak_data:

I will send you keycloak config, oauth-proxy configs, nginx configs on separate messages as i am not allowed to post a message that is too long The error i am gettting: When I try to visit http://nginx on my virtual browser, it routes me to http://keycloak:8080/realms/wizard/protocol/.... with error

invalid parameter: redirect uri