Hey Prefect community, I'm working on deploying an instance of an Orion server to ECS using AWS CDK. One issue I've run into is how the database url is being specified via env vars. In ECS, there is a distinction between environment_vars and secrets and, to my knowledge, they cannot be concatenated. This presents a dilemma as you know must: • Have the database password in the clear as an env var. ◦ This also opens up an issue, where if the password is updated, it is not reflected until the next deployment. • resolve the secret key, concat with the whole URI, put into SSM

orionServer.addContainer('ServiceContainer', {
            essential: true,
            image: ContainerImage.fromDockerImageAsset(serviceImage),
            logging: new AwsLogDriver({
                streamPrefix: 'orion-server'
            }),
            portMappings: [{
                containerPort: this.props.orionEnvVars?.port || Number(this.CONTAINER_PORT)
            }],
            environment: {
                PREFECT_ORION_API_HOST: this.props.orionEnvVars?.host || '0.0.0.0',
                PREFECT_ORION_API_PORT: this.props.orionEnvVars?.port?.toString() || this.CONTAINER_PORT,
                PREFECT_ORION_DATABASE_CONNECTION_URL: `postgres+asyncpg:///${this.props.username}:${ecsSecret.fromSecretsManager(this.props.dbPassword)}@${this.props.databaseHost}/orion`

            },
            secrets: {

            },
            memoryLimitMiB: 300,
        });

We can overcome this by altering the settings provided, and fetching the username and password separately from the host to build the DB URI, perhaps as a second set of options. Has anyone found an alternate way to concat the strings together? If not, are we open to adding additional options to distinguish DB_PASSWORD to be injected as secrets?

Hey all, 👋 We face several problems with flows that run in the Kubernetes cluster. • Pods often stuck in the

Running

state with the last message in logs

DEBUG - prefect.CloudFlowRunner | Checking flow run state...

• The flow in Prefect Cloud stucks in the

Cancelling

state and the pod stucks in the

Running

state in the Kubernetes cluster. Context: • prefect version

0.15.13

• Private Azure AKS cluster • We've tried to set

PREFECT__CLOUD__HEARTBEAT_MODE

to

"thread"

, but it only got worse (more stucked pods in the

Running

state). Now we have

PREFECT__CLOUD__HEARTBEAT_MODE

with

"process"

value and

tini -- prefect execute flow-run

as PID 1 to handle zombie process. It seems like the problem with the heartbeat process detecting the change to

Cancelling

or

Cancelled

states of the flow. I appreciate any help, thanks 🙂

Hi! Is it possible in Prefect to create a sub-flow that works as follows: • task A runs and may fail. Retry A until success. • task B runs and may fail (or the task itself succeeds, but it returns a result indicating failure of the sub-flow). In that case, loop back to retry A. This seems like it could be possible to do using looping on a nested flow as described in this stackoverflow post.

hi, is there a quick way to check the health and status of the lazarus process/service? perhaps a REST endpoint we could query or a command line utility we can invoke?

@Marvin I have a self hosted prefect, how do I add security to the API and UI?

@Marvin what is ephermeral mode

@Marvin how can I get a list of all flows and their IDs using the python sdk library for prefect 3.X ?

Is there an example prefect.yaml for a modal work pool deployment?

prefect server start --host 0.0.0.0

start the server. but it went sleep after few hours without any log , error. any solution? can @Marvin help with it.

Hello I have a strange error when I run a deployment. To add context, I deploy a worker on Kubernetes with Helm (with this: https://docs.prefect.io/2.10.13/guides/deployment/helm-worker/ ). Then I deploy a flow with build_docker_run (with

namespace: prefect

specified in my deployment.yaml (I use projects concept). This is my error at launch :

kubernetes.client.exceptions.ApiException: (422)
Reason: Unprocessable Entity
HTTP response headers: HTTPHeaderDict({'Audit-Id': '90369801-cb01-411b-b3a8-43018a11c756', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Warning': '299 - "unknown field \\"spec.template.spec.completions\\"", 299 - "unknown field \\"spec.template.spec.parallelism\\""', 'X-Kubernetes-Pf-Flowschema-Uid': 'ff4e4556-b309-4791-8d69-60463f550a9d', 'X-Kubernetes-Pf-Prioritylevel-Uid': '972bc841-dafb-496e-89cd-a53f07f46a9f', 'Date': 'Thu, 15 Jun 2023 15:22:37 GMT', 'Transfer-Encoding': 'chunked'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Job.batch \"-fqr7x\" is invalid: [metadata.generateName: Invalid value: \"-\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. '<http://example.com|example.com>', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), metadata.name: Invalid value: \"-fqr7x\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. '<http://example.com|example.com>', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), spec.template.labels: Invalid value: \"-fqr7x\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')]","reason":"Invalid","details":{"name":"-fqr7x","group":"batch","kind":"Job","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \"-\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. '<http://example.com|example.com>', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')","field":"metadata.generateName"},{"reason":"FieldValueInvalid","message":"Invalid value: \"-fqr7x\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. '<http://example.com|example.com>', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')","field":"metadata.name"},{"reason":"FieldValueInvalid","message":"Invalid value: \"-fqr7x\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')","field":"spec.template.labels"}]},"code":422}

I try to deploy worker and flow on default namespace but with same results. I try to upgrade my GCP clusters to be with the last stable k8s version but nothing change. Do you have any idea?