Hi Team, Is it possible to run an agent for two different workspaces for one container? Can we have multiple profiles inside a container? I don't think it is possible, because the Prefect CLI will be pointing to a specific workspace, but I thought I'd ask instead of just spinning up two ECS instances.

Can I use both .map and .submit on a task?

Hello again! is there a way to not display flow run parameters of type pydantic.SecretStr in the UI?

Hello Team, is there a reason parameter entered in UI doesn't work in the run? When I make a change to the parameter in the UI, the flow still picks up the default parameter instead of the newly entered one.

hey all, im trying to pass a task's result to a function that's not a task. I'm getting:

TypeError: not all arguments converted during string formatting

is there a way to get the parameter's value without giving function with a task decorator?

"attempted to send logs to Orion without a flow run id": We are running into an issue where our flow imports libraries that do some logging on import. As outlined here: https://linen.prefect.io/t/3323130/must-get-run-logger-be-called-in-each-task-flow-i-have-a-flo Is there a known work around? We tried to disable logging & re-enable logging post imports, but that didn't help.

Hey, congrats on releasing 2.6.5. Why is it, that for manual retries via UI we need to enable result persistence? Don't we simply rerun a flow - why do we need the results from the failed flow to be persisted? (Not that it's a problem, I'd just like to understand the underlying architecture)

Hello all, After my last problem was immediately and perfectly solved here in the channel (thank you, again, @Q !), I would like to ask if anyone could help me with a second question, concerning the same example project (https://prefect-community.slack.com/archives/CL09KU1K7/p1666555308227089)? Namely, now that the creation of the deployment on the SFTP remote storage block works, I'm unfortunately unable to start a flow run from it. Specifically, I started a local Prefect agent and afterwards a local flow run from the UI (Run -> Now with defaults). After that, the paramiko library gives the following error:

File "C:\Users\Testbenutzer\anaconda3\envs\prefect_test_project_v3\lib\site-packages\paramiko\sftp_client.py", line 810, in get
    with open(localpath, "wb") as fl:
PermissionError: [Errno 13] Permission denied: '.'
23:16:29.384 | INFO    | prefect.infrastructure.process - Process 'upbeat-crab' exited cleanly.

To solve the PermissionError, I tried two approaches, which unfortunately both failed: 1. Try to change the

PREFECT_LOCAL_STORAGE_PATH

2. Running Powershell, from which I start Prefect, as admin My problem seems to be practically the same as that reported by @Mike He in the discourse forum (https://discourse.prefect.io/t/agent-debugging-agent-execution-error-with-ftp-storage-in-windows/1587) and here in Slack (https://prefect-community.slack.com/archives/C03D12VV4NN/p1663319368100279). (The only difference I see is that in my case, the last error of the traceback comes from the paramiko library instead of fsspec.) I saw that he already developed a workaround, which however includes changing the Prefect source code which I have not tried yet. I would write details, e.g. the full traceback, in the thread. If anyone had an idea or a hint how this problem could be solved, preferably without manually changing the Prefect source code, you would help me again a lot. Thank you very much in advance!

Hi. Does anybody had any issues with Prefect 1.x ECS deployment ? Regarding the flow register not having the Fargate machine you requested ? I registered 16vCPU and have in the task definition only a 2vCPU machine. Spoke with AWS support but it was not helpful and our AWS Account does have a vCPU limit at 2000. So enough bandwidth.

Hi - I have my task and flow in seperate files, but would like to use this

def cache_within_flow_run(context, parameters):
    return f"{context.flow_run_id}-{task_input_hash(context, parameters)}"

How would i pass the flow_run_id into the task this function and further into the task decorator?

@task(cache_key_fn=cache_within_flow_run)
def cached_task():
    print('running an expensive operation')
    return 42

Hi, is it possible to add the ECR as a block in Prefect 2? I was wondering if this is doable, I'm was trying in the UI, but the credentials part got me thinking. I'm using IAM Roles to authenticate the flow runs, so I don't even have an ECR username and password which the Docker Registry block in the UI requires. Maybe someone knows a way to achieve that. Thanks in any case

i had a conceptual question about the deployment running infra. when we run same deployment multiple times (with same or different params) does each of them create new pods to run new job or are these pods created specific to flows ? and if i want my parallel processing(parallel running tasks or flows) to be distributed into multiple jobs so as to make sure the amount of resources each process is taking is not huge.. how can i achieve that ? is creating separate deployments a way to do it ?

We are on the latest prefect & prefect_aws, and seeing scheduled jobs terminate without being run (intermittently). The UI for the flow run shows no logs, no task runs, and no subflow runs. In cloudwatch logs for the flow run, I found the below. Any thoughts?

23:02:20.896 | INFO    | prefect.engine - Engine execution of flow run '682567a2-e507-4aa1-b5c7-25bb2cc2573f' aborted by orchestrator: This run has already terminated.

quo https://docs.prefect.io/concepts/deployments/#parameters-in-deployments it's possible to add default parameters in yaml and python deployments... Is it possible or in the works, to also add these via the cli deploy?

hey there, i'm trying to use a task's result as a directory name. is this possible?

Hello, I have a flow that I run on my laptop using DockerContainer infrastructure and takes around 4 minutes to complete. When I run that same flow on a kubernetes cluster using KubernetesJob it times out after 30 minutes (limit that I set in my flow). Any tips on how I could troubleshoot what is going on? I am using the default job template.

Hi, I'm running a task that creates a flow run from a deployment ID using create_flow_run_from_deployment. I used to run the following code in prefect 2.5.0:

await my_task.submit(_flow_name_=FLOW_NAME, _env_=env, _run_name_=run_name, _wait_for_=[preprocess], _parameters_=parameters)

However after upgrading to 2.6.5 my_task isn't even being run. My workaround is to just run this:

await my_task(_flow_name_=FLOW_NAME, _env_=env, _run_name_=run_name, _wait_for_=[preprocess], _parameters_=parameters)

But it seems really weird that submit should have stopped working. Has anyone else ran into this issue?

I'm running a Scheduled flow that calls another flow using the

startFlowRun

method. Initially when I was calling the first flow without using the

wait

parameter, the flows worked fine, but after I added the parameter, the parent flow failed with

TypeError: Object of type SUCCESS is not JSON serializable

. I was wondering if this is a known issue. Prefect version is 1.4.0

Hello. Prefect 1. Is there a way in a graphql query to sort nulls last like:

order by column_name nulls last

? I have this line

order_by: {start_time: desc}

in my query. More context in comment

This message was deleted.

Hi all, I’m not sure how i am supposed to leverage secret string from a customised block. I’m trying to run a shell where i pass a password.

hey folks, looking for some help here with setting up prefect 2 on K8S When running an agent in kubernetes, the agent does not receive logs from workers... among other things, this means I can not see if flow run succeeded/failed from agent logs Here is my setup: • all flows are stored in S3 • all flows run on the same infra block which is a KubernetesJob - new pod is spun up for each flow • using Prefect Cloud to orchestrate Everything was great when i was running prefect agent locally -- agent produces logs (output in CLI) similar to:

19:41:45.583 | INFO    | prefect.agent - Submitting flow run 'd03564b9-ad4e-4310-8662-184616d6406f'
19:41:47.416 | INFO    | prefect.agent - Completed submission of flow run 'd03564b9-ad4e-4310-8662-184616d6406f'
19:41:48.471 | INFO    | prefect.infrastructure.kubernetes-job - Job 'expert-lyrebird-xjq42': Pod has status 'Pending'.
19:41:49.155 | INFO    | prefect.infrastructure.kubernetes-job - Job 'expert-lyrebird-xjq42': Pod has status 'Running'.
19:41:52.098 | INFO    | Flow run 'expert-lyrebird' - Created task run 'My Example Task-c06c9343-0' for task 'My Example Task'
19:41:52.098 | INFO    | Flow run 'expert-lyrebird' - Executing 'My Example Task-c06c9343-0' immediately...
19:41:52.345 | INFO    | Task run 'My Example Task-c06c9343-0' - Finished in state Completed()
19:41:52.443 | INFO    | Flow run 'expert-lyrebird' - Finished in state Completed()

When running the agent locally but in docker container, the logs look similar. Now I want the agent to run in a kubernetes deployment to automatically manage its uptime. Only 1 replica of agent pod is used. However I only receive part of the log in the agent cli output -- it looks like no info is coming back from the worker pods back to the agent pod...

19:59:58.687 | INFO    | prefect.agent - Submitting flow run 'a510b13d-b0b5-4067-8610-518e6a2b45a8'
19:59:59.268 | INFO    | prefect.agent - Completed submission of flow run 'a510b13d-b0b5-4067-8610-518e6a2b45a8'
19:59:59.305 | INFO    | prefect.infrastructure.kubernetes-job - Job 'secret-iguana-zlm4f': Pod has status 'Pending'.
20:00:01.349 | INFO    | prefect.infrastructure.kubernetes-job - Job 'secret-iguana-zlm4f': Pod has status 'Running'.

What is going on ? how can I make the agent running on K8S produce a complete log ? let me know if you have any ideas -- I am fairly new to k8s so might be something basic, but looks unexpected to me

Hi When using prefect 2, I assume we still need the agent to run in the background all the time. do we still need to run something like

nohup prefect agent start -q …  > ~/tmp/prefect_agent.log &

or is there a different way to keep the agent running? is there best practice around this in the docs? I did not find it so far Thanks

I saw a new feature to restart flows from failed/crashed state getting released a while ago but I can't find it from UI or it's still not ready?

Hi, I’m trying to create a deployment with a minio object store as storage layer. However it seems that the flow code is not registered. I’m trying to debug this but I can’t seem to adjust the log level of Orion. I’ve tried setting PREFECT_LOGGING_SERVER_LEVEL to DEBUG but I still only see INFO logging from my server. I’m running my own server so no cloud here.

I just started getting API errors on all my scheduled flows on Prefect 1

All my flows are failing..

Anyone monitoring chat tonight? my error is

Failed to load and execute flow run: ClientError([{'path': ['secret_value'], 'message': 'Unable to complete operation', 'extensions': {'code': 'API_ERROR'}}])

we are hard down...

Hi, I am trying to replicate a local deployment of prefect (incl.

orion

+

agent

) on a remote server. However, I get an error message in the orion logs after changing

PREFECT_ORION_API_HOST

from

0.0.0.0

to

SERVER_IP

. The particular error is:

ERROR:    [Errno 99] error while attempting to bind on address ('SERVER_IP', 4200): cannot assign requested address

This does not make sense, since I can specifically access the same

SERVER_IP

through the browser and the particular port has been allowed through the firewall as well. (PS - orion appears to connect for a brief second, and then immediately throws this error and attempts to restart). Additionally, attached below is the relevant section of my `docker-compose.yml`:

version: "3.9"
services:
  db:
    image: timescale/timescaledb:latest-pg14
    volumes:
      - $PWD/data/db:/var/lib/postgresql/data
      - $PWD/config/db:/docker-entrypoint-initdb.d/
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready" ]
      interval: 10s
      timeout: 5s
      retries: 5
    environment:
      - POSTGRES_DB=$POSTGRES_DB
      - POSTGRES_USER=$POSTGRES_USER
      - POSTGRES_PASSWORD=$POSTGRES_PASSWORD
    networks:
      - db_network

  orion:
    image: prefecthq/prefect:2.6.3-python3.10
    restart: always
    volumes:
      - $PWD/prefect:/root/.prefect
    entrypoint: [ "prefect", "orion", "start" ]
    environment:
      - PREFECT_ORION_API_HOST=$SERVER_IP
      - PREFECT_ORION_DATABASE_CONNECTION_URL=$PREFECT_DB_URL
    ports:
      - 4200:4200
    depends_on:
      - db
    networks:
      - prefect_network
      - db_network

  agent:
    build: ./prefect
    restart: always
    entrypoint: [ "prefect", "agent", "start", "-q", "main_queue" ]
    environment:
      - PREFECT_API_URL=<http://orion:4200/api>
    networks:
      - prefect_network
      - db_network

  cli:
    build: ./prefect
    entrypoint: "bash"
    working_dir: "/root/flows"
    volumes:
      - "$PWD/prefect/flows:/root/flows"
    environment:
      - PREFECT_API_URL=<http://orion:4200/api>
    networks:
      - prefect_network
      - db_network

networks:
  db_network:
  prefect_network:

i'm a new user i want to save the flows to the database PostgreSQL, I added the connection and the prefect tables was added to my database, I have a record in table configuration , But the flows and the logs not inserted to my PostgreSQL database, i don’t know what i’m missing, can someone help me please?