Hello there I am seeing my flow logs showing up intermixed w Prefect Community #ask-community

Hello there, I am seeing my flow logs showing up i...

Billy McMonagle

02/23/2023, 9:59 PM

Hello there, I am seeing my flow logs showing up intermixed with my agent logs, and am wondering why. I'm running Prefect 2.8.0 (cloud), running on EKS with

KubernetesJob

infrastructure.

✅ 1

Billy McMonagle

02/23/2023, 10:00 PM

For example, here's what I see when tailing stdout on the agent pod:

Copy code

│ 21:49:34.025 | INFO    | Flow run 'authentic-termite' - Created task run 'dbt_build-0' for task 'dbt_build'                                                                                                                                       
│ 21:49:34.026 | INFO    | Flow run 'authentic-termite' - Executing 'dbt_build-0' immediately...                                                                                                                                                    
│ Wrote profile to /home/grebe/.dbt/profiles.yml                                                                                                                                                                                                    
│ Running dbt command: dbt --log-format json build --select XXX --exclude XXX --profiles-dir /home/grebe/.dbt --project-dir /home/grebe/src/dbt

Snipped but it shows the rest of the logs from that flow run.

Billy McMonagle

02/23/2023, 10:02 PM

The flows are definitely running in separate jobs, I am able to inspect and tail those pods separately.

Ryan Peden

02/23/2023, 10:03 PM

KubernetesJob

has output streaming enabled by default

Ryan Peden

02/23/2023, 10:03 PM

Unless you've turned it off, this is what I'd expect to see

Ryan Peden

02/23/2023, 10:04 PM

In the UI, you'll see a 'Stream Output' option you can edit, and if you create your blocks in Python you can set the

stream_output

attribute to

False

👍 1

Billy McMonagle

02/23/2023, 10:04 PM

No kidding, OK! We just deployed this as our first production flows on prefect 2, and I never noticed this behavior during development.

Billy McMonagle

02/23/2023, 10:05 PM

I'll try that setting, thanks for the quick tip.

👍 1

Billy McMonagle

02/23/2023, 10:07 PM

Docs say...

If set, stream output from the container to local standard output.

I still want to see the output in the job itself, just not the agent... is this still what I want to change?

Ryan Peden

02/23/2023, 10:09 PM

Yes, it should still show up in the job itself. By 'local standard output' I believe it means the local output of the agent running the flow. We could probably update the docs to make that more clear, so we appreciate the feedback.

Billy McMonagle

02/23/2023, 10:10 PM

Sweet, that seems very logical but ambiguous at the same time. Appreciate your help!

Open in Slack

Previous Next