Hey folks, I'm going through the "getting started" with prefect cloud and I can't seem to be able t...

Federico Zambelli

04/10/2023, 4:28 PM

Hey folks, I'm going through the "getting started" with prefect cloud and I can't seem to be able to see any logs on the Cloud UI. It does show my flow runs, but logs are empty, no matter what. I used

log_prints=True

, as well as

get_run_logger

, and neither case worked. What am I missing? for the record, this is my .py module:

Copy code

from prefect import flow, get_run_logger


@flow(log_prints=True)
def hi():
    logger = get_run_logger()
    logger.warning("Hit from Prefect! 😅")


if __name__ == "__main__":
    hi()

✅ 1

Will Raphaelson

04/10/2023, 4:32 PM

Hey Federico, just tried that code and I can see the logs. Mind showing me the output of the CLI command

prefect version

Federico Zambelli

04/10/2023, 4:32 PM

Hey will, thanks for the quick reply. Here's the output:

Federico Zambelli

04/10/2023, 4:33 PM

Could this possibly be due to the fact that months ago I was using

prefect

on the same system but with a local orion server?

Will Raphaelson

04/10/2023, 4:34 PM

it shouldn’t. just to confirm, you see the flow run in the ui, just not the logs? And the name of the flow run that you see in the terminal is the same as the one you see in the ui?

Will Raphaelson

04/10/2023, 4:34 PM

also I misspelled your name before, sorry! edited.

Federico Zambelli

04/10/2023, 4:36 PM

Yup I do see it in the UI, but no logs, it says "This run didn't generate any logs". The name is exactly the same, see: (no worries about name misspell, happens all the time 😄 )

Will Raphaelson

04/10/2023, 4:37 PM

okay i have one more thing if you dont mind. We’ve seen this elusive bug when only sending one log. Can you duplicate the log line and run again?

Federico Zambelli

04/10/2023, 4:37 PM

Yup, gimme 1 second

Nate

04/10/2023, 4:37 PM

hmm I assume this is related to using log_prints=True as well as the logger (instead of print) since log_prints literally patches the builtin print - do you see the same if you use

log_prints=True

and

print

instead of

logger.warning

Will Raphaelson

04/10/2023, 4:38 PM

yeah good line of questioning nate. This was mine as well but im able to run that code and have the logs appear anyway.

🧐 1

Nate

04/10/2023, 4:40 PM

interesting, I'm seeing the same (logs appear both in terminal and ui logs for me)

Federico Zambelli

04/10/2023, 4:40 PM

ok so I tried all possibilities: • logger with

log_prints

• logger without

log_prints

• print with

log_prints

All of them logging two separate messages. None of them showing up in the UI

Nate

04/10/2023, 4:42 PM

do you get the same result if you check your LOGGING_TO_API_ENABLED setting?

Copy code

In [8]: import prefect

In [9]: prefect.settings.PREFECT_LOGGING_TO_API_ENABLED.value()
Out[9]: True

Federico Zambelli

04/10/2023, 4:44 PM

lemme check

Federico Zambelli

04/10/2023, 4:45 PM

it does return

True

yes

🤔 1

Federico Zambelli

04/10/2023, 4:46 PM

Lemme try something: how do I switch from a cloud backend to a local orion server? I wanna see if the logs get printed there

Will Raphaelson

04/10/2023, 4:48 PM

Copy code

prefect config set PREFECT_API_URL=<http://127.0.0.1:4200/api>

Will Raphaelson

04/10/2023, 4:49 PM

(youll want to run

prefect server start

as well, before switching the config value. You can also create a profile for quick switching. https://docs.prefect.io/latest/concepts/settings/#configuration-profiles

upvote 2

Federico Zambelli

04/10/2023, 4:50 PM

ok let's see

Federico Zambelli

04/10/2023, 4:51 PM

local UI does show the logs

Federico Zambelli

04/10/2023, 4:51 PM

wtf

Federico Zambelli

04/10/2023, 4:51 PM

Nate

04/10/2023, 4:54 PM

have you tried sending an INFO level log to cloud instead?

Federico Zambelli

04/10/2023, 4:54 PM

yeah I had actually tired with

INFO

first, then as I saw no logs happening I switched to

WARN

to try

👍 1

Will Raphaelson

04/10/2023, 4:57 PM

i feel determined to catch this ghost. The only difference between our configs is the os/arch. I can ask a colleague with a linux machine to try this just to rule that out as a thing.

Federico Zambelli

04/10/2023, 4:57 PM

for the record, I'm running this on WSL2

Will Raphaelson

04/10/2023, 5:08 PM

Thanks, my linuxy colleage also couldnt reproduce. Might have to file the quickstart issue for deeper investigation later. In the interim, are you able to just use the get_run_logger path to show logs in the cloud ui?

Federico Zambelli

04/10/2023, 5:09 PM

wdym the get_run_logger path? If you mean the function to get the logger object, that didn't work either

Will Raphaelson

04/10/2023, 5:10 PM

oh sorry i see you tried that before

Will Raphaelson

04/10/2023, 5:15 PM

(thinking through some options internally, ill circle back on this)

Federico Zambelli

04/10/2023, 5:16 PM

yup no worries, thanks for the help. Do you want me to test directly on windows, without WSL2 ?

Will Raphaelson

04/10/2023, 5:16 PM

that would be excellent and help narrow it down

Federico Zambelli

04/10/2023, 5:16 PM

ok gimme a few minutes or so

🙏 1

Federico Zambelli

04/10/2023, 5:28 PM

@Will Raphaelson ok this is interesting. It seems that the issue is not with Linux, but with something related to my machine. I tried running the same flow under windows and it didn't log anything to the cloud UI as well

Federico Zambelli

04/10/2023, 5:29 PM

and just as an FYI, I tried deleting the local

.prefect

folder in home in Windows, and let prefect recreate it, but nonetheless I see no logs. 🤔 🤔 🤔 🤔

Will Raphaelson

04/10/2023, 5:39 PM

okay well thats progress at least.

Will Raphaelson

04/10/2023, 5:41 PM

can you show me the contents of that .prefect folder please?

Federico Zambelli

04/10/2023, 5:41 PM

linux or windows one?

Will Raphaelson

04/10/2023, 5:42 PM

i guess lets start with windows.

Federico Zambelli

04/10/2023, 5:42 PM

sure

Federico Zambelli

04/10/2023, 5:45 PM

ah since I deleted it right now there's only

profiles.toml

Federico Zambelli

04/10/2023, 5:46 PM

Federico Zambelli

04/10/2023, 5:46 PM

Will Raphaelson

04/10/2023, 5:47 PM

and after removing and recreating, you ran the script again and its still not showing logs right? We may have to open a github issue on this, reaching the end of my investigative abilities

Federico Zambelli

04/10/2023, 5:47 PM

yeah, I pip installed prefect in a virtual environment, pasted the script, ran

prefect cloud login

, authenticated via browser, ran

python my_flow.py

, that's it

Will Raphaelson

04/10/2023, 5:56 PM

got it, thanks. I think unfortunately ill need to file an issue and escalate to our team. My only thought is that something on the machine is blocking the calls out to the logs endpoint, but i dont know why that would happen.

Federico Zambelli

04/10/2023, 5:58 PM

yeah no worries Will, thanks for the help. Let me know if you or your team needs to get in touch since this is quite a peculiar situation

🙏 1

Will Raphaelson

04/10/2023, 5:58 PM

thanks will do, and I’ll post the github issue here when i file it in a few.

🙌 1

Will Raphaelson

04/10/2023, 7:26 PM

Writing it up now, can you give me a few more details about the infra here? Is this an azure VM, a laptop?

Will Raphaelson

04/10/2023, 7:33 PM

feel free to comment directly on https://github.com/PrefectHQ/prefect/issues/9172

Ryan Peden

04/10/2023, 7:53 PM

You can try running

prefect config set PREFECT_DEBUG_MODE=True

to get a deeper look at what's going on. Specifically, you should see messages from

prefect.logging.handlers.APILogWorker

. First, you should see one like

enqueueing item {'name': 'prefect.flow_runs', 'level': 30, 'message': 'Hit from Prefect! 😅' ...timestamp and ID fields }

And then another one a bit later that looks the same except instead of

enqueueing item

it will say

added item

. It seems like the flow itself is able to make API calls to create the flow and report its completion, but

APILogWorker

isn't sending anything to the API. It would be interesting to see if the log worker reports any problems. Even if it doesn't report any problem, that would be a useful clue as well! 🔎

Federico Zambelli

04/10/2023, 11:07 PM

@Will Raphaelson @Ryan Peden done, sorry for the delay, I was out for dinner. Also added the debug logs, thanks for the headsup

Ryan Peden

04/10/2023, 11:14 PM

Thanks! If you have a chance, could you check and see if any errors show up in the Chrome dev tools console when you try to view the flow run logs in the cloud UI?

👍 1

Will Raphaelson

04/10/2023, 11:21 PM

(and or give us a url for the flow run and we can check the cloud db to see if these logs even got to us)

Federico Zambelli

04/11/2023, 7:39 AM

Ok so, it seems the problem fixed itself 🤔 . I tried again this morning and now it's working. Both on Windows and WSL2. I didn't even restart my computer, it just sat there in standby for a whole night. For the record, here's two run links: • with logs showing: https://app.prefect.cloud/account/c8940b12-1562-4d20-b8fe-7420a54ef7b5/workspace/fc1b[…]26a2/flow-runs/flow-run/e6f3e6ae-d7dd-460d-8465-01898aea4373 • with logs not showing: https://app.prefect.cloud/account/c8940b12-1562-4d20-b8fe-7420a54ef7b5/workspace/fc1b[…]26a2/flow-runs/flow-run/de4efef3-cb5c-4775-be2e-360b86bc6f74 The console doesn't show any errors when clicking on any specific flow run, it does however when clicking on "Flow runs", but I don't think it's related (screenshot attached).

🙌 1

Will Raphaelson

04/11/2023, 3:37 PM

relatively flumoxed by this one

Federico Zambelli

04/11/2023, 3:38 PM

tell me about it

Will Raphaelson

04/11/2023, 3:38 PM

ill close out the issue but keep it on my radar in case it pops up again, Appreciate you working through this wonky one with us, let me know if I can do anything else for ya

Federico Zambelli

04/11/2023, 3:39 PM

Yup alright, thanks for the help, I will inform you if anything else happens again!

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.