https://prefect.io logo
Powered by
Title
f

Federico Zambelli

04/10/2023, 4:28 PM
Hey folks, I'm going through the "getting started" with prefect cloud and I can't seem to be able to see any logs on the Cloud UI. It does show my flow runs, but logs are empty, no matter what. I used 
log_prints=True
, as well as 
get_run_logger
, and neither case worked. What am I missing? for the record, this is my .py module: 
from prefect import flow, get_run_logger


@flow(log_prints=True)
def hi():
    logger = get_run_logger()
    logger.warning("Hit from Prefect! 😅")


if __name__ == "__main__":
    hi()
✅ 1
w

Will Raphaelson

04/10/2023, 4:32 PM
Hey Federico, just tried that code and I can see the logs. Mind showing me the output of the CLI command 
prefect version
?
f

Federico Zambelli

04/10/2023, 4:32 PM
Hey will, thanks for the quick reply. Here's the output:
Could this possibly be due to the fact that months ago I was using 
prefect
on the same system but with a local orion server?
w

Will Raphaelson

04/10/2023, 4:34 PM
it shouldn’t. just to confirm, you see the flow run in the ui, just not the logs? And the name of the flow run that you see in the terminal is the same as the one you see in the ui?
also I misspelled your name before, sorry! edited.
f

Federico Zambelli

04/10/2023, 4:36 PM
Yup I do see it in the UI, but no logs, it says "This run didn't generate any logs". The name is exactly the same, see: (no worries about name misspell, happens all the time 😄 )
w

Will Raphaelson

04/10/2023, 4:37 PM
okay i have one more thing if you dont mind. We’ve seen this elusive bug when only sending one log. Can you duplicate the log line and run again?
f

Federico Zambelli

04/10/2023, 4:37 PM
Yup, gimme 1 second
n

Nate

04/10/2023, 4:37 PM
hmm I assume this is related to using log_prints=True as well as the logger (instead of print) since log_prints literally patches the builtin print - do you see the same if you use 
log_prints=True
and 
print
instead of 
logger.warning
?
w

Will Raphaelson

04/10/2023, 4:38 PM
yeah good line of questioning nate. This was mine as well but im able to run that code and have the logs appear anyway.
🧐 1
n

Nate

04/10/2023, 4:40 PM
interesting, I'm seeing the same (logs appear both in terminal and ui logs for me)
f

Federico Zambelli

04/10/2023, 4:40 PM
ok so I tried all possibilities: • logger with 
log_prints
• logger without 
log_prints
• print with 
log_prints
All of them logging two separate messages. None of them showing up in the UI
n

Nate

04/10/2023, 4:42 PM
do you get the same result if you check your LOGGING_TO_API_ENABLED setting? 
In [8]: import prefect

In [9]: prefect.settings.PREFECT_LOGGING_TO_API_ENABLED.value()
Out[9]: True
f

Federico Zambelli

04/10/2023, 4:44 PM
lemme check
it does return 
True
yes
🤔 1
Lemme try something: how do I switch from a cloud backend to a local orion server? I wanna see if the logs get printed there
w

Will Raphaelson

04/10/2023, 4:48 PM
prefect config set PREFECT_API_URL=<http://127.0.0.1:4200/api>
(youll want to run 
prefect server start
as well, before switching the config value. You can also create a profile for quick switching. https://docs.prefect.io/latest/concepts/settings/#configuration-profiles
:upvote: 2
f

Federico Zambelli

04/10/2023, 4:50 PM
ok let's see
local UI does show the logs
wtf
image.png
n

Nate

04/10/2023, 4:54 PM
have you tried sending an INFO level log to cloud instead?
f

Federico Zambelli

04/10/2023, 4:54 PM
yeah I had actually tired with 
INFO
first, then as I saw no logs happening I switched to 
WARN
to try
👍 1
w

Will Raphaelson

04/10/2023, 4:57 PM
i feel determined to catch this ghost. The only difference between our configs is the os/arch. I can ask a colleague with a linux machine to try this just to rule that out as a thing.
f

Federico Zambelli

04/10/2023, 4:57 PM
for the record, I'm running this on WSL2
w

Will Raphaelson

04/10/2023, 5:08 PM
Thanks, my linuxy colleage also couldnt reproduce. Might have to file the quickstart issue for deeper investigation later. In the interim, are you able to just use the get_run_logger path to show logs in the cloud ui?
f

Federico Zambelli

04/10/2023, 5:09 PM
wdym the get_run_logger path? If you mean the function to get the logger object, that didn't work either
w

Will Raphaelson

04/10/2023, 5:10 PM
oh sorry i see you tried that before
(thinking through some options internally, ill circle back on this)
f

Federico Zambelli

04/10/2023, 5:16 PM
yup no worries, thanks for the help. Do you want me to test directly on windows, without WSL2 ?
w

Will Raphaelson

04/10/2023, 5:16 PM
that would be excellent and help narrow it down
f

Federico Zambelli

04/10/2023, 5:16 PM
ok gimme a few minutes or so
:thank-you: 1
@Will Raphaelson ok this is interesting. It seems that the issue is not with Linux, but with something related to my machine. I tried running the same flow under windows and it didn't log anything to the cloud UI as well
and just as an FYI, I tried deleting the local 
.prefect
folder in home in Windows, and let prefect recreate it, but nonetheless I see no logs. 🤔 🤔 🤔 🤔
w

Will Raphaelson

04/10/2023, 5:39 PM
okay well thats progress at least.
can you show me the contents of that .prefect folder please?
f

Federico Zambelli

04/10/2023, 5:41 PM
linux or windows one?
w

Will Raphaelson

04/10/2023, 5:42 PM
i guess lets start with windows.
f

Federico Zambelli

04/10/2023, 5:42 PM
sure
ah since I deleted it right now there's only 
profiles.toml
image.png
image.png
w

Will Raphaelson

04/10/2023, 5:47 PM
and after removing and recreating, you ran the script again and its still not showing logs right? We may have to open a github issue on this, reaching the end of my investigative abilities
f

Federico Zambelli

04/10/2023, 5:47 PM
yeah, I pip installed prefect in a virtual environment, pasted the script, ran 
prefect cloud login
, authenticated via browser, ran 
python my_flow.py
, that's it
w

Will Raphaelson

04/10/2023, 5:56 PM
got it, thanks. I think unfortunately ill need to file an issue and escalate to our team. My only thought is that something on the machine is blocking the calls out to the logs endpoint, but i dont know why that would happen.
f

Federico Zambelli

04/10/2023, 5:58 PM
yeah no worries Will, thanks for the help. Let me know if you or your team needs to get in touch since this is quite a peculiar situation
:thank-you: 1
w

Will Raphaelson

04/10/2023, 5:58 PM
thanks will do, and I’ll post the github issue here when i file it in a few.
🙌 1
Writing it up now, can you give me a few more details about the infra here? Is this an azure VM, a laptop?
feel free to comment directly on https://github.com/PrefectHQ/prefect/issues/9172
r

Ryan Peden

04/10/2023, 7:53 PM
You can try running 
prefect config set PREFECT_DEBUG_MODE=True
to get a deeper look at what's going on. Specifically, you should see messages from 
prefect.logging.handlers.APILogWorker
. First, you should see one like 
enqueueing item {'name': 'prefect.flow_runs', 'level': 30, 'message': 'Hit from Prefect! 😅' ...timestamp and ID fields }
And then another one a bit later that looks the same except instead of 
enqueueing item
it will say 
added item
. It seems like the flow itself is able to make API calls to create the flow and report its completion, but 
APILogWorker
isn't sending anything to the API. It would be interesting to see if the log worker reports any problems. Even if it doesn't report any problem, that would be a useful clue as well! 🔎
f

Federico Zambelli

04/10/2023, 11:07 PM
@Will Raphaelson @Ryan Peden done, sorry for the delay, I was out for dinner. Also added the debug logs, thanks for the headsup
r

Ryan Peden

04/10/2023, 11:14 PM
Thanks! If you have a chance, could you check and see if any errors show up in the Chrome dev tools console when you try to view the flow run logs in the cloud UI?
👍 1
w

Will Raphaelson

04/10/2023, 11:21 PM
(and or give us a url for the flow run and we can check the cloud db to see if these logs even got to us)
f

Federico Zambelli

04/11/2023, 7:39 AM
Ok so, it seems the problem fixed itself 🤔 . I tried again this morning and now it's working. Both on Windows and WSL2. I didn't even restart my computer, it just sat there in standby for a whole night. For the record, here's two run links: • with logs showing: https://app.prefect.cloud/account/c8940b12-1562-4d20-b8fe-7420a54ef7b5/workspace/fc1b[…]26a2/flow-runs/flow-run/e6f3e6ae-d7dd-460d-8465-01898aea4373 • with logs not showing: https://app.prefect.cloud/account/c8940b12-1562-4d20-b8fe-7420a54ef7b5/workspace/fc1b[…]26a2/flow-runs/flow-run/de4efef3-cb5c-4775-be2e-360b86bc6f74 The console doesn't show any errors when clicking on any specific flow run, it does however when clicking on "Flow runs", but I don't think it's related (screenshot attached).
🙌 1
w

Will Raphaelson

04/11/2023, 3:37 PM
relatively flumoxed by this one
f

Federico Zambelli

04/11/2023, 3:38 PM
tell me about it
w

Will Raphaelson

04/11/2023, 3:38 PM
ill close out the issue but keep it on my radar in case it pops up again, Appreciate you working through this wonky one with us, let me know if I can do anything else for ya
f

Federico Zambelli

04/11/2023, 3:39 PM
Yup alright, thanks for the help, I will inform you if anything else happens again!
#prefect-cloudJoin Slack
Powered by