Prefect 2.0.4, dask, prefect UI displays logs from flows and subflows only, I have set loggers in task but those logs are not showing up in UI, I am using info logs only
✅ 1
c
Christopher Boyd
08/30/2022, 1:30 PM
Hi Faheem,
Would it be feasible to test this in another venv using the latest (2.2.0) as of currently?
Additionally, what does your code look like to log those tasks?
I have seen reports in the past where tasks do eventually get logged, but it’s not immediately - that is, the flows and subflows log in real-time, but tasks (because they are distributed exection in dask) are not quite real-time, as the results are batch written out
With the logging issue I think, as u said it would be because of the task being distruibet on dask cluster. what do u suggest the best approach would for logging in the tasks?
c
Christopher Boyd
08/31/2022, 12:31 PM
By their nature, they are ephemeral, so have to be logged in the task themselves.
I would ensure you are getting getting a new task logger context (in the task itself) and logging the output during that task. This might be slightly asynchronous (not real-time logging) as I mentioned, because the tasks and dask pods will queue writes to the batch writers,
Christopher Boyd
08/31/2022, 12:32 PM
If you can post a snippet of how you have your logging setup for the tasks?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.