https://prefect.io logo
Title
g

Ghassan Hallaq

10/03/2022, 4:53 PM
Hi All, I am working on building profiler(to measure the CPU, Memory and Network IO utilizations) to attach it to any prefect2 task. Any ideas about where I can start from, does ORION collect those kind of data, so we can grab it, or need to create something from scratch?
m

Mason Menges

10/03/2022, 6:10 PM
Hey @Ghassan Hallaq One of our engineers built one for 1.0 here https://discourse.prefect.io/t/prefect-tasks-for-memory-profiling-in-prefect-1-0-recipe-by-nate-nowack/1221 the concepts here would be pretty similar as a reference
g

Ghassan Hallaq

10/03/2022, 6:14 PM
Hi @Mason Menges, yes I checked that, we need to re-factore that as we don't have FactionalTask in Prefect 2.0, and also that profiller only profile the memory.
n

Nate

10/03/2022, 6:26 PM
Hi @Ghassan Hallaq, the implementation (at least for memory profiling) is almost identical in 2.0, except you swap
FunctionTask
for
Task
as shown here as it stands, the 2.0 version here is unfinished, as the profiling info is only written to stdout by this version (and it is not currently published to PYPI). Down the road, I'd love to flesh out a collection dedicated to profiling of all kinds (would be happy to review any contributions!)
🙌 2
g

Ghassan Hallaq

10/03/2022, 6:37 PM
Thanks @Mason Menges, @Nate for your instant reply, definitely I will check those links.
🙌 1