Hi All, I am working on building profiler(to measure the CPU, Memory and Network IO utilizations) to attach it to any prefect2 task. Any ideas about where I can start from, does ORION collect those kind of data, so we can grab it, or need to create something from scratch?

Hey @Ghassan Hallaq One of our engineers built one for 1.0 here https://discourse.prefect.io/t/prefect-tasks-for-memory-profiling-in-prefect-1-0-recipe-by-nate-nowack/1221 the concepts here would be pretty similar as a reference

Hi @Mason Menges, yes I checked that, we need to re-factore that as we don't have FactionalTask in Prefect 2.0, and also that profiller only profile the memory.

Hi @Ghassan Hallaq, the implementation (at least for memory profiling) is almost identical in 2.0, except you swap

FunctionTask

for

Task

as shown here as it stands, the 2.0 version here is unfinished, as the profiling info is only written to stdout by this version (and it is not currently published to PYPI). Down the road, I'd love to flesh out a collection dedicated to profiling of all kinds (would be happy to review any contributions!)

Thanks @Mason Menges, @Nate for your instant reply, definitely I will check those links.