Hi, I’m a backend engineer and have never migrated data from a legacy system to a new one. However, I’m currently building a prototype ETL/migration pipeline. My question is: can I create a Prefect task for each record in my database (assuming it contains millions of records)? Would that approach make sense? I’d like to track the status of each record through the pipeline—whether it succeeded in the extract, transform, or load steps, or failed during transformation. That way, I can pinpoint which records failed, determine where and why they failed, and restart them as needed. Any suggestions would be greatly appreciated, as I’m not very familiar with this aspect of development. Thanks

hi @Ahmad Bilal Khalid - welcome! you are free to create that many task runs, but oftentimes it’s not necessary. for example, you could write a task for each step (E, T, L), where each task can process N records. you could use a “global” dictionary of some kind to track failures (and their associated record id for example) in on_completion hooks, using try/except within these tasks as necessary to trap any exceptions that might occur. the main reason i’d suggest this is that the UI will become bloated if you have on the order of millions of runs. does that help?

Yep, thanks 🙂