What would be the best practice/patterns how to implement a web crawler with Prefect ? Using the LOOP signal to recurse until some kind of depth limit?

Without knowing the details, a rule of thumb for deciding how to iterate in Prefect is if the inputs are possibly unbounded, then we recommend using LOOP. For example, if you’re walking each page of a news aggregator or comments section and don’t know how many there are in advance. If on the other hand the input is bounded and known/knowable, we recommend using a map over a list. For example, use a task to generate all the url’s you need to check, then map over them. And if you’re not sure, prefer map to loop.

Thanks for the answer. It sound it would be a combo of loop to gather the urls and then map over a list of the urls. Is it possible to map over a mutable list ?

I don’t want to push you too strongly to use map if it isn’t appropriate — the reason we generally prefer it is just that working with a flow is easier if each step is known, and looping implicitly hides steps from you (and is slightly more complex to code). However if it fits your use case, go for it.

Thanks for the precision. Map looks more elegant to me too.