Using Scrapy and CrawlProcess in Prefect 2
Today I learned Scrapy CrawlProcess complains about needing to be the main process if used in tasks or subflows. I had to move the Scrapy process under the main flow for it to work.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
