Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

sample-flow.jpg

Hi, I am starting with prefect and stuck in a point. I am migrating my scraping system in prefect where I have to run my tasks in different remote servers. As per the sample flow attached, for the sake of distribution I need my second task to run in two remote servers in parallel. Can someone help me with right resource to refer?

What do the tasks do?
How is this implemented currently?

The tasks take input from the previous task and scrape data from the website based on that input.
Currently it is handled manually across different remote servers

<@U042HHJJG3D> So does this also mean that your `task1` and `task2` need to complete before `task3` can run as it is needing the input from the other two tasks?

I believe you’ll need some method of storing the results of those tasks to retrieve and pass into `task3`.