Hello community, has anyone had issues implementing Prefect with Selenium? My case is really special and I can only use web scraping to get the data I need.
✅ 1
r
Ryan Peden
09/08/2022, 3:04 PM
I've seen Selenium come up here on Slack (and a similar use case using Playwright instead of Selenium), and the issues people encountered were related to Selenium, not Prefect.
In one case, neither Selenium not Playwright worked properly when the user tried to use a headless browser in the environment where they ran their Prefect agent because the site the were trying to scrape treated headless browsers as bots.
In general I expect Selenium will work inside a Prefect task, though.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.