Felipe Fernandez

09/08/2022, 2:54 PM
Hello community, has anyone had issues implementing Prefect with Selenium? My case is really special and I can only use web scraping to get the data I need.

Ryan Peden

09/08/2022, 3:04 PM
I've seen Selenium come up here on Slack (and a similar use case using Playwright instead of Selenium), and the issues people encountered were related to Selenium, not Prefect. In one case, neither Selenium not Playwright worked properly when the user tried to use a headless browser in the environment where they ran their Prefect agent because the site the were trying to scrape treated headless browsers as bots. In general I expect Selenium will work inside a Prefect task, though.
In case it's useful, here's a previous thread where I provided an example using Playwright:

Felipe Fernandez

09/08/2022, 4:37 PM
I'm sorry Ryan and thank you for your answer, I was confused with my issue, I'm having difficulties with Paramiko to upload some files into SFTP