I m in need of an easy way to login to google account in pre Prefect Community #ask-community

I'm in need of an easy way to login to google acco...

Lucas Brum

08/25/2022, 5:11 PM

I'm in need of an easy way to login to google account in prefect, but I didn't want to use Selenium. Can I use Playwright in prefect?

Ryan Peden

08/25/2022, 6:49 PM

Hi Lucas, in your previous message it looked like you were trying to sign in to Gmail and get the text of individual emails. Instead of scripting a web browser using Selenium or Playwright, might it be easier to use the Python client for the Gmail API to accomplish your goals? I only ask because using Selenium or Playwright to extract data from Gmail's HTML will be tricky since it is a very complex JavaScript SPA. Having said that, Playwright should run inside a Prefect task as long as you don't try to return any non-serializable Playwright objects like Pages.

Lucas Brum

08/25/2022, 6:58 PM

I need to log into my google account, in order to get a token that the ADS API forces me to manually get once a week. (Sorry for my english, this is not my native language.)

Ryan Peden

08/25/2022, 7:48 PM

Your English is excellent, Lucas. I'd also look into using a refresh token to obtain an updated token for the Ads API, but if that is not an option, then Playwright might help. I can't speak to the specifics of how to use Playwright to get your token, but I can confirm that it runs happily in Prefect tasks, as the following example demonstrates:

Copy code

import asyncio

from prefect import flow, task, get_run_logger
from playwright.async_api import async_playwright

@task
async def get_page_title(url: str):
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.goto(url)
        page_text = await page.inner_text("title")
        return page_text

@flow(name="Page downloader")
async def main():
    logger = get_run_logger()
    text_one = await get_page_title('<http://whatsmyuseragent.org/>')
    <http://logger.info|logger.info>(f"Page title: {text_one}")
    text_two = await get_page_title('<https://www.noaa.gov>')
    <http://logger.info|logger.info>(f"Page title: {text_two}")
    
if __name__ == '__main__':
    asyncio.run(main())

Lucas Brum

08/25/2022, 8:17 PM

I tried running it locally with "browser = p.chromium.launch(headless=False)" and it worked perfectly, but in prefect it forces me to put this "browser = p.chromium.launch(headless=True)", but with this True it does not find the html elements

Lucas Brum

08/25/2022, 8:18 PM

Copy code

Error during execution of task: Error("\n╔════════════════════════════════════════════════════════════════════════════════════════════════╗\n║ Looks like you launched a headed browser without having a XServer running. ║\n║ Set either 'headless: true' or use 'xvfb-run <your-playwright-app>' before running Playwright. ║\n║ ║\n║ <3 Playwright Team ║\n╚════════════════════════════════════════════════════════════════════════════════════════════════╝\n=========================== logs ===========================\n<launching> /root/.cache/ms-playwright/chromium-1019/chrome-linux/chrome --disable-field-trial-config --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-back-forward-cache --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=ImprovedCookieControls,LazyFrameLoading,GlobalMediaControls,DestroyProfileOnBrowserClose,MediaRouter,DialMediaRouteProvider,AcceptCHFrame,AutoExpandDetailsElement,CertificateTransparencyComponentUpdater,AvoidUnnecessaryBeforeUnloadCheckSync,Translate --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --disable-sync --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --no-service-autorun --export-tagged-pdf --no-sandbox --user-data-dir=/tmp/playwright_chromiumdev_profile-cnysrE --remote-debugging-pipe --no-startup-window\n<launched> pid=3636\n[pid=3636][err] [3636:3636:0825/200216.120549:ERROR:<http://ozone_platform_x11.cc|ozone_platform_x11.cc>(240)] Missing X server or $DISPLAY\n[pid=3636][err] [3636:3636:0825/200216.122184:ERROR:<http://env.cc|env.cc>(255)] The platform failed to initialize. Exiting.\n============================================================")

Lucas Brum

08/25/2022, 8:26 PM

If i use true receve this:

Copy code

State Message:
Error during execution of task: TimeoutError('Timeout 30000ms exceeded.\n=========================== logs ===========================\nwaiting for selector "input[name=\'identifier\']"\n============================================================')

And if I run with false on my machine it finds. Is that because it doesn't render doom?? Have other metode to acesse in this case? Or make this in prefect is impossible?

Ryan Peden

08/25/2022, 8:35 PM

It might be because Google thinks you are a bot if using a headless browser. The DOM should still be there, but Google might not be rendering the HTML you are expecting. Are you running your Prefect tasks in a Docker container?

Lucas Brum

08/29/2022, 4:57 PM

Yes

6 Views

Open in Slack

Previous Next