Thread
#prefect-community
    k

    Kevin Weiler

    2 years ago
    hi there - I’m using prefect server and a Docker agent and encountering an issue I haven’t seen before - I submit my job and then go to the UI and click run. It seems to pull the image down from storage ok, but then it just hangs and all of my jobs are in a submitted state - not sure where to debug from here
    j

    josh

    2 years ago
    Hi @Kevin Weiler do you see the docker images running on your machine when you run something like: docker ps
    k

    Kevin Weiler

    2 years ago
    no - I see some previous exits in
    docker ps -a
    but I’m not sure that’s related
    j

    josh

    2 years ago
    What do the logs of the agent say? Could you also try running the agent with the --show-flow-logs flag to see if it shows anything that may help 🙂
    k

    Kevin Weiler

    2 years ago
    yep, will do
    lol
    [2020-07-08 15:08:01,896] INFO - agent | Successfully pulled image <http://artifactory.aq.tc/prefect/prefect-demo:2020-07-08t15-07-30-188405-00-00|artifactory.aq.tc/prefect/prefect-demo:2020-07-08t15-07-30-188405-00-00>...
    Traceback (most recent call last):
      File "/opt/miniconda3/bin/prefect", line 10, in <module>
        sys.exit(cli())
      File "/opt/miniconda3/lib/python3.7/site-packages/click/core.py", line 764, in __call__
        return self.main(*args, **kwargs)
      File "/opt/miniconda3/lib/python3.7/site-packages/click/core.py", line 717, in main
        rv = self.invoke(ctx)
      File "/opt/miniconda3/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/opt/miniconda3/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/opt/miniconda3/lib/python3.7/site-packages/click/core.py", line 956, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/opt/miniconda3/lib/python3.7/site-packages/click/core.py", line 555, in invoke
        return callback(*args, **kwargs)
      File "/opt/miniconda3/lib/python3.7/site-packages/prefect/cli/execute.py", line 52, in cloud_flow
        client = Client()
      File "/opt/miniconda3/lib/python3.7/site-packages/prefect/client/client.py", line 82, in __init__
        self._access_token_expires_at = pendulum.now()
      File "/opt/miniconda3/lib/python3.7/site-packages/pendulum/__init__.py", line 211, in now
        dt = _datetime.datetime.now(local_timezone())
      File "/opt/miniconda3/lib/python3.7/site-packages/pendulum/tz/__init__.py", line 60, in local_timezone
        return get_local_timezone()
      File "/opt/miniconda3/lib/python3.7/site-packages/pendulum/tz/local_timezone.py", line 35, in get_local_timezone
        tz = _get_system_timezone()
      File "/opt/miniconda3/lib/python3.7/site-packages/pendulum/tz/local_timezone.py", line 63, in _get_system_timezone
        return _get_unix_timezone()
      File "/opt/miniconda3/lib/python3.7/site-packages/pendulum/tz/local_timezone.py", line 242, in _get_unix_timezone
        raise RuntimeError("Unable to find any timezone configuration")
    RuntimeError: Unable to find any timezone configuration
    j

    josh

    2 years ago
    Looks fun 😄 what is the base image you are using for this flow storage?
    k

    Kevin Weiler

    2 years ago
    ubuntu - but I should probably confirm that
    yeah
    ubuntu:bionic-20200311
    I think I can set an env variable to fix this (maybe)
    j

    josh

    2 years ago
    Give that a try, also maybe try a docker run of that image and call pendulum.now to see if the same error occurs. I'm thinking it isn't prefect related but some other timezone stuff
    k

    Kevin Weiler

    2 years ago
    yeah, agreed
    if the agent bombs out on running - it should probably fail the flow though
    j

    josh

    2 years ago
    Yeah agreed and usually it does. I think this error occurs prior to where we "expect" things could error out. I'll take note and open an issue for that because we don't want things getting stuck in submitted
    k

    Kevin Weiler

    2 years ago
    👍
    you guys are super helpful BTW - thanks
    j

    josh

    2 years ago
    Thank you! We love hearing that