https://prefect.io logo
Title
k

Kevin Weiler

07/08/2020, 3:02 PM
hi there - I’m using prefect server and a Docker agent and encountering an issue I haven’t seen before - I submit my job and then go to the UI and click run. It seems to pull the image down from storage ok, but then it just hangs and all of my jobs are in a submitted state - not sure where to debug from here
j

josh

07/08/2020, 3:03 PM
Hi @Kevin Weiler do you see the docker images running on your machine when you run something like: docker ps
k

Kevin Weiler

07/08/2020, 3:05 PM
no - I see some previous exits in
docker ps -a
but I’m not sure that’s related
j

josh

07/08/2020, 3:06 PM
What do the logs of the agent say? Could you also try running the agent with the --show-flow-logs flag to see if it shows anything that may help 🙂
k

Kevin Weiler

07/08/2020, 3:06 PM
yep, will do
lol
[2020-07-08 15:08:01,896] INFO - agent | Successfully pulled image <http://artifactory.aq.tc/prefect/prefect-demo:2020-07-08t15-07-30-188405-00-00|artifactory.aq.tc/prefect/prefect-demo:2020-07-08t15-07-30-188405-00-00>...
Traceback (most recent call last):
  File "/opt/miniconda3/bin/prefect", line 10, in <module>
    sys.exit(cli())
  File "/opt/miniconda3/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/opt/miniconda3/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/opt/miniconda3/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/miniconda3/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/miniconda3/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/miniconda3/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/opt/miniconda3/lib/python3.7/site-packages/prefect/cli/execute.py", line 52, in cloud_flow
    client = Client()
  File "/opt/miniconda3/lib/python3.7/site-packages/prefect/client/client.py", line 82, in __init__
    self._access_token_expires_at = pendulum.now()
  File "/opt/miniconda3/lib/python3.7/site-packages/pendulum/__init__.py", line 211, in now
    dt = _datetime.datetime.now(local_timezone())
  File "/opt/miniconda3/lib/python3.7/site-packages/pendulum/tz/__init__.py", line 60, in local_timezone
    return get_local_timezone()
  File "/opt/miniconda3/lib/python3.7/site-packages/pendulum/tz/local_timezone.py", line 35, in get_local_timezone
    tz = _get_system_timezone()
  File "/opt/miniconda3/lib/python3.7/site-packages/pendulum/tz/local_timezone.py", line 63, in _get_system_timezone
    return _get_unix_timezone()
  File "/opt/miniconda3/lib/python3.7/site-packages/pendulum/tz/local_timezone.py", line 242, in _get_unix_timezone
    raise RuntimeError("Unable to find any timezone configuration")
RuntimeError: Unable to find any timezone configuration
j

josh

07/08/2020, 3:09 PM
Looks fun 😄 what is the base image you are using for this flow storage?
😂 1
k

Kevin Weiler

07/08/2020, 3:09 PM
ubuntu - but I should probably confirm that
yeah
ubuntu:bionic-20200311
I think I can set an env variable to fix this (maybe)
j

josh

07/08/2020, 3:12 PM
Give that a try, also maybe try a docker run of that image and call pendulum.now to see if the same error occurs. I'm thinking it isn't prefect related but some other timezone stuff
k

Kevin Weiler

07/08/2020, 3:15 PM
yeah, agreed
if the agent bombs out on running - it should probably fail the flow though
j

josh

07/08/2020, 3:17 PM
Yeah agreed and usually it does. I think this error occurs prior to where we "expect" things could error out. I'll take note and open an issue for that because we don't want things getting stuck in submitted
k

Kevin Weiler

07/08/2020, 3:18 PM
👍
you guys are super helpful BTW - thanks
j

josh

07/08/2020, 3:19 PM
Thank you! We love hearing that
❤️ 1