Hi guys anybody have this issue Stack trace in the reply I t Prefect Community #ask-community

Join Slack

Hi guys, anybody have this issue? Stack trace in t...

# ask-community

Kien Nguyen

08/13/2021, 2:55 PM

Hi guys, anybody have this issue? Stack trace in the reply I think it happens after I add S3Result 🥲

Kevin Kho

08/13/2021, 2:56 PM

Hey @Kien Nguyen , a bit hard to tell what this is, but Prefect doesn’t have official Python 3.9 support, though it doesn’t seem to be that. Could you move the traceback to the thread so we don’t crowd the main channel when you get the chance?

Kevin Kho

08/13/2021, 2:57 PM

Removing S3 Result makes this work?

Kien Nguyen

08/13/2021, 2:58 PM

Sure. This is the stack trace

Copy code

2021-08-13T14:52:07.815047+00:00 app[worker.1]: No module named '/app/'
2021-08-13T14:52:07.815101+00:00 app[worker.1]: Traceback (most recent call last):
2021-08-13T14:52:07.815103+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/runpy.py", line 197, in _run_module_as_main
2021-08-13T14:52:07.815619+00:00 app[worker.1]: return _run_code(code, main_globals, None,
2021-08-13T14:52:07.815632+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/runpy.py", line 87, in _run_code
2021-08-13T14:52:07.815705+00:00 app[worker.1]: exec(code, run_globals)
2021-08-13T14:52:07.815715+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/prefect/__main__.py", line 4, in <module>
2021-08-13T14:52:07.815765+00:00 app[worker.1]: cli()
2021-08-13T14:52:07.815775+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/click/core.py", line 829, in __call__
2021-08-13T14:52:07.815950+00:00 app[worker.1]: return self.main(*args, **kwargs)
2021-08-13T14:52:07.815959+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/click/core.py", line 782, in main
2021-08-13T14:52:07.816121+00:00 app[worker.1]: rv = self.invoke(ctx)
2021-08-13T14:52:07.816130+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
2021-08-13T14:52:07.816367+00:00 app[worker.1]: return _process_result(sub_ctx.command.invoke(sub_ctx))
2021-08-13T14:52:07.816368+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
2021-08-13T14:52:07.816606+00:00 app[worker.1]: return _process_result(sub_ctx.command.invoke(sub_ctx))
2021-08-13T14:52:07.816616+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
2021-08-13T14:52:07.816823+00:00 app[worker.1]: return ctx.invoke(self.callback, **ctx.params)
2021-08-13T14:52:07.816832+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/click/core.py", line 610, in invoke
2021-08-13T14:52:07.816965+00:00 app[worker.1]: return callback(*args, **kwargs)
2021-08-13T14:52:07.816974+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/prefect/cli/execute.py", line 96, in flow_run
2021-08-13T14:52:07.817032+00:00 app[worker.1]: raise exc
2021-08-13T14:52:07.817042+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/prefect/cli/execute.py", line 73, in flow_run
2021-08-13T14:52:07.817093+00:00 app[worker.1]: flow = storage.get_flow(flow_data.name)
2021-08-13T14:52:07.817102+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/prefect/storage/local.py", line 102, in get_flow
2021-08-13T14:52:07.817162+00:00 app[worker.1]: return extract_flow_from_module(
2021-08-13T14:52:07.817176+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/prefect/utilities/storage.py", line 126, in extract_flow_from_module
2021-08-13T14:52:07.817233+00:00 app[worker.1]: module = importlib.import_module(mod_name)
2021-08-13T14:52:07.817242+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/importlib/__init__.py", line 127, in import_module
2021-08-13T14:52:07.817304+00:00 app[worker.1]: return _bootstrap._gcd_import(name[level:], package, level)
2021-08-13T14:52:07.817313+00:00 app[worker.1]: File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
2021-08-13T14:52:07.817383+00:00 app[worker.1]: File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
2021-08-13T14:52:07.817421+00:00 app[worker.1]: File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
2021-08-13T14:52:07.817458+00:00 app[worker.1]: File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
2021-08-13T14:52:07.817496+00:00 app[worker.1]: File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
2021-08-13T14:52:07.817532+00:00 app[worker.1]: File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
2021-08-13T14:52:07.817569+00:00 app[worker.1]: File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
2021-08-13T14:52:07.817618+00:00 app[worker.1]: ModuleNotFoundError: No module named '/app/'
2021-08-13T14:52:48.669744+00:00 app[worker.1]: [2021-08-13 14:52:48,665] INFO - agent | Process PID 20 returned non-zero exit code 1!

Kien Nguyen

08/13/2021, 3:00 PM

Hmm, I'm trying to revert back step by step

Kien Nguyen

08/13/2021, 3:04 PM

So I commented out the disabling of

add_default_lables

Copy code

with Flow('Historical Trades Yearly ETL', result=s3_result) as yearly_flow:
    yearly_dates = get_yearly_trading_dates()
    symbol = Parameter('symbol', 'SPY')
    limit = Parameter('limit', 10000)
    dates = Parameter('trading_dates', yearly_dates)
    loop_over_trading_dates(symbol, limit, dates)

# yearly_flow.storage.add_default_labels = False

Kien Nguyen

08/13/2021, 3:04 PM

and it works

Kien Nguyen

08/13/2021, 3:05 PM

is it the correct way to disable the default hostname label @Kevin Kho?

Kevin Kho

08/13/2021, 3:08 PM

I think there is another way one sec

Kevin Kho

08/13/2021, 3:12 PM

It does look right but I guess you can right it this way. I think it will also error out though

flow = Flow("local-flow", storage=Local(add_default_labels=False))

Kien Nguyen

08/13/2021, 3:15 PM

alright, will check it out later

Kien Nguyen

08/13/2021, 3:15 PM

Thanks @Kevin Kho

Kien Nguyen

08/14/2021, 3:42 AM

Hmm, it happens again. I didn't change anything. ~15 first run it is okay, the last run have the issue 😞

Kevin Kho

08/14/2021, 3:53 AM

Is this a Dask Executor?

Kien Nguyen

08/14/2021, 3:53 AM

Nope, it's the default LocalExecutor

Kevin Kho

08/14/2021, 4:07 AM

Are your versions always the same for each run? And does it work if you run it again?

Kien Nguyen

08/14/2021, 4:15 AM

I think the agent restarted, but not sure why

Kien Nguyen

08/19/2021, 2:38 PM

Now I have issue with even python-3.8

Copy code

2021-08-19T14:20:05.362756+00:00 app[worker.1]: Backend switched to cloud
2021-08-19T14:20:06.039248+00:00 app[worker.1]: Logged in to Prefect Cloud tenant 'Maneki Automation Team' (maneki-automation)
2021-08-19T14:20:07.285665+00:00 app[worker.1]: Flow URL: <https://cloud.prefect.io/maneki-automation/flow/16c0d823-38d4-4cce-9c0a-1db5be4a80dc>
2021-08-19T14:20:07.285674+00:00 app[worker.1]: └── ID: e9a8288e-1909-4d5b-9787-b6c67072dbd5
2021-08-19T14:20:07.285675+00:00 app[worker.1]: └── Project: Maneki DCT Batch Engine
2021-08-19T14:20:07.285675+00:00 app[worker.1]: └── Labels: ['faff87ea-a277-45e6-956f-08cdd7299dfa']
2021-08-19T14:20:07.661952+00:00 app[worker.1]: Flow URL: <https://cloud.prefect.io/maneki-automation/flow/d55ecd9a-b246-42b8-bea9-6ba7109b89cd>
2021-08-19T14:20:07.661960+00:00 app[worker.1]: └── ID: 6c22a982-5e09-4e77-bdcc-1a817150b873
2021-08-19T14:20:07.661961+00:00 app[worker.1]: └── Project: Maneki DCT Batch Engine
2021-08-19T14:20:07.661963+00:00 app[worker.1]: └── Labels: ['faff87ea-a277-45e6-956f-08cdd7299dfa']
2021-08-19T14:20:08.166493+00:00 app[worker.1]: [2021-08-19 14:20:08,166] INFO - agent | Registering agent...
2021-08-19T14:20:08.412561+00:00 app[worker.1]: [2021-08-19 14:20:08,412] INFO - agent | Registration successful!
2021-08-19T14:20:08.561671+00:00 app[worker.1]: 
2021-08-19T14:20:08.561683+00:00 app[worker.1]: ____            __           _        _                    _
2021-08-19T14:20:08.561683+00:00 app[worker.1]: |  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
2021-08-19T14:20:08.561684+00:00 app[worker.1]: | |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
2021-08-19T14:20:08.561685+00:00 app[worker.1]: |  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
2021-08-19T14:20:08.561685+00:00 app[worker.1]: |_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
2021-08-19T14:20:08.561685+00:00 app[worker.1]: |___/
2021-08-19T14:20:08.561690+00:00 app[worker.1]: 
2021-08-19T14:20:08.562130+00:00 app[worker.1]: [2021-08-19 14:20:08,561] INFO - agent | Starting LocalAgent with labels []
2021-08-19T14:20:08.562203+00:00 app[worker.1]: [2021-08-19 14:20:08,562] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
2021-08-19T14:20:08.562244+00:00 app[worker.1]: [2021-08-19 14:20:08,562] INFO - agent | Waiting for flow runs...
2021-08-19T14:33:50.658385+00:00 app[worker.1]: [2021-08-19 14:33:50,658] INFO - agent | Deploying flow run 580a9b5a-e064-48ce-b7df-bb5367c9976c to execution environment...
2021-08-19T14:33:50.805546+00:00 app[worker.1]: [2021-08-19 14:33:50,805] INFO - agent | Completed deployment of flow run 580a9b5a-e064-48ce-b7df-bb5367c9976c
2021-08-19T14:33:51.578645+00:00 app[worker.1]: No module named '/app/'
2021-08-19T14:33:51.578723+00:00 app[worker.1]: Traceback (most recent call last):
2021-08-19T14:33:51.578735+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/runpy.py", line 194, in _run_module_as_main
2021-08-19T14:33:51.579284+00:00 app[worker.1]: return _run_code(code, main_globals, None,
2021-08-19T14:33:51.579286+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/runpy.py", line 87, in _run_code
2021-08-19T14:33:51.579365+00:00 app[worker.1]: exec(code, run_globals)
2021-08-19T14:33:51.579375+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/prefect/__main__.py", line 4, in <module>
2021-08-19T14:33:51.579430+00:00 app[worker.1]: cli()
2021-08-19T14:33:51.579431+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/click/core.py", line 829, in __call__
2021-08-19T14:33:51.579622+00:00 app[worker.1]: return self.main(*args, **kwargs)
2021-08-19T14:33:51.579630+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/click/core.py", line 782, in main
2021-08-19T14:33:51.579830+00:00 app[worker.1]: rv = self.invoke(ctx)
2021-08-19T14:33:51.579832+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
2021-08-19T14:33:51.580126+00:00 app[worker.1]: return _process_result(sub_ctx.command.invoke(sub_ctx))
2021-08-19T14:33:51.580134+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
2021-08-19T14:33:51.580389+00:00 app[worker.1]: return _process_result(sub_ctx.command.invoke(sub_ctx))
2021-08-19T14:33:51.580397+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
2021-08-19T14:33:51.580651+00:00 app[worker.1]: return ctx.invoke(self.callback, **ctx.params)
2021-08-19T14:33:51.580651+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/click/core.py", line 610, in invoke
2021-08-19T14:33:51.580789+00:00 app[worker.1]: return callback(*args, **kwargs)
2021-08-19T14:33:51.580790+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/prefect/cli/execute.py", line 96, in flow_run
2021-08-19T14:33:51.580860+00:00 app[worker.1]: raise exc
2021-08-19T14:33:51.580861+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/prefect/cli/execute.py", line 73, in flow_run
2021-08-19T14:33:51.580918+00:00 app[worker.1]: flow = storage.get_flow(flow_data.name)
2021-08-19T14:33:51.580927+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/prefect/storage/local.py", line 102, in get_flow
2021-08-19T14:33:51.580990+00:00 app[worker.1]: return extract_flow_from_module(
2021-08-19T14:33:51.580999+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/prefect/utilities/storage.py", line 126, in extract_flow_from_module
2021-08-19T14:33:51.581071+00:00 app[worker.1]: module = importlib.import_module(mod_name)
2021-08-19T14:33:51.581072+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/importlib/__init__.py", line 127, in import_module
2021-08-19T14:33:51.581146+00:00 app[worker.1]: return _bootstrap._gcd_import(name[level:], package, level)
2021-08-19T14:33:51.581148+00:00 app[worker.1]: File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
2021-08-19T14:33:51.581218+00:00 app[worker.1]: File "<frozen importlib._bootstrap>", line 991, in _find_and_load
2021-08-19T14:33:51.581258+00:00 app[worker.1]: File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
2021-08-19T14:33:51.581297+00:00 app[worker.1]: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2021-08-19T14:33:51.581333+00:00 app[worker.1]: File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
2021-08-19T14:33:51.581370+00:00 app[worker.1]: File "<frozen importlib._bootstrap>", line 991, in _find_and_load
2021-08-19T14:33:51.581406+00:00 app[worker.1]: File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
2021-08-19T14:33:51.581461+00:00 app[worker.1]: ModuleNotFoundError: No module named '/app/'
2021-08-19T14:34:09.391643+00:00 app[worker.1]: [2021-08-19 14:34:09,388] INFO - agent | Process PID 24 returned non-zero exit code 1!

Kien Nguyen

08/19/2021, 2:59 PM

and it seems persistent also

Kevin Kho

08/19/2021, 3:12 PM

Is this when you run a flow or just leaving an agent on?

Kevin Kho

08/19/2021, 3:13 PM

Does the agent die or does it continue after not running a flow?

Kevin Kho

08/19/2021, 5:39 PM

Are you trying to import the

app

module?

Kien Nguyen

08/20/2021, 8:09 AM

Hmm, no I don't import any app module

Kien Nguyen

08/20/2021, 8:10 AM

It's kinda intermittent though. I restart and no problem

Kevin Kho

08/20/2021, 3:34 PM

What Storage are you using for this Flow?

Kien Nguyen

08/21/2021, 2:53 AM

Flow storage is Github, and result storage is AzureResult

Kevin Kho

08/22/2021, 3:39 AM

That is so confusing I have no idea what is loading app. I suspect it's on the Heroku sided but I don't know enough unfortunately on Heroku to know.

8 Views

Open in Slack

Previous Next