https://prefect.io logo
Title
j

Joe Schmid

04/30/2020, 12:26 AM
Hi folks, does anyone have a recipe for building a multi-flow docker image on your own, i.e. without calling
storage.build()
or
flow.register(build=True)
I'm calling
flow.save()
and
flow.register(build=False)
in my Dockerfile and the registered flows look correct(Storage, Flow Locations, etc. look good in Cloud UI) but my flow runs fail with:
Unexpected error while running flow: KeyError('a60e3215-88c7-499c-b525-83ba87b817f2')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/flow_runner.py", line 241, in run
    parameters=parameters,
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/cloud/flow_runner.py", line 185, in initialize_run
    task = tasks[task_run.task_slug]
KeyError: 'a60e3215-88c7-499c-b525-83ba87b817f2'
(I do see this doc: https://docs.prefect.io/orchestration/recipes/multi_flow_storage.html and that's close but not quite what I'm looking for)
Additional details:
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/cloud/flow_runner.py", line 119, in call_runner_target_handlers
    flow_run_id=flow_run_id, version=version, state=cloud_state
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 939, in set_flow_run_state
    version=version,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 217, in graphql
    token=token,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 177, in post
    token=token,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 314, in _request
    response.raise_for_status()
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: <https://api.prefect.io/graphql/alpha>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/environments/execution/remote.py", line 84, in execute
    runner_cls(flow=flow).run(executor=executor)
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/flow_runner.py", line 275, in run
    state = self.handle_state_change(state or Pending(), new_state)
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 162, in handle_state_change
    new_state = self.call_runner_target_handlers(old_state, new_state)
  File "/usr/local/lib/python3.7/site-packages/prefect/engine/cloud/flow_runner.py", line 125, in call_runner_target_handlers
    raise ENDRUN(state=new_state)
prefect.engine.signals.ENDRUN
c

Chris White

04/30/2020, 12:28 AM
Hey @Joe Schmid! So I’m not sure at what step this is happening, but that error is almost always a symptom of the flow being recreated in between registering with Cloud and running it (because the task IDs changed)
are you calling
flow.save
in a different process than where you call
flow.register
?
j

Joe Schmid

04/30/2020, 12:29 AM
Ah, that's probably it. I'm doing them as separate RUN steps in a Dockerfile, but I can easily combine. Thanks for the quick help!
c

Chris White

04/30/2020, 12:29 AM
awesome, anytime!
🤘 1