https://prefect.io logo
Title
n

Nathaniel Russell

08/26/2022, 3:04 PM
Task 'sync_canvas': Exception encountered during task execution!
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 859, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/executors.py", line 445, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "<string>", line 28, in sync_canvas
TypeError: 'NoneType' object is not iterable
We are using prefect on EC2 spot instances and we think this is likely caused by AWS dropping our spot instances. Is there a way around this while still using spot instances or is this unavoidable with spot instances?
t

Tim Enders

08/26/2022, 3:07 PM
Due to the nature of spot instances being able to be killed by AWS at any time, I would say it is unavoidable 100%. But what version of PRefect are you running?
n

Nathaniel Russell

08/26/2022, 3:08 PM
prefect==0.15.10
t

Tim Enders

08/26/2022, 3:10 PM
The error is telling me that you have a NoneType that you are trying to iterate over. I found that to be possible if I return a None from a Task/Function whose result I map over. The error comes like that from the executor, not either Task. My workaround was to return [] when I had no data, then it is iterable, and can be evaluated in the mapped Task
n

Nathaniel Russell

08/26/2022, 3:11 PM
Also interestingly enough, rescheduling the flow works, so if there was a way to automate rescheduling then that should solve this.
t

Tim Enders

08/26/2022, 3:12 PM
Is the Task on a Spot instance, or the Flow?
n

Nathaniel Russell

08/26/2022, 3:13 PM
also yes our code returns [] if events is empty and never returns None. Thus why we think it is spot instances problem. I believe both the task and the flow are hosted on a spot instance as that is where we build our prefect agents.
t

Tim Enders

08/26/2022, 3:16 PM
Hmmm if the Task is on its own instance, I think that Prefect might return
None
from the crash when a spot is closed down? If it is within the Flow instance it all should come crashing down and not give you that error. I think. Might need someone to weigh in from Prefect