https://prefect.io logo
#prefect-community
Title
# prefect-community
f

Frederick Thomas

05/13/2022, 9:32 PM
Hi all, we've just upgraded the python version to 3.10 and re-registered all the flows but we are getting this error, could someone assist?:
Copy code
Exception raised while calling state handlers: SystemError('unknown opcode')
Traceback (most recent call last):
  File "/mnt/data/prefect/venv/lib/python3.8/site-packages/prefect/engine/cloud/flow_runner.py", line 119, in call_runner_target_handlers
    new_state = super().call_runner_target_handlers(
  File "/mnt/data/prefect/venv/lib/python3.8/site-packages/prefect/engine/flow_runner.py", line 116, in call_runner_target_handlers
    new_state = handler(self.flow, old_state, new_state) or new_state
  File "/mnt/data/prefect/venv3.10/lib/python3.10/site-packages/prefect/utilities/notifications/notifications.py", line 65, in state_handler
    def state_handler(
SystemError: unknown opcode
k

Kevin Kho

05/13/2022, 9:42 PM
unknown opcode is highly likely a mismatch of Dask versions somewhere between Flow, scheduler, and worker
h

Haleemur Ali

05/14/2022, 3:12 AM
Also, note that the logs say you have python 3.8
f

Frederick Thomas

05/16/2022, 4:02 PM
Good Morning, We're still having the issue when we upgrade to python 3.10, however we have found that both the server and the agent are being run from supervisorctl. We renamed the 3.10 virtual environment to venv, re-registered again and now the UI only shows yellow, the flows never run. We are able to rollback to 3.8, here is conf.d for the server:
cat /etc/supervisor/conf.d/prefect-server.conf
[program:prefect-server]
startsecs=30
stopwaitsecs=30
autorestart=true
user=prefect
group=prefect
stdout_logfile=/mnt/data/prefect/server/log/server.log
environment=PATH="/mnt/data/prefect/venv/bin:%(ENV_PATH)s"
command=prefect server start --use-volume --volume-path /mnt/data/prefect/server/pg_data
priority=400
stopsignal=INT
Any help would be appreciated, thanks.
k

Kevin Kho

05/16/2022, 5:45 PM
Is it stuck in scheduled or submitted?
f

Frederick Thomas

05/16/2022, 6:05 PM
Hold on I have to ask my coding buddy...
k

Kevin Kho

05/16/2022, 6:09 PM
The specific issue though is that your Flow is running in 3.10 but the Dask workers are spinning up 3.8 I think. What image are you using for that?
f

Frederick Thomas

05/16/2022, 6:12 PM
Dunno, I do know there are a bunch of sus symlinks... still waiting on my coding buddy
If we're using the wrong <docker> image where would we find the correct image??
k

Kevin Kho

05/16/2022, 8:05 PM
Not sure I can answer that for you. It would be whatever you used for the Flow so that Flow, and cluster have same dependency versions