Hi all, we've just upgraded the python version to 3.10 and re-registered all the flows but we are getting this error, could someone assist?:

Exception raised while calling state handlers: SystemError('unknown opcode')
Traceback (most recent call last):
  File "/mnt/data/prefect/venv/lib/python3.8/site-packages/prefect/engine/cloud/flow_runner.py", line 119, in call_runner_target_handlers
    new_state = super().call_runner_target_handlers(
  File "/mnt/data/prefect/venv/lib/python3.8/site-packages/prefect/engine/flow_runner.py", line 116, in call_runner_target_handlers
    new_state = handler(self.flow, old_state, new_state) or new_state
  File "/mnt/data/prefect/venv3.10/lib/python3.10/site-packages/prefect/utilities/notifications/notifications.py", line 65, in state_handler
    def state_handler(
SystemError: unknown opcode

unknown opcode is highly likely a mismatch of Dask versions somewhere between Flow, scheduler, and worker

Also, note that the logs say you have python 3.8

Good Morning, We're still having the issue when we upgrade to python 3.10, however we have found that both the server and the agent are being run from supervisorctl. We renamed the 3.10 virtual environment to venv, re-registered again and now the UI only shows yellow, the flows never run. We are able to rollback to 3.8, here is conf.d for the server:

cat /etc/supervisor/conf.d/prefect-server.conf

[program:prefect-server]

startsecs=30

stopwaitsecs=30

autorestart=true

user=prefect

group=prefect

stdout_logfile=/mnt/data/prefect/server/log/server.log

environment=PATH="/mnt/data/prefect/venv/bin:%(ENV_PATH)s"

command=prefect server start --use-volume --volume-path /mnt/data/prefect/server/pg_data

priority=400

stopsignal=INT

Is it stuck in scheduled or submitted?

Hold on I have to ask my coding buddy...

The specific issue though is that your Flow is running in 3.10 but the Dask workers are spinning up 3.8 I think. What image are you using for that?

Dunno, I do know there are a bunch of sus symlinks... still waiting on my coding buddy

Not sure I can answer that for you. It would be whatever you used for the Flow so that Flow, and cluster have same dependency versions