Hello i m stuck in the following issue setting up Prefect +

Question

Hello i m stuck in the following issue setting up Prefect + Dask cluster When the lfow is executed multiple tasks are run on the Dask Worker When it run i get an exception see thread

Massimiliano Fanciulli · Answer

This happens on a docker based architecture 1 docker for prefect agent + dask scheduler other for dask worker only

Massimiliano Fanciulli · Answer

I get the error on the worker nodes

Massimiliano Fanciulli · Answer

If i setup ec3 machines and run the same simple example i have no errors

Massimiliano Fanciulli · Answer

Any clue

Anna Geller · Answer

Could you move code blocks to the thread

Massimiliano Fanciulli · Answer

Sure

Massimiliano Fanciulli · Answer

this is the deployment file

Massimiliano Fanciulli · Answer

from prefect deployments import DeploymentSpec from prefect orion schemas schedules import IntervalSchedule from prefect flow runners import SubprocessFlowRunner from prefect blocks storage import S3StorageBlock from datetime import timedelta DeploymentSpec schedule=IntervalSchedule interval=timedelta minutes=1 flow location= info py flow storage=S3StorageBlock bucket= newpreproc test name= Processor

Massimiliano Fanciulli · Answer

and this is info py

Massimiliano Fanciulli · Answer

import datetime from datetime import timedelta from prefect task runners import DaskTaskRunner from prefect import flow task flow name= GW Score task runner=DaskTaskRunner address= localhost 8786 def calculate score for i in range 10 calculate score group i task def calculate score group index print SCORE FOR INDEX +index+ IS 100

Anna Geller · Answer

I meant moving your code block to this thread instead of having the code and exception tracebacks in your original message This helps us keep the main channel clean Could you move that part here Thanks so much in advance

Massimiliano Fanciulli · Answer

Exceptionfollows

Massimiliano Fanciulli · Answer

10 14 42 021 | ERROR | prefect orion Encountered exception in request Traceback most recent call last File usr local lib python3 8 dist packages sqlalchemy engine base py line 3280 in wrap pool connect return fn File usr local lib python3 8 dist packages sqlalchemy pool base py line 310 in connect return ConnectionFairy checkout self File usr local lib python3 8 dist packages sqlalchemy pool base py line 868 in checkout fairy = ConnectionRecord checkout pool File usr local lib python3 8 dist packages sqlalchemy pool base py line 476 in checkout rec = pool do get File usr local lib python3 8 dist packages sqlalchemy pool impl py line 256 in do get return self create connection File usr local lib python3 8 dist packages sqlalchemy pool base py line 256 in create connection return ConnectionRecord self File usr local lib python3 8 dist packages sqlalchemy pool base py line 371 in init self connect File usr local lib python3 8 dist packages sqlalchemy pool base py line 666 in connect pool logger debug Error on connect s e File usr local lib python3 8 dist packages sqlalchemy util langhelpers py line 70 in exit compat raise File usr local lib python3 8 dist packages sqlalchemy util compat py line 207 in raise raise exception File usr local lib python3 8 dist packages sqlalchemy pool base py line 661 in connect self dbapi connection = connection = pool invoke creator self File usr local lib python3 8 dist packages sqlalchemy engine create py line 590 in connect return dialect connect cargs cparams File usr local lib python3 8 dist packages sqlalchemy engine default py line 597 in connect return self dbapi connect cargs cparams File usr local lib python3 8 dist packages sqlalchemy dialects sqlite aiosqlite py line 292 in connect await only connection File usr local lib python3 8 dist packages sqlalchemy util concurrency py3k py line 68 in await only return current driver switch awaitable File usr local lib python3 8 dist packages sqlalchemy util concurrency py3k py line 121 in greenlet spawn value = await result File usr local lib python3 8 dist packages aiosqlite core py line 137 in connect self connection = await future File usr local lib python3 8 dist packages aiosqlite core py line 102 in run result = function File usr local lib python3 8 dist packages aiosqlite core py line 397 in connector return sqlite3 connect loc kwargs sqlite3 OperationalError unable to open database file The above exception was the direct cause of the following exception Traceback most recent call last File usr local lib python3 8 dist packages starlette middleware errors py line 162 in call await scope receive send File usr local lib python3 8 dist packages starlette exceptions py line 93 in call raise exc File usr local lib python3 8 dist packages starlette exceptions py line 82 in call await scope receive sender File usr local lib python3 8 dist packages fastapi middleware asyncexitstack py line 21 in call raise e File usr local lib python3 8 dist packages fastapi middleware asyncexitstack py line 18 in call await scope receive send File usr local lib python3 8 dist packages starlette routing py line 670 in call await route handle scope receive send File usr local lib python3 8 dist packages starlette routing py line 266 in handle await scope receive send File usr local lib python3 8 dist packages starlette routing py line 65 in app response = await func request File usr local lib python3 8 dist packages prefect orion utilities server py line 87 in handle response scoped depends response = await default handler request File usr local lib python3 8 dist packages fastapi routing py line 227 in app raw response = await run endpoint function File usr local lib python3 8 dist packages fastapi routing py line 160 in run endpoint function return await dependant call values File usr local lib python3 8 dist packages prefect orion api task runs py line 188 in set task run state orchestration result = await models task runs set task run state File usr local lib python3 8 dist packages prefect orion models task runs py line 279 in set task run state run = await models task runs read task run session=session task run id=task run id File usr local lib python3 8 dist packages prefect orion database dependencies py line 112 in async wrapper return await fn args kwargs File usr local lib python3 8 dist packages prefect orion models task runs py line 97 in read task run model = await session get db TaskRun task run id File usr local lib python3 8 dist packages sqlalchemy ext asyncio session py line 296 in get return await greenlet spawn File usr local lib python3 8 dist packages sqlalchemy util concurrency py3k py line 126 in greenlet spawn result = context throw sys exc info File usr local lib python3 8 dist packages sqlalchemy orm session py line 2789 in get return self get impl File usr local lib python3 8 dist packages sqlalchemy orm session py line 2896 in get impl return db load fn File usr local lib python3 8 dist packages sqlalchemy orm loading py line 530 in load on pk identity session execute File usr local lib python3 8 dist packages sqlalchemy orm session py line 1695 in execute conn = self connection for bind bind File usr local lib python3 8 dist packages sqlalchemy orm session py line 1536 in connection for bind return self transaction connection for bind File usr local lib python3 8 dist packages sqlalchemy orm session py line 747 in connection for bind conn = bind connect File usr local lib python3 8 dist packages sqlalchemy future engine py line 406 in connect return super Engine self connect File usr local lib python3 8 dist packages sqlalchemy engine base py line 3234 in connect return self connection cls self close with result=close with result File usr local lib python3 8 dist packages sqlalchemy engine base py line 96 in init else engine raw connection File usr local lib python3 8 dist packages sqlalchemy engine base py line 3313 in raw connection return self wrap pool connect self pool connect connection File usr local lib python3 8 dist packages sqlalchemy engine base py line 3283 in wrap pool connect Connection handle dbapi exception noconnection File usr local lib python3 8 dist packages sqlalchemy engine base py line 2117 in handle dbapi exception noconnection util raise File usr local lib python3 8 dist packages sqlalchemy util compat py line 207 in raise raise exception File usr local lib python3 8 dist packages sqlalchemy engine base py line 3280 in wrap pool connect return fn File usr local lib python3 8 dist packages sqlalchemy pool base py line 310 in connect return ConnectionFairy checkout self File usr local lib python3 8 dist packages sqlalchemy pool base py line 868 in checkout fairy = ConnectionRecord checkout pool File usr local lib python3 8 dist packages sqlalchemy pool base py line 476 in checkout rec = pool do get File usr local lib python3 8 dist packages sqlalchemy pool impl py line 256 in do get return self create connection File usr local lib python3 8 dist packages sqlalchemy pool base py line 256 in create connection return ConnectionRecord self File usr local lib python3 8 dist packages sqlalchemy pool base py line 371 in init self connect File usr local lib python3 8 dist packages sqlalchemy pool base py line 666 in connect pool logger debug Error on connect s e File usr local lib python3 8 dist packages sqlalchemy util langhelpers py line 70 in exit compat raise File usr local lib python3 8 dist packages sqlalchemy util compat py line 207 in raise raise exception File usr local lib python3 8 dist packages sqlalchemy pool base py line 661 in connect self dbapi connection = connection = pool invoke creator self File usr local lib python3 8 dist packages sqlalchemy engine create py line 590 in connect return dialect connect cargs cparams File usr local lib python3 8 dist packages sqlalchemy engine default py line 597 in connect return self dbapi connect cargs cparams File usr local lib python3 8 dist packages sqlalchemy dialects sqlite aiosqlite py line 292 in connect await only connection File usr local lib python3 8 dist packages sqlalchemy util concurrency py3k py line 68 in await only return current driver switch awaitable File usr local lib python3 8 dist packages sqlalchemy util concurrency py3k py line 121 in greenlet spawn value = await result File usr local lib python3 8 dist packages aiosqlite core py line 137 in connect self connection = await future File usr local lib python3 8 dist packages aiosqlite core py line 102 in run result = function File usr local lib python3 8 dist packages aiosqlite core py line 397 in connector return sqlite3 connect loc kwargs sqlalchemy exc OperationalError sqlite3 OperationalError unable to open database file Background on this error at 2022 06 12 10 14 42 045 distributed worker WARNING Compute Failed Key calculate score group 63e95d0e 764 3ba520ff831e460cb1a3d0d81044ea9b Function begin task run args kwargs task lt prefect tasks Task object at 0x7f8af03338b0 gt task run TaskRun id=UUID 3ba520ff 831e 460c b1a3 d0d81044ea9b name= calculate score group 63e95d0e 764 flow run id=UUID 14f976c7 545f 4837 a0c2 75d76c26f82c task key= 63e95d0ee90b584403f98497c3a130bf dynamic key= 764 cache key=None cache expiration=None task version=None empirical policy=TaskRunPolicy max retries=0 retry delay seconds=0 0 tags= state id=UUID c65a9c67 9641 49a3 80be fb6f90327637 task inputs= index state type=StateType PENDING run count=0 expected start time=datetime datetime 2022 6 12 10 14 41 573413 tzinfo=datetime timezone utc next scheduled start time=None start time=None end time=None total run time=datetime timedelta 0 estimated run time=datetime timedelta 0 estimated start time delta=datetime timedelta microseconds=295338 state=Pending message=None type=PENDING result=None task run id=3ba520ff 831e 460c b1a3 d0d81044ea9b parameters index 7 Exception OperationalError sqlite3 OperationalError unable to open database file

Anna Geller · Answer

Re your issue it seems like you are running Orion on a bind mount volume can you try running it in a virtual environment instead of Docker This would make debugging and permissioning easier

Anna Geller · Answer

Thanks for moving the traceback

Massimiliano Fanciulli · Answer

tried on a virtualized env Ec2 on Amazon AWS and had no issue

Massimiliano Fanciulli · Answer

i m wondering what sqllite file is searching on the dask worker

Massimiliano Fanciulli · Answer

maybe my docker image need some adjustment for this

Anna Geller · Answer

Exactly as I thought Docker makes things harder due to network I d recommend a virtual environment until we have some official dockerized recipe