hullo! i'm attempting to build a flow that kicks o...
# prefect-community
j
hullo! i'm attempting to build a flow that kicks off subflows asynchronously in Prefect 2.0, and seeing an error that i'm a bit stumped on:
Copy code
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
details in the thread!
the system i'm trying to create does the following: • i have a
parent_flow
• the
parent_flow
kicks of
subflow_1
,
subflow_2
asynchronously •
subflow_1
has no dependencies, but
subflow_2
depends on the result of
subflow_1
, so it triggers
subflow_1
as well • ** i don't want to have to topologically sort all my subflows to figure out the optimal execution order, i'd rather rely on prefect caching to minimize work
here's code to reproduce the issue (there's stuff that is duplicated e.g. the logic between subflows, but in reality each subflow might do some different stuff)
here's the full error message
fwiw: the subflow stuff works fine, but the second i introduce `task`s into the subflows, it falls apart
bumping this error from yesterday!
1
r
Hi Jai, Setting Prefect to use Postgres insytead of Sqlite should solve the issue you are seeing: https://docs.prefect.io/concepts/database/#configuring-a-postgresql-database
j
In-memory databases are only supported by Prefect Orion for testing purposes and are not compatible with multiprocessing.
derp i completely missed this! i'll try using postgres and update if i see it working 🙂
r
I tested your code and encountered the same issue you did, and switching to Postgres solved it on my machine. So, hopefully it solves it for you as well 😄
z
We should be able to improve this as well! https://github.com/PrefectHQ/prefect/pull/6526
🙌 1
👍 2
j
amazing 🙂. fwiw using postgres also fixed the issue for me!
👍 1
🙌 1
z
It looks like when I try to run your example there are some other issues that crop up with SQLite despite my change 🙂
j
got it! i think we can definitely get by for the time being with a postgres instance running in a container locally so it should be fine. one thing that i've noticed though is that tags don't appear to be populating when using postgres locally? is that a known issue?
Copy code
@task
def test_task():
    print("hi")

@flow
def test_flow():
    with tags("testing"):
        test_task()

if __name__ == '__main__':
    test_flow()
isn't showing tags in the orion ui when using postgres, but does show them when using sqlite
z
That definitely sounds like a bug (and a weird one at that). Can you open an issue on GitHub? Do the tags exist in the database?
j
lemme check the db but yup happy to open an issue!
doesnt look like the tags exist in the db either
ah, this doesnt look like a postgres vs sqlite issue, i think its an async issue. i'll create a reproducible example and create the issue!
closing the loop here: this isn't an issue. i realized i'm doing the
with tags(...):
block in the wrong place, and it should be around the
asyncio.gather
call instead:
Copy code
# not this
subflows = []
with tags("some_tag"):
    subflows.append(my_subflow())

asyncio.gather(*subflows)

# do this instead

subflows = []
subflows.append(my_subflow())

with tags("some_tag"):
    asyncio.gather(*subflows)
thank you for addressing my original issue though! and i'll keep an eye out for any release notes regarding using sqlite for async flows/tasks locally 🙂