Marcin Grzybowski
08/16/2022, 11:26 AMfor x in range(10):
snowflake_multiquery.submit(["select count(*) from (select seq4() from TABLE(GENERATOR(ROWCOUNT => 1000000000000)) )"], snowflake_credentials, as_transaction=transaction)
with connection.cursor(cursor_type) as cursor:
results = []
for query in queries:
response = cursor.execute_async(query, params=params)
query_id = response["queryId"]
while connection.is_still_running(
connection.get_query_status_throw_if_error(query_id)
):
await asyncio.sleep(0.05)
cursor.get_results_from_sfqid(query_id)
result = cursor.fetchall()
results.append(result)
There is a
`asyncio.sleep(0.05)`
and I believe that when processor is a bit busy, those "mini sleeps" (0.05s) are not really happening, and other threads cannot start...
When I change 0.05 to for example 1s, I am able to run 10 queries concurrentlyBianca Hoch
08/16/2022, 8:58 PMAnd only 4 or 5 or 6 are picked, the rest queries run when some previous query finishes.At first glance, this sounds like maybe a concurrency limit issue. Have you set a limit for the flows/tasks? https://docs-v1.prefect.io/orchestration/flow-runs/concurrency-limits.html#flow-run-limits
Marcin Grzybowski
08/17/2022, 7:14 AM