Does anyone have experience using map() to call a function taking a **kwargs parameter? I'm not sure if this is even a Prefect question or more of a Python question, but it's not obvious to me how I can get this to work, if it is at all possible.
Specifically, I am trying to call trigger_dbt_cli_command which is part of the prefect-dbt package, on prefect 2.0.3. I'm not sure what other information I should include, since the types of error (or error messages) I get varies depending on how I try to approach the problem. 😅
08/08/2022, 2:35 PM
Hi Oscar, can you tell me more about why you want to use
when working with prefect-dbt?
08/08/2022, 2:40 PM
Sure! We have a test phase where we make 4 separate dbt test calls:
dbt test --select tag:int_test
dbt test --select tag:curated_test
dbt test --select tag:data_quantity
dbt test --select tag:data_quality
We probably could tag our tests in such a way in dbt that we can run them all with a single command, but separating them helps us categorize the log outputs by the type of test (quality, quantity, etc). Since these are mostly fairly simple tests we would prefer to run them in parallell.
If there is no other way to solve this, I'm okay just updating our dbt tag structure and calling dbt with --threads 4 (or 8 ), but that does fog up the log output.