Kevin Mullins
02/15/2022, 8:34 PMPrefect Core encourages the use of small "helper" tasks. Won't those these inflate my billable usage?
Tasks that run for less than one second do not count as billable usage, so you can continue to design workflows that follow best practice and include as many helper tasks as you need.
Now that I’m using Prefect more and watching my task usage. I’ve noticed that both Secrets (EnvVarSecret in this case) which behind the scenes are a task and very simple tasks that just wrap parameters/task results into another structure vary quite a bit in runtime and often exceed the <1 second rule and thus will get billed. This is especially worrisome to me for secrets these tasks are necessary for securely handling things and some of our flows have quite a few secrets.
I’m not sure what a solution would be, but wanted to see if anybody has noticed this and if there could be anything done to ensure these simple tasks don’t inflate actual bills.Kevin Mullins
02/15/2022, 8:34 PM@task
def build_create_target_schema_queries(target_config: Dict[str, Any]) -> str:
return f"CREATE SCHEMA IF NOT EXISTS {target_config['postgres_schema']}"
Anna Geller
Kevin Mullins
02/15/2022, 9:01 PMKevin Mullins
02/15/2022, 9:06 PMAnna Geller
Kevin Mullins
02/15/2022, 9:07 PMAnna Geller
Kevin Mullins
02/15/2022, 9:10 PMKevin Mullins
02/15/2022, 9:15 PMAnna Geller
Kevin Mullins
02/15/2022, 9:24 PMKevin Mullins
02/15/2022, 9:26 PMKevin Mullins
02/15/2022, 9:29 PMKevin Mullins
02/15/2022, 9:30 PM<1 second
like it used to. I could be crazyKevin Mullins
02/15/2022, 9:41 PMPrefectSecret
tasks that show 1 second or more do increment the task runs.Chris Reuter
02/15/2022, 9:59 PMKevin Mullins
02/15/2022, 10:04 PMKevin Mullins
02/15/2022, 10:09 PMKevin Mullins
02/15/2022, 10:12 PMKevin Kho
Kevin Kho
@task
def some_task():
something = Secret().get()
or
PrefectSecret.run()
and by running it inside, it won’t be a task.
Same thing with functions
def some_func():
return 1
@task
def some_task():
return 1 + some_func()
will work.
To your comment on needing to reshape tasks to confirm to the task library or other code, you can also subclass if it helps
class MyPostgres(PostgresTask):
def run(takes_in_some strings):
query = do_the_formatting_here()
super().run(query)
But really it’s a lot more work that it may or may not be worth it. If your Flows at a scale where this is really feeling like it’s a large dent in your bill, we’d be happy to discuss discounts on the sales side.
We understand the sentiment, but can’t really make it a priority given current Orion development and eventual migration. Also, the fact that pricing will change and you actually will even get a lot more for free in Orion.Kevin Mullins
02/16/2022, 5:10 AM