https://prefect.io logo
Channels
Powered by
k

Kevin Mullins

02/15/2022, 8:34 PM
Not sure how to start, but I wanted to talk a bit about a concern I have with predictable task usage and task-based billing. When I originally started looking into the pricing, I had the concern most people would have about the need to use a lot of tasks in prefect (fetching secrets, unpacking parameters, working with output from other tasks, etc.). Then I came across the following documentation (https://www.prefect.io/pricing/) which helped alleviate some of these fears.
Copy code
Prefect Core encourages the use of small "helper" tasks. Won't those these inflate my billable usage?
Tasks that run for less than one second do not count as billable usage, so you can continue to design workflows that follow best practice and include as many helper tasks as you need.
Now that I’m using Prefect more and watching my task usage. I’ve noticed that both Secrets (EnvVarSecret in this case) which behind the scenes are a task and very simple tasks that just wrap parameters/task results into another structure vary quite a bit in runtime and often exceed the <1 second rule and thus will get billed. This is especially worrisome to me for secrets these tasks are necessary for securely handling things and some of our flows have quite a few secrets. I’m not sure what a solution would be, but wanted to see if anybody has noticed this and if there could be anything done to ensure these simple tasks don’t inflate actual bills.
Simple Task Code:
Copy code
@task
def build_create_target_schema_queries(target_config: Dict[str, Any]) -> str:
    return f"CREATE SCHEMA IF NOT EXISTS {target_config['postgres_schema']}"
a

Anna Geller

02/15/2022, 9:00 PM
What happens when you use PrefectSecret instead? Does it also take more than 1 second?
k

Kevin Mullins

02/15/2022, 9:01 PM
I haven’t used PrefectSecrets. I can experiment and switch them out and see if I see the same behavior.
👍 1
I’m still wondering about simple functions as well. I do have a lot of simple string/structure manipulating tasks that setup parameters going into other tasks. I might have to re-evaluate the design of these if the runtime varies so much.
a

Anna Geller

02/15/2022, 9:07 PM
I don't know how others approach this, but I would only use EnvVarSecret if I were on Server because Server doesn't have the Secret backend. With Prefect Cloud, you can safely use PrefectSecret and it shouldn't take longer than 1 second
k

Kevin Mullins

02/15/2022, 9:07 PM
We are using EnvVarSecrets as we are on K8S and our CI/CD pipelines deploy K8S secrets for the flows.
a

Anna Geller

02/15/2022, 9:09 PM
Sure, it's always useful to profile. I think the good direction would be to check for which tasks do you need observability and other Prefect features like caching, retries etc. Things like string manipulations can perhaps be implemented as "normal" non-task functions and just called in Prefect tasks
k

Kevin Mullins

02/15/2022, 9:10 PM
Part of it may be a limited understanding of how task arguments get resolved and bound. Most of the reasons I used a task is the input arguments to build say a string or simple dictionary is dynamically discovered and returned from a previous task. I’m not sure of another way to approach this.
And eventually the inputs may get passed to an open-source task. So I can’t control what the final task takes in. So my though was Task to discover sources --> map task to take source info and build input for say PostgresExecute or some other task --> map PostgresExecutable to execute them all. Since these go between tasks are just a simple dictionary lookup and string concat, I wouldn’t think they would take much execution time. I can’t really optimize those any more (this kind of conflicts with encouraging use of small “helper” tasks.
a

Anna Geller

02/15/2022, 9:23 PM
I see. We definitely encourage small tasks and you get automatic volume discounts as you run more tasks over time. Additionally, if you get to a really massive scale, you can talk to Sales and they can find a plan suitable to your needs - you can reach out to sales@prefect.io
k

Kevin Mullins

02/15/2022, 9:24 PM
If that is the only solution then I might recommend removing that verbiage from the FAQ page as it would be inaccurate.
And sorry, I’m not being combative all though it sometimes is hard to tell with text. But just trying to provide the perspective of a user and a real-world use-case.
I switched over to PrefectSecret and gave it a whirl. It looks like 3 out of 7 secrets of this run might be billable tasks:
I think. I believe the UI changed recently to have no duration instead of showing 
<1 second
like it used to. I could be crazy
Yea, I’ve run it more and counted the changes to Task Runs today and the secrets with no duration don’t increment the Task Runs, but 
PrefectSecret
tasks that show 1 second or more do increment the task runs.
c

Chris Reuter

02/15/2022, 9:59 PM
@Kevin Mullins we truly do appreciate your perspective that you're providing, feedback like this is invaluable. One thing I do want to flag (although it doesn't help you right this second) is that Orion can orchestrate code without the need for it to be in a task. Therefore, as long as the individual task granularity doesn't matter to you, you could fetch secrets outside of a task scheduled & orchestrated via Prefect. Want to give you this context so that you know where we're headed.
upvote 1
👍 1
k

Kevin Mullins

02/15/2022, 10:04 PM
@Chris Reuter - I’ve only looked at Orion briefly; however, I really like the direction the API is going. I have no need for these to actually be tasks. For the small helper functions I believe tasks are the only way to work with data from another task. So it forced my hand a bit (please correct me if I’m wrong). I’d prefer it if they weren’t all tasks actually.
I’m not sure how the secrets will change in Orion; however, current documented use of secrets pre-orion automatically runs a task which appears to sometimes run long and incur a billable task. This IMHO isn’t very user-friendly as using secrets could cause a variance in billing. From my perspective, I already use nested subflows to handle a fan-out/fan-in approach. Each sub-flow has to grab secrets again to be secure, With these two things (ultra-small tasks using task runs, and secrets using task runs), it is causing a lot of wasted task runs I wish I could avoid.
Thanks for all the discussion btw, I really appreciate the communication and transparency. Just trying to make sure Prefect continues to get better for everyone! Like I said, I can’t wait until Orion is production ready, it’s going to be great.
k

Kevin Kho

02/15/2022, 10:14 PM
I’ll talk to some team members and see if I can get a more concrete answer for you
🙌 2
Kevin, I chatted with the team, and like you mentioned, you can always do:
Copy code
@task
def some_task():
    something = Secret().get()
or
Copy code
PrefectSecret.run()
and by running it inside, it won’t be a task. Same thing with functions
Copy code
def some_func():
    return 1

@task
def some_task():
    return 1 + some_func()
will work. To your comment on needing to reshape tasks to confirm to the task library or other code, you can also subclass if it helps
Copy code
class MyPostgres(PostgresTask):
    def run(takes_in_some strings):
        query = do_the_formatting_here()
        super().run(query)
But really it’s a lot more work that it may or may not be worth it. If your Flows at a scale where this is really feeling like it’s a large dent in your bill, we’d be happy to discuss discounts on the sales side. We understand the sentiment, but can’t really make it a priority given current Orion development and eventual migration. Also, the fact that pricing will change and you actually will even get a lot more for free in Orion.
k

Kevin Mullins

02/16/2022, 5:10 AM
Thanks Kevin, those are some great ideas I hadn't thought about and I think will help. In my case I think some of the extra effort may be worth it. I'm not at a point it's causing a billing problem, and understand all the effort put towards Orion. I agree it will change thinfs. I am just trying to make sure I can understand how things will scale and ensure everything is sustainable. Thanks again to you and your team. Fantastic help as always.
3 Views
Powered by