Bruno Murino

    Bruno Murino

    11 months ago
    Hi everyone — We’ve been running a local prefect server for a while now and I’m assessing whether we should move to prefect cloud or not — however I want to know how many tasks have run successfully, so that I can get an idea of cost. Is there any way to find that out? maybe an api query?
    Kevin Kho

    Kevin Kho

    11 months ago
    Hey @Bruno Murino, yes this would be a GraphQL API query in the interactive API…it would be painful to load all successful task runs though. Let me ask the team for ideas.
    But we only charge also for tasks longer than 1 second, and you get the 10000 free task runs per month in Prefect Cloud
    Bruno Murino

    Bruno Murino

    11 months ago
    thanks @Kevin Kho!
    yep I know — just wanted a real measure of our current usage so I could more accurately assess the cost for us
    Kevin Kho

    Kevin Kho

    11 months ago
    query {
      task_run_aggregate(where: {state: { _eq: "Success"}}){
          aggregate {
          	count
      	}
    	}
    }
    Bruno Murino

    Bruno Murino

    11 months ago
    this is very useful! Is there a way to group by month? Sorry, I’m really not familiar with graphql
    Kevin Kho

    Kevin Kho

    11 months ago
    Let me try one sec
    You need to do something like this and loop through:
    query {
      task_run_aggregate(where: 
        {state: { _eq: "Success"},
        start_time: {_gt: "2021-09-01T00:00:00"}
        end_time: {_lt: "2021-09-30T00:00:00"}}){
          aggregate {
          	count
      	}
    	}
    }
    Bruno Murino

    Bruno Murino

    11 months ago
    This is very useful, thanks! Also, would it be possible to filter out the tasks that run under 1 sec? Sorry to keep asking more, but I really appreciate the help!
    Kevin Kho

    Kevin Kho

    11 months ago
    This I am not seeing a way because the duration is calculated by the UI and not stored
    Bruno Murino

    Bruno Murino

    11 months ago
    @Kevin Kho so how does prefect workout cost? I’m keen on doing the same calculation as prefect, regardless of what it is
    Kevin Kho

    Kevin Kho

    11 months ago
    Let me see
    Dylan

    Dylan

    11 months ago
    Hey @Bruno Murino we take the difference between the
    start_time
    and
    end_time
    fields on-the-fly in the UI. That’s what you’d need to calculate duration for your task runs
    Bruno Murino

    Bruno Murino

    11 months ago
    is that how prefect works out the count of tasks run for the purposes of billing?
    Dylan

    Dylan

    11 months ago
    Essentially, yes. Cloud has a process that does this calculation and writes it to a separate table that’s easy to
    SUM
    But the calculation is identical
    “The count of task runs where state is
    Success
    and the difference between
    start_time
    and
    end_time
    is greater than one second” is the official calculation. If there is no start time or end time it means that we don’t charge for the task
    @Marvin open “Tool to estimate Cloud cost” on server
    Marvin

    Marvin

    11 months ago
    Bruno Murino

    Bruno Murino

    11 months ago
    This is great! Makes sense — additionally, is there anything we can get from the underlying postgres database?
    I mean, instead of a graphql query, is there something we can query on the underlying postgres database used by prefect server?
    Dylan

    Dylan

    11 months ago
    Absolutely
    Bruno Murino

    Bruno Murino

    11 months ago
    trying that now
    I think that will work
    Dylan

    Dylan

    11 months ago
    It would be something like:
    SELECT start_time, end_time, TIMESTAMP_DIFF(start_time, end_time) FROM task_run WHERE state = 'Success' AND start_time IS NOT NULL AND end_time IS NOT NULL
    (my sql is BigQuery flavored these days but I think that’s the general gist)
    Bruno Murino

    Bruno Murino

    11 months ago
    this is the full query I ran on Postgres:
    select
           date_trunc('month', start_time) as month,
           extract(epoch from end_time-start_time) >= 1 as count_towards_quota,
           count(*)
    from public.task_run
    where true
        and state = 'Success'
        and start_time is not null
        and end_time is not null
        and extract(epoch from end_time-start_time) >= 1
    group by 1,2
    order by date_trunc('month', start_time);
    Dylan

    Dylan

    11 months ago
    That looks right to me!
    Bruno Murino

    Bruno Murino

    11 months ago
    sweet, happy with the numbers I see — now I’ll dig into the difference between prefect server and prefect cloud