https://prefect.io
Join Slack
Hi everyone — We’ve been running a local prefect server for a while now and I’m assessing whether we...
# ask-community
b
Hi everyone — We’ve been running a local prefect server for a while now and I’m assessing whether we should move to prefect cloud or not — however I want to know how many tasks have run successfully, so that I can get an idea of cost. Is there any way to find that out? maybe an api query?
k
Hey @Bruno Murino, yes this would be a GraphQL API query in the interactive API…it would be painful to load all successful task runs though. Let me ask the team for ideas.
But we only charge also for tasks longer than 1 second, and you get the 10000 free task runs per month in Prefect Cloud
b
thanks @Kevin Kho!
yep I know — just wanted a real measure of our current usage so I could more accurately assess the cost for us
k
Copy code
query {
  task_run_aggregate(where: {state: { _eq: "Success"}}){
      aggregate {
      	count
  	}
	}
}
b
this is very useful! Is there a way to group by month? Sorry, I’m really not familiar with graphql
k
Let me try one sec
You need to do something like this and loop through:
Copy code
query {
  task_run_aggregate(where: 
    {state: { _eq: "Success"},
    start_time: {_gt: "2021-09-01T00:00:00"}
    end_time: {_lt: "2021-09-30T00:00:00"}}){
      aggregate {
      	count
  	}
	}
}
b
This is very useful, thanks! Also, would it be possible to filter out the tasks that run under 1 sec? Sorry to keep asking more, but I really appreciate the help!
k
This I am not seeing a way because the duration is calculated by the UI and not stored
b
@Kevin Kho so how does prefect workout cost? I’m keen on doing the same calculation as prefect, regardless of what it is
k
Let me see
d
Hey @Bruno Murino we take the difference between the 
start_time
and 
end_time
fields on-the-fly in the UI. That’s what you’d need to calculate duration for your task runs
b
is that how prefect works out the count of tasks run for the purposes of billing?
d
Essentially, yes. Cloud has a process that does this calculation and writes it to a separate table that’s easy to 
SUM
But the calculation is identical
“The count of task runs where state is 
Success
and the difference between 
start_time
and 
end_time
is greater than one second” is the official calculation. If there is no start time or end time it means that we don’t charge for the task
@Marvin open “Tool to estimate Cloud cost” on server
m
b
This is great! Makes sense — additionally, is there anything we can get from the underlying postgres database?
I mean, instead of a graphql query, is there something we can query on the underlying postgres database used by prefect server?
d
Absolutely
b
trying that now
I think that will work
d
It would be something like: 
SELECT start_time, end_time, TIMESTAMP_DIFF(start_time, end_time) FROM task_run WHERE state = 'Success' AND start_time IS NOT NULL AND end_time IS NOT NULL
(my sql is BigQuery flavored these days but I think that’s the general gist)
b
this is the full query I ran on Postgres:
Copy code
select
       date_trunc('month', start_time) as month,
       extract(epoch from end_time-start_time) >= 1 as count_towards_quota,
       count(*)
from public.task_run
where true
    and state = 'Success'
    and start_time is not null
    and end_time is not null
    and extract(epoch from end_time-start_time) >= 1
group by 1,2
order by date_trunc('month', start_time);
d
That looks right to me!
b
sweet, happy with the numbers I see — now I’ll dig into the difference between prefect server and prefect cloud
👍 1
d
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by