Hi I have some questions related to disk space usage on pref Prefect Community #ask-community

Hi I have some questions related to disk space usa...

Deceivious

06/08/2023, 8:22 AM

Hi I have some questions related to disk space usage on prefect database. Mostly related to how block documents are stored. Details will be added soon in the thread.

Deceivious

06/08/2023, 8:25 AM

Ive noticed a significant disk usage increase in prefect database. I looked into the usage and found that

block_document

table is growing at an alarming rate. ~1-1.2 GB perday

Deceivious

06/08/2023, 8:26 AM

Query I ran.

Copy code

select 
	bt."name" as block_type ,
	extract(month from bd.created) as month, 
	extract(day from bd.created) as day, count(1) 
from block_document bd , block_type bt  
where 
	bd.block_type_id  =bt.id 
group by 
	bt."name" ,
	extract(month from bd.created), 
	extract(day from bd.created)
order by block_type,month desc,day desc

Result.

Copy code

Azure	6	8	102171
Azure	6	7	252417
Azure	6	6	247796
Azure	6	5	251646
Azure	6	4	252944
Azure	6	3	252332
Azure	6	2	252440
Azure	6	1	253898
Azure	5	31	254765
Azure	5	30	254290
Azure	5	29	254048
Azure	5	28	254570
Azure	5	27	254046
Azure	5	26	254614
Azure	5	25	254965
Azure	5	24	253799
Azure	5	23	248840
Azure	5	22	247909
Azure	5	21	254712
Azure	5	20	255658
Azure	5	19	246496
Azure	5	18	252260
Azure	5	17	252821
Azure	5	16	252160
Azure	5	15	251961
Azure	5	14	252362
Azure	5	13	253300
Azure	5	12	252055
Azure	5	11	245653
Azure	5	10	251813
Azure	5	9	251273
Azure	5	8	231937
Azure	5	7	226121
Azure	5	6	221572
Azure	5	5	59424
Date Time	5	12	1
Date Time	5	11	1
Kubernetes Job	6	6	14
Kubernetes Job	6	5	14
Kubernetes Job	6	1	12
Kubernetes Job	5	31	1
Kubernetes Job	5	30	1
Kubernetes Job	5	26	1
Kubernetes Job	5	25	3
Kubernetes Job	5	24	11
Kubernetes Job	5	23	22
Kubernetes Job	5	19	23
Kubernetes Job	5	16	1
Kubernetes Job	5	15	1
Kubernetes Job	5	10	1
Kubernetes Job	5	8	14
Kubernetes Job	5	5	11
Local File System	6	8	942
Local File System	6	7	2176
Local File System	6	6	2183
Local File System	6	5	2198
Local File System	6	4	2198
Local File System	6	3	2191
Local File System	6	2	2214
Local File System	6	1	2064
Local File System	5	31	1912
Local File System	5	30	1901
Local File System	5	29	1898
Local File System	5	28	1893
Local File System	5	27	1909
Local File System	5	26	1911
Local File System	5	25	1899
Local File System	5	24	1895
Local File System	5	23	1900
Local File System	5	22	1905
Local File System	5	21	1903
Local File System	5	20	1904
Local File System	5	19	1191
Local File System	5	18	304
Local File System	5	17	304
Local File System	5	16	300
Local File System	5	15	301
Local File System	5	14	302
Local File System	5	13	304
Local File System	5	12	399
Local File System	5	11	294
Local File System	5	10	292
Local File System	5	9	297
Local File System	5	8	279
Local File System	5	7	253
Local File System	5	6	255
Local File System	5	5	79

Deceivious

06/08/2023, 8:28 AM

The Azure blocks are being used for storage/ caching of tasks.

Deceivious

06/08/2023, 8:31 AM

I have created a storage block using the UI.

Deceivious

06/08/2023, 8:31 AM

Deceivious

06/08/2023, 8:32 AM

But I see new block documents being generated every day.

Deceivious

06/08/2023, 8:33 AM

Does prefect copy over the base block I have supplied and create a new block document for each cached task?

Deceivious

06/08/2023, 8:34 AM

Looking for help / responses from devs @ prefect and ppl who know internal workings of Prefect. Or ppl who can comment on what I am doing wrong.

Nate

06/08/2023, 1:58 PM

hey @Deceivious - how are you specifying your result storage with Azure? are you loading the same block each time and passing it into task decorators? or doing something else there's an idea of anonymous blocks that are registered when, for example, you instantiate an infra block and pass it to a deployment without explicitly calling

save

on it

Deceivious

06/08/2023, 1:59 PM

Yes I am using the

.load()

method.

Nate

06/08/2023, 1:59 PM

hmm

Deceivious

06/08/2023, 2:00 PM

Ill wirte a bit more detail on my workflow.

Deceivious

06/08/2023, 2:02 PM

1. A flow sets up all the required blocks and ensure they are present with correct naming. Basically this is our means of versioning the blocks. Blocks are versioned along with the flow version. 2. All the other flow that uses the blocks have tasks decorated as

@task(.....result_starage=Azure.load(<NAMEHERE>).....)

Deceivious

06/08/2023, 2:03 PM

I do call .save() on flow #1 .

Deceivious

06/08/2023, 2:12 PM

Deceivious

06/08/2023, 2:13 PM

@Nate We use

cached_task

as decorator - just tasks that are primed with standard parameters.

alex

06/08/2023, 2:17 PM

@Deceivious Can you check to see how many block documents in your DB have

is_anonymous

set to

True

? When Prefect creates block documents on the user’s behalf, we set that equal to

True

, so if you have a large number of anonymous blocks, something within Prefect is likely causing your high number of block documents.

Deceivious

06/08/2023, 2:19 PM

I think most of them is True

Deceivious

06/08/2023, 2:20 PM

Deceivious

06/08/2023, 2:20 PM

[v] is True.

Deceivious

06/08/2023, 2:22 PM

@alex

Nate

06/08/2023, 2:23 PM

how does the number of task runs compare to that anonymous block number (or total block number, since anon is dominating the number)? have you noticed that they grow in proportion?

Deceivious

06/08/2023, 2:28 PM

Not all of our tasks are cached so a direct count wouldnt give the right data i guess.

Deceivious

06/08/2023, 2:32 PM

---7338392

Deceivious

06/08/2023, 2:33 PM

@Nate thats the number of tasks that I know are cached.

Deceivious

06/08/2023, 2:37 PM

Also I noticed, we are passing the Azure.load(NAMEHERE) into a

wiih_options

. Would using

Task.with_options

cause this issue?

alex

06/08/2023, 2:49 PM

If using

result_storage

is causing the overzealous block saving (which is my hunch) then we’ll want to check if the block that you’re passing has

_block_document_id

set. If it doesn’t then that would prompt the results functionality to save a new block document.

Deceivious

06/08/2023, 2:52 PM

I am passing the result of

Azure.load("name")

into the

result_storage

parameter. Would there be any cases where

Azure.load

would return object with no

_block_document_id

attribute?

Deceivious

06/08/2023, 2:52 PM

https://prefect-community.slack.com/files/U03RN8W7DPU/F05B6J0QEB1/image.png▾

Deceivious

06/08/2023, 2:53 PM

Here\s the actual code.

alex

06/08/2023, 2:54 PM

Called

.load

should always set

_block_document_id

on the returned object, but it might be getting taken off somewhere. This is worth checking because the presence of

_block_document_id

determines whether or not a new block document is created

🙌 1

Deceivious

06/09/2023, 9:23 AM

@alex and @Nate Ive verified that we are using .load on all instances of our code. I will test this out with a minimal code and if I see the issues in that as well, ill share the code here.

Deceivious

06/09/2023, 11:03 AM

Heres the minimum code.

Copy code

import datetime

import prefect
from prefect import flow,task
from prefect.filesystems import Azure
from prefect.serializers import JSONSerializer
from prefect.tasks import task_input_hash
from datetime import timedelta

def get_cache_parameters():
    cache_params = {
        "cache_key_fn": task_input_hash,
        "persist_result": True,
        "result_storage": Azure.load("test-block"),
        "result_serializer": JSONSerializer(jsonlib="json"),
        "cache_expiration":timedelta(minutes=1)
    }
    return cache_params

@task
def mock_test(i:int,now):
    prefect.get_run_logger().info(f"{i}{now}")


@flow
def flow():
    now= datetime.datetime.now()
    for i in range(10):
        mock_test.with_options(name=f"Name_{i}",**get_cache_parameters())(i,now)


flow()

Deceivious

06/09/2023, 11:04 AM

test-block

Azure block must be pre-created. Everytime the code is run - a new block gets created in the

block_document

table.

Deceivious

06/09/2023, 11:04 AM

Prefect version 2.10.12

Deceivious

06/14/2023, 8:39 AM

Hi @alex and @Nate is this expected? or should I create an github issue regarding this?

Deceivious

06/14/2023, 8:42 AM

Ive also verified that

_block_document_id

is present.

alex

06/14/2023, 1:44 PM

Yes, please create an issue and we’ll look into it further.

5 Views

Open in Slack

Previous Next