Matthew Scanlon
01/12/2023, 5:09 PMmy_s3_bucket_block = S3Bucket.load("my-s3-bucket", validate=False)
but upon trying to execute this code, i am seeing Block.load() got an unexpected keyword argument 'validate'
Kalise Richmond
01/12/2023, 5:22 PMMatthew Scanlon
01/12/2023, 5:30 PM# Bypass Pydantic validation to allow your local Block class to load the old block version
my_s3_bucket_block = S3Bucket.load("my-s3-bucket", validate=False)
# Set the new field to an appropriate value
my_s3_bucket_block.bucket_path = "my-default-bucket-path"
# Overwrite the old block values and update the expected fields on the block
my_s3_bucket_block.save("my-s3-bucket", overwrite=True)
Nate
01/12/2023, 5:42 PMvalidate
kwarg was added to Block.load
quite recently, what version of prefect are you on?Matthew Scanlon
01/12/2023, 5:56 PMclass SampleBlock(Block):
field1: str
field2: str
SampleBlock.load('test-block', validate=False)
This reproduces the error
Where test-block was created via the prefect cloud ui and SampleBlock was registered via the CLINate
01/12/2023, 6:06 PMprefect version
from where you encounter the error?Matthew Scanlon
01/12/2023, 9:32 PMvalidate=False
which lends itself to a plethora of other issues.
I would imagine there has to be some way in which you guys are managing changes made to your own "official" blocksNate
01/12/2023, 10:03 PMIt seems like the UI does not update block fields given the underlying block is re-registered (updated) with different fields.one thing that might be helpful to note is that when you call
prefect block register -m some_module
, you're not registering a single block instance, you're registering a type of block (whatever class Foo(Block):
lives in file / module you pass to the block register
command
the purpose of the validate
kwarg accepted by Block.load
is just to migrate existing block instances from one version of a block type to another, so for example if I have
from prefect.blocks.core import Block
class Foo(Block):
x: int = 1
foo = Foo()
foo.save("testfoo")
then, as you'd expect, we see (first image)
then, simulating a change to the block implementation:
from prefect.blocks.core import Block
class Foo(Block):
x: int = 1
y: int = 2
# pydantic validation would fail, bc block instance in
# orion database doesn't know about new local definition of class Foo
# so we use validate=False
foo = Foo.load("testfoo", validate=False)
assert foo == Foo(x=1, y=2) # passes
# saving with overwrite will re-register the block type for you,
# if and only if schema has changed, based on the schema
# of the updated local class Foo (e.g. adding y)
foo.save("testfoo", overwrite=True)
and we should now see (second image)Matthew Scanlon
01/13/2023, 6:02 PMtestfoo
with x = 1, and the do prefect blocks register --file my_block_file
on the second instance of class Foo, i am not going to see Y show up in the UI for testfoo. If i try to create a new Foo block then yes, that field is available, but testFoo only has field 1.
As a cloud user managing many blocks across many users, I think it is quiet dangerous to have programmers interacting with blocks programmatically. It makes much more sense from my perspective that if i register a block with a schema change, and then go to an old version of that block, the new schema is present and i can update the fields accordinglyvalidate
kwarg accepted by Block.load
is just to migrate existing block instances from one version of a block type to another") This makes it out to sound like blocks are tied to a specific version of a block instance, which they are not since if i do not update the fields for a block instance after overwriting the block types schema, the pydantic model complainsNate
01/13/2023, 6:13 PMThis makes it out to sound like blocks are tied to a specific version of a block instance, which they are notBlock documents are in fact tied to a specific version of the block schema, which means that in principle its possible to have block documents associated with different schemas for a given block type. This is why the documentation example shows the migration of a single block document it sounds like you'd like to see a cascade, where if you reregister a block type, all block documents of that type would be conformed to the new schema, but that is not currently our intended behavior
I think it is quiet dangerous to have programmers interacting with blocks programmaticallyI think of blocks as easiest to interact with programmatically, but I am of course biased
validate=False
would be a part of regular development, its just an escape hatch for when you'd want to avoid recreating an existing block instance after re-implementing the class
If you'd like to see different behavior, I'd be happy to bring that back to the team and discuss
and if for some reason you anticipate having some unknown amount of fields on a block, then I would suggest just putting a dict
field on the block where you can happily add / remove keys without worrying about block schema changesMatthew Scanlon
01/13/2023, 7:21 PMBlock documents are in fact tied to a specific version of the block schemaI see, i think this finally clicked for me. So in your example above. TestFoo gets saved with schema 1, but then after re-registering Foo, on Foo.load('TestFoo'), the engine is going to use FooV2 but TestFoo was written with FooV1, so i need to re-write fooV2 in order for the validator to not complain.
I think of blocks as easiest to interact with programmatically, but I am of course biasedI think it is only in the enterprise version of prefect cloud that you can have role based permissions, correct? We have been discussing internally whether upgrading makes sense as right now, it is very easy for anyone on our prefect account to do something like
Secret.load('SomeSecretBlock).get_secret_value()
and see what the token associated with that secret is. This was a pain point in prefect 1 with being able to query the client for secret values. So the idea that we could have a block with an associated secret field, that a programmer can go in and change, and then i can not even log into the ui to see if the value is correct, just seems problematic to me. As i will have to write a script to log into prefect, load the secret, and programmatically compare it/ update it to ensure it is correct.
Specifically in my case, not everyone we have writing flows is a software engineer, so expecting them to not only learn how to write flows but then also maintain them programmatically i see as a pain point for our specific use cases. We as programmers are not perfect, mistakes get made all the time, i just think the UI lends it self to being less prone to errors.
its just an escape hatch for when you'd want to avoid recreating an existing block instance after re-implementing the classi agree, and i discussed this quite extensively with my colleagues, as i have taken a position quite similar to your own. The dict is a great idea for achieving this.
Nate
01/13/2023, 7:32 PMSo in your example above. TestFoo gets saved with schema 1, but then after re-registering Foo, on Foo.load('TestFoo'), the engine is going to use FooV2 but TestFoo was written with FooV1, so i need to re-write fooV2 in order for the validator to not complain.yep exactly!
validate=False
is just to tell pydantic to relax for a moment so we can we update TestFoo
from FooV1
to FooV2
This was a pain point in prefect 1 with being able to query the client for secret values. So the idea that we could have a block with an associated secret field, that a programmer can go in and change, and then i can not even log into the ui to see if the value is correct, just seems problematic to me. As i will have to write a script to log into prefect, load the secret, and programmatically compare it/ update it to ensure it is correct.this is an interesting point. I believe at the organization pricing tier you get normal RBAC with Owner, Collaborator and Read-Only (enterprise tier you also get to create custom RBAC roles). So if you have an organization, you should be able to to restrict certain people to Read-Only, which would prevent them from Create / Delete / Edit of block instances (including secrets)
Matthew Scanlon
01/17/2023, 3:48 PM