I have a Prefect flow that fails when it is unpickled, the error message is related to me decrypting variables in pydantic's Settings().. is there any good way to move forward and debug pickling issues like this?
k
Kevin Kho
05/18/2022, 10:45 PM
Is that Prefect 1 or Prefect 2?
p
Pasha Kravtsov
05/18/2022, 10:47 PM
1.1.0
k
Kevin Kho
05/18/2022, 10:48 PM
What is the exact error?
p
Pasha Kravtsov
05/18/2022, 10:48 PM
I'm using something similar to this:
Copy code
from pydantic import BaseSettings
from pydantic_kms_secrets import KMSSecretStr, decrypt_kms_secrets
class Settings(BaseSettings):
env: str
secrets_kms_key_id: str # This model attribute must exist to decrypt secrets
kms_secret_1: KMSSecretStr
kms_secret_2: KMSSecretStr
class Config:
env_file = ".env"
settings = decrypt_kms_secrets(Settings())
which works without issue outside of Prefect in my main application
Pasha Kravtsov
05/18/2022, 10:48 PM
Copy code
Failed to load and execute flow run: FlowStorageError('An error occurred while unpickling the flow:\n ClientError("An error occurred (ValidationException) when calling the Decrypt operation: 1 validation error detected: Value at \'ciphertextBlob\' failed to satisfy constraint: Member must have length greater than or equal to 1")')
k
Kevin Kho
05/18/2022, 11:01 PM
Maybe you can store as a script to bypass the serialization?
p
Pasha Kravtsov
05/18/2022, 11:09 PM
is there some documentation that I could take a look at please for that?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.