MarcoM
11/29/2023, 10:18 AMMarvin
11/29/2023, 10:18 AMprefect block register -m prefect_aws
. However, I couldn't find a direct way to instantiate a block like AwsCredentials
without importing the class from prefect_aws
.
It's also recommended to ensure that the runtime environment is authenticated if the bucket is private.
For more detailed information, you might want to check the following resources:
- Prefect Storage Guide
- Moving Data with Prefect
- Block Based Deployments
Please note that this information might not fully answer your question. For more specific guidance, you may want to reach out to the Prefect community or support.Marvin
11/29/2023, 10:19 AMprefect block register -m prefect_aws
. However, the documentation does not provide a direct way to instantiate a block such as an AwsCredentials Block without importing the class AwsCredentials from prefect_aws.
The method you're currently using, _globals()<name=model.name, **model.params|class_name>
, is a common way to dynamically instantiate classes in Python when the class name is only known at runtime. However, this does indeed require that the class is imported into the script.
If you want to avoid having to import every type of block that you might use for different customers, you might consider creating a factory function that imports and instantiates the necessary block classes on demand. This would allow you to only import the classes that are actually used in a given script, rather than having to import all possible block classes upfront.
Here's a rough example of what this might look like:
def block_factory(class_name, name, params):
module = importlib.import_module(f"prefect_aws.{class_name}")
class_ = getattr(module, class_name)
return class_(name=name, **params)
You could then use this function to instantiate your blocks like so:
block = block_factory(model.class_name, model.name, model.params)
Please note that this is a general Python solution and might need to be adapted to your specific use case and the Prefect-AWS library. If Prefect-AWS provides a better way to handle this, it would be preferable to use that.