Nataliia Sherstneva
03/16/2023, 3:24 PMdef upload_dir_to_s3(local_model_dir, logger, target_model_dir=None):
s3_bucket = S3Bucket.load(BLOCK_NAME_IRS_TRAIN_DATA_S3)
if not target_model_dir:
target_model_dir = local_model_dir
try:
s3_bucket.upload_from_folder(from_folder=local_model_dir, to_folder=f"{target_model_dir}")
<http://logger.info|logger.info>(f"Success! Directory '{local_model_dir}' was uploaded to '{s3_bucket.bucket_name}' Bucket.")
except Exception:
raise IOError(f"Saving to '{s3_bucket.bucket_name}' Bucket Failed!")
Then I download this model from S3 Bucket, however not within Prefect Workflow, but separately with boto3.client
. Locally it works, but in K8s Cluster I get an Forbidden Error.
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
Could you please tell me, what the problem might be? Should I save or read the model in another way?