Michael Bell
01/28/2022, 10:02 PMError uploading to S3: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
. If I run the flow w/out the DaskExecutor, it works just fine. I have previously tested using dask_cloudprovider.aws.FargateCluster
directly within a task to create a cluster, send a dask array calculation to the cluster, and tear down all within one Prefect task and that worked fine as well. Any thoughts as to where the permissions error might be coming from? The execution role I specify in the cluster_kwargs
has s3:* permissions.
import os
import sys
from typing import List
sys.path.append('.')
import prefect
from prefect import task, Flow
from prefect.storage.s3 import S3
from prefect.run_configs.ecs import ECSRun
from prefect.executors.dask import DaskExecutor
from dask_cloudprovider.aws import FargateCluster
from config.build import ECR_IMAGE
@task(log_stdout=True)
def prefect_task(n: int) -> int:
print(f'Running task w/ input {n}')
time.sleep(2)
return n * 10
@task(log_stdout=True)
def reduce_task(ns: List[int]) -> int:
print(f'Running reduction task')
return sum(ns)
schedule = None
account = os.environ.get("AWSENV")
with Flow("dask_poc", storage=S3(bucket=f'prefect-{account}-us-east-1')) as flow:
results = prefect_task.map([1,2,3,4,5])
reduced_result = reduce_task(results)
flow.run_config = ECSRun(
image=ECR_IMAGE,
execution_role_arn=...,
task_role_arn=...,
run_task_kwargs={
"cluster": "PrefectCluster",
},
labels=[account]
)
flow.executor = DaskExecutor(
cluster_class="dask_cloudprovider.aws.FargateCluster",
cluster_kwargs={
"image": ECR_IMAGE,
"n_workers": 5,
"cluster_arn": ...,
"execution_role_arn": ...,
"vpc": ...,
"subnets": [..., ],
"security_groups": [..., ],
},
)
Anna Geller
task_role_arn
. Execution role is mainly to pull the image and create log group and write logs.Michael Bell
01/28/2022, 10:33 PMbotocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameterException) when calling the RunTask operation: TaskDefinition is inactive
Any idea what's going on here?Anna Geller