Robin
09/29/2020, 9:36 AMINFO - prefect.Docker | Pushing image to the registry...
) I get a shutil PermissionError: [WinError 5] Access is denied
.
Has anybody had the same issues?
We are using Python 3.7.9 due to some issues with Python 3.8 of another third party package ...nicholas
Robin
09/29/2020, 2:04 PMnicholas
Read-only
attribute from the offending folder (I'd guess a Docker-specific folder or a temp folder at the C:
level) through the windows GUI folder properties
2. Using os.chmod
in your script to remove read-only permissions from that folderRobin
09/29/2020, 2:22 PMprefect.utilities.exceptions.ClientError: [{'path': ['create_flow_from_compressed_string'], 'message': '[{\'extensions\': {\'path\': \'$\', \'code\': \'invalid-json\'}, \'message\': "Error in $: Failed reading: takeWhile1. Expecting object value at \'Infinity,Infinity],tags:[],type:prefect.core.parameter.Parameter,outputs:typing.Any\'"}]', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
nicholas
Robin
09/29/2020, 3:29 PMimport sys
from prefect import Parameter
from prefect import Flow, task, unmapped
from prefect.environments import DaskKubernetesEnvironment
from prefect.environments.storage import Docker
from prefect.tasks.aws.secrets_manager import AWSSecretsManager
from prefect.engine.executors import DaskExecutor
import accure_<http://analytics.io|analytics.io>.snowflake_utils as snowflake_io
import prefect
from os import path
from datetime import datetime
@task
def hello_world_task():
print("hello world")
with Flow("e3dc-test-flow") as flow:
hello_world_task()
flow.environment = DaskKubernetesEnvironment(
min_workers=1, max_workers=3, labels=["k8s"]
)
# register flow on AWS ECR
flow.storage = Docker(
python_dependencies=[
"numpy",
"pandas",
# "snowflake-connector-python[pandas]",
# "snowflake-sqlalchemy==1.2.3",
# "tqdm",
# "boto3",
# '"docutils<0.16>=0.10"',
],
registry_url="<http://someidcr.eu-central-1.amazonaws.com|someidcr.eu-central-1.amazonaws.com>",
image_name="flow",
image_tag="beta_"
+ datetime.now().strftime("%Y%m%d_%H%M%S"), # Unique tag avoids AWS caching
)
flow.register(project_name="eks_test_01")
nicholas
Robin
09/29/2020, 4:09 PMnicholas
Robin
09/29/2020, 4:22 PMnicholas
Robin
09/29/2020, 7:38 PMfrom prefect import Flow, task
from prefect.environments import DaskKubernetesEnvironment
from prefect.environments.storage import Docker
from os import path
from datetime import datetime
@task
def hello_world_task():
print("hello world")
with Flow("test-flow") as flow:
hello_world_task()
flow.environment = DaskKubernetesEnvironment(
min_workers=1, max_workers=3, labels=["k8s"]
)
module_dir = path.dirname(path.dirname(path.abspath(__file__)))
flow.storage = Docker(
registry_url=".<http://dkr.ecr.eu-central-1.amazonaws.com|dkr.ecr.eu-central-1.amazonaws.com>",
image_name="flow",
image_tag="beta_"
+ datetime.now().strftime("%Y%m%d_%H%M%S"), # Unique tag avoids AWS caching
files={
module_dir: "/modules/accure_analytics \n RUN pip install /modules/accure_analytics"
},
)
flow.register(project_name="eks_test_01")
Hence, shutil.copytree seems to cause the problems in combination with windows 10 🤔nicholas
Robin
09/29/2020, 9:35 PMnicholas
Robin
09/29/2020, 9:57 PMnicholas
Robin
09/29/2020, 10:21 PMnicholas
Robin
09/30/2020, 8:01 AMsrc=src, dst=full_fname, symlinks=False, ignore=None
to
src=src, dst=full_fname, symlinks=False, ignore=None, copy_function=shutil.copyfile
See related pull requestnicholas
Robin
09/30/2020, 3:50 PM