https://prefect.io logo
Join Slack
Powered by
Hi, I have prefect 0.13.8 installed and running o...
# ask-community
r
Hi, I have prefect 0.13.8 installed and running on Win 10, Python 3.7.9. At the very end of trying to registering the flow (
INFO - prefect.Docker | Pushing image to the registry...
) I get a 
shutil PermissionError: [WinError 5] Access is denied
. Has anybody had the same issues? We are using Python 3.7.9 due to some issues with Python 3.8 of another third party package ...
✔️ 1
n
I haven't seen this personally @Robin but is it possible that the directory to which images are being written isn't one your process has permission to modify? This can happen if the files were created as read only
r
Thanks for the support! Might be possible. Is there a way to influence this via prefect docker?
n
I don't think this is something Prefect has much control over, but I think you have a couple of options: 1. manually remove the 
Read-only
attribute from the offending folder (I'd guess a Docker-specific folder or a temp folder at the 
C:
level) through the windows GUI folder properties 2. Using 
os.chmod
in your script to remove read-only permissions from that folder
r
Ah, I see thanks!
After having changed permissions via windows explorer as you suggested, I get the following error: 
prefect.utilities.exceptions.ClientError: [{'path': ['create_flow_from_compressed_string'], 'message': '[{\'extensions\': {\'path\': \'$\', \'code\': \'invalid-json\'}, \'message\': "Error in $: Failed reading: takeWhile1. Expecting object value at \'Infinity,Infinity],tags:[],type:prefect.core.parameter.Parameter,outputs:typing.Any\'"}]', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
n
That's looking like it's related to your flow code, are you able to share that or reproduce with a small example?
r
Copy code
import sys
from prefect import Parameter
from prefect import Flow, task, unmapped
from prefect.environments import DaskKubernetesEnvironment
from prefect.environments.storage import Docker
from prefect.tasks.aws.secrets_manager import AWSSecretsManager
from prefect.engine.executors import DaskExecutor
import accure_<http://analytics.io|analytics.io>.snowflake_utils as snowflake_io
import prefect
from os import path
from datetime import datetime

@task
def hello_world_task():
    print("hello world")

with Flow("e3dc-test-flow") as flow:
    hello_world_task()


flow.environment = DaskKubernetesEnvironment(
    min_workers=1, max_workers=3, labels=["k8s"]
)
# register flow on AWS ECR

flow.storage = Docker(
        python_dependencies=[
        "numpy",
        "pandas",
        # "snowflake-connector-python[pandas]",
        # "snowflake-sqlalchemy==1.2.3",
        # "tqdm",  
        # "boto3",
        # '"docutils<0.16>=0.10"',  
    ],
    registry_url="<http://someidcr.eu-central-1.amazonaws.com|someidcr.eu-central-1.amazonaws.com>",  
    image_name="flow",
    image_tag="beta_"
    + datetime.now().strftime("%Y%m%d_%H%M%S"),  # Unique tag avoids AWS caching
)
flow.register(project_name="eks_test_01")
The flow looks a bit more complex than the above inserted simplified flow. However, only some custom tasks are added. I am just now trying to import more and more of the python dependencies to identify, which one creates the error, as the flow uploads successfully when no python_dependencies are provided
n
Any luck @Robin?
r
Not yet, unfortunately I have slow upload and dependencies are 600 mb, so slow iteration speed
problem might be related to snowflake dependency
n
Hm got it, thanks for working through it! I'm sure we'll learn something important by the end
It seems at least that the read-only removal solved the initial problem
r
yes, although it might make sense to change the default docker temp folder path to something that usually works for windows users
we currently have this issue across different machines
n
Hm yeah I'll look into if that's possible (or if we'd just be moving the problem)
r
Hmm, getting the permissions error again
I guess it's related to the prefect docker task
I might also be able to check it out with @Severin Ryberg [sevberg], but he is in holiday until Thursday
boiled it down to this:
Copy code
from prefect import Flow, task
from prefect.environments import DaskKubernetesEnvironment
from prefect.environments.storage import Docker
from os import path
from datetime import datetime

@task
def hello_world_task():
    print("hello world")

with Flow("test-flow") as flow:
    hello_world_task()


flow.environment = DaskKubernetesEnvironment(
    min_workers=1, max_workers=3, labels=["k8s"]
)

module_dir = path.dirname(path.dirname(path.abspath(__file__)))

flow.storage = Docker(
    registry_url=".<http://dkr.ecr.eu-central-1.amazonaws.com|dkr.ecr.eu-central-1.amazonaws.com>",  
    image_name="flow",
    image_tag="beta_"
    + datetime.now().strftime("%Y%m%d_%H%M%S"),  # Unique tag avoids AWS caching
    files={
    module_dir: "/modules/accure_analytics \n RUN pip install /modules/accure_analytics"
},
)
flow.register(project_name="eks_test_01")
Hence, shutil.copytree seems to cause the problems in combination with windows 10 🤔
n
Hm ok thanks @Robin - would you mind opening a ticket with that code and describe the error you see with it? This might be a little more difficult to fix than I expected and it'd be good to track
r
Good point! We will come up with a solution to this in the next days 👍
I will just try a couple of more things before submitting
n
oh! that'd be great, thank you @Robin!
r
PS: even running e.g. powershell as admin does not solve the problem
@nicholas, does it make sense to create a github issue using @Marvin?
n
Hm I think the old-fashioned way might be better in this case since you've synthesized a good amount of data
r
OK, will do it tomorrow morning 🙂 Greetings from Germany!
n
greetings indeed, you're working very late! have a good night 🙂
👍 1
r
Once again, it seems like a one-liner solves the problem from:
Copy code
src=src, dst=full_fname, symlinks=False, ignore=None
to
Copy code
src=src, dst=full_fname, symlinks=False, ignore=None, copy_function=shutil.copyfile
See related pull request
n
Great! The Core team will have a look as soon as possible. Are you unblocked @Robin?
r
Yes, flows are running again on cloud 🙏
Open in Slack
PreviousNext
Powered by