https://prefect.io logo
Channels
Powered by
r

Robin

09/29/2020, 9:36 AM
Hi, I have prefect 0.13.8 installed and running on Win 10, Python 3.7.9. At the very end of trying to registering the flow (
INFO - prefect.Docker | Pushing image to the registry...
) I get a 
shutil PermissionError: [WinError 5] Access is denied
. Has anybody had the same issues? We are using Python 3.7.9 due to some issues with Python 3.8 of another third party package ...
✔️ 1
n

nicholas

09/29/2020, 1:30 PM
I haven't seen this personally @Robin but is it possible that the directory to which images are being written isn't one your process has permission to modify? This can happen if the files were created as read only
r

Robin

09/29/2020, 2:04 PM
Thanks for the support! Might be possible. Is there a way to influence this via prefect docker?
n

nicholas

09/29/2020, 2:12 PM
I don't think this is something Prefect has much control over, but I think you have a couple of options: 1. manually remove the 
Read-only
attribute from the offending folder (I'd guess a Docker-specific folder or a temp folder at the 
C:
level) through the windows GUI folder properties 2. Using 
os.chmod
in your script to remove read-only permissions from that folder
r

Robin

09/29/2020, 2:22 PM
Ah, I see thanks!
After having changed permissions via windows explorer as you suggested, I get the following error: 
prefect.utilities.exceptions.ClientError: [{'path': ['create_flow_from_compressed_string'], 'message': '[{\'extensions\': {\'path\': \'$\', \'code\': \'invalid-json\'}, \'message\': "Error in $: Failed reading: takeWhile1. Expecting object value at \'Infinity,Infinity],tags:[],type:prefect.core.parameter.Parameter,outputs:typing.Any\'"}]', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
n

nicholas

09/29/2020, 3:19 PM
That's looking like it's related to your flow code, are you able to share that or reproduce with a small example?
r

Robin

09/29/2020, 3:29 PM
Copy code
import sys
from prefect import Parameter
from prefect import Flow, task, unmapped
from prefect.environments import DaskKubernetesEnvironment
from prefect.environments.storage import Docker
from prefect.tasks.aws.secrets_manager import AWSSecretsManager
from prefect.engine.executors import DaskExecutor
import accure_<http://analytics.io|analytics.io>.snowflake_utils as snowflake_io
import prefect
from os import path
from datetime import datetime

@task
def hello_world_task():
    print("hello world")

with Flow("e3dc-test-flow") as flow:
    hello_world_task()


flow.environment = DaskKubernetesEnvironment(
    min_workers=1, max_workers=3, labels=["k8s"]
)
# register flow on AWS ECR

flow.storage = Docker(
        python_dependencies=[
        "numpy",
        "pandas",
        # "snowflake-connector-python[pandas]",
        # "snowflake-sqlalchemy==1.2.3",
        # "tqdm",  
        # "boto3",
        # '"docutils<0.16>=0.10"',  
    ],
    registry_url="<http://someidcr.eu-central-1.amazonaws.com|someidcr.eu-central-1.amazonaws.com>",  
    image_name="flow",
    image_tag="beta_"
    + datetime.now().strftime("%Y%m%d_%H%M%S"),  # Unique tag avoids AWS caching
)
flow.register(project_name="eks_test_01")
The flow looks a bit more complex than the above inserted simplified flow. However, only some custom tasks are added. I am just now trying to import more and more of the python dependencies to identify, which one creates the error, as the flow uploads successfully when no python_dependencies are provided
n

nicholas

09/29/2020, 4:06 PM
Any luck @Robin?
r

Robin

09/29/2020, 4:09 PM
Not yet, unfortunately I have slow upload and dependencies are 600 mb, so slow iteration speed
problem might be related to snowflake dependency
n

nicholas

09/29/2020, 4:21 PM
Hm got it, thanks for working through it! I'm sure we'll learn something important by the end
It seems at least that the read-only removal solved the initial problem
r

Robin

09/29/2020, 4:22 PM
yes, although it might make sense to change the default docker temp folder path to something that usually works for windows users
we currently have this issue across different machines
n

nicholas

09/29/2020, 4:28 PM
Hm yeah I'll look into if that's possible (or if we'd just be moving the problem)
r

Robin

09/29/2020, 7:38 PM
Hmm, getting the permissions error again
I guess it's related to the prefect docker task
I might also be able to check it out with @Severin Ryberg [sevberg], but he is in holiday until Thursday
boiled it down to this:
Copy code
from prefect import Flow, task
from prefect.environments import DaskKubernetesEnvironment
from prefect.environments.storage import Docker
from os import path
from datetime import datetime

@task
def hello_world_task():
    print("hello world")

with Flow("test-flow") as flow:
    hello_world_task()


flow.environment = DaskKubernetesEnvironment(
    min_workers=1, max_workers=3, labels=["k8s"]
)

module_dir = path.dirname(path.dirname(path.abspath(__file__)))

flow.storage = Docker(
    registry_url=".<http://dkr.ecr.eu-central-1.amazonaws.com|dkr.ecr.eu-central-1.amazonaws.com>",  
    image_name="flow",
    image_tag="beta_"
    + datetime.now().strftime("%Y%m%d_%H%M%S"),  # Unique tag avoids AWS caching
    files={
    module_dir: "/modules/accure_analytics \n RUN pip install /modules/accure_analytics"
},
)
flow.register(project_name="eks_test_01")
Hence, shutil.copytree seems to cause the problems in combination with windows 10 🤔
n

nicholas

09/29/2020, 9:32 PM
Hm ok thanks @Robin - would you mind opening a ticket with that code and describe the error you see with it? This might be a little more difficult to fix than I expected and it'd be good to track
r

Robin

09/29/2020, 9:35 PM
Good point! We will come up with a solution to this in the next days 👍
I will just try a couple of more things before submitting
n

nicholas

09/29/2020, 9:35 PM
oh! that'd be great, thank you @Robin!
r

Robin

09/29/2020, 9:57 PM
PS: even running e.g. powershell as admin does not solve the problem
@nicholas, does it make sense to create a github issue using @Marvin?
n

nicholas

09/29/2020, 10:10 PM
Hm I think the old-fashioned way might be better in this case since you've synthesized a good amount of data
r

Robin

09/29/2020, 10:21 PM
OK, will do it tomorrow morning 🙂 Greetings from Germany!
n

nicholas

09/29/2020, 10:22 PM
greetings indeed, you're working very late! have a good night 🙂
👍 1
r

Robin

09/30/2020, 8:01 AM
Once again, it seems like a one-liner solves the problem from:
Copy code
src=src, dst=full_fname, symlinks=False, ignore=None
to
Copy code
src=src, dst=full_fname, symlinks=False, ignore=None, copy_function=shutil.copyfile
See related pull request
n

nicholas

09/30/2020, 12:42 PM
Great! The Core team will have a look as soon as possible. Are you unblocked @Robin?
r

Robin

09/30/2020, 3:50 PM
Yes, flows are running again on cloud 🙏
Powered by