s

    Scarlett King

    11 months ago
    Hi, is there a way for my local agent to run flows that are stored on Azure Blob storage? I can see the flows are registered locally and stored on the blob storage but when I tried to run them, they are not started
    Kevin Kho

    Kevin Kho

    11 months ago
    Hey @Scarlett King, what error do you get?
    s

    Scarlett King

    11 months ago
    Just no error, when I kicked off the quick run on the UI, the flow just doesn’t start
    I can see that it is stored on the blob storage container that I specified and I can see it on the UI on localhost:8080
    The local agent just doesn’t seem to pick up this flow
    Kevin Kho

    Kevin Kho

    11 months ago
    Is the Flow stuck in Scheduled? or does it get Submitted?
    s

    Scarlett King

    11 months ago
    It’s stuck in scheduled
    Kevin Kho

    Kevin Kho

    11 months ago
    What are the labels for your flow and agent?
    s

    Scarlett King

    11 months ago
    The flow is None, the agent is my machine name
    Kevin Kho

    Kevin Kho

    11 months ago
    I think None itself is a label which is why it’s not getting picked up. Can you try matching them or removing the agent one with
    prefect agent … start --no-hostname-label
    ?
    s

    Scarlett King

    11 months ago
    Great, thank you for that. I now have an error on the agent terminal: ‘run_config’ of type ‘KubernetesRun’, only ‘LocalRun’ is supported. I have set up the flow with KubernetesRun for flow run on our AKS cluster. Do you recommend to use LocalRun when develop locally and change to KubernetesRun when pushing to the cluster or is there a way to do KubernetesRun locally so we don’t have to change the codes before pushing?
    @Kevin Kho I removed the KubernetesRun part and now the flow is stopped at execute flow-run step with the error: Failed to load and execute Flow’s environment: AttributeError(‘NoneType’ object has no attribute ‘rstrip’)
    I think it’s just to do with version between flow built and execution env
    Is there a way to specify which python version it is for the prefect image pulling from Docker? I need to register the flow on python 3.8.10 to make it the same as my local machine since I can’t easily update my python version to 3.8.12
    Kevin Kho

    Kevin Kho

    11 months ago
    Ohh one sec I know that error
    See this , there was a bug with that connection string not being serialized. Fixed in 0.15.6 i think
    within the same Python major version should be fine
    s

    Scarlett King

    11 months ago
    I changed to the docker image for prefect 0.14.22 as the python image it bases on is 3.8.10 and reinstalled prefect on my machine to match this version. Now it’s back to show the AttributeError that I mentioned above
    Ah, I didn’t see your message above. So what does this mean? Should I upgrade everything back to latest?
    Kevin Kho

    Kevin Kho

    11 months ago
    I think there might be a workaround here ?
    s

    Scarlett King

    11 months ago
    So I added the env var AZURE_STORAGE_CONNECTION_STRING in both config.toml and as —env when starting the agent
    it seems to move to a new error. Now it’s saying it can’t instantiate PosixPath due to python version differences between the flow built (3.8.12) and the execution environment 3.8.10
    Kevin Kho

    Kevin Kho

    11 months ago
    Could you show me the error message? Haven’t seen this before
    If you make a local environment with 3.8.10, does the same error appear when you do
    flow.run()
    ?
    s

    Scarlett King

    11 months ago
    Hello, sorry I can’t copy the error message out as VPN blocks that. However, I did a few tests.
    flow.run()
    works fine. On this flow, I’m using Path from pathlib to handle file path and such. I would have a task that take filepath as an input and would pass
    filepath= ROOT_DIR / ‘filename.csv’
    where
    ROOT_DIR = Path(__file__).resolve().parent
    . So I removed these and set the file path as 1 string, I believe this only occurs when I pass the value to a task function. When I run this inside a task it works fine with no error.
    def save_file(df):
      file_path = ROOT_DIR / ‘file_name.csv’
      df.to_csv(file_path)
    It throws the PosixPath error if I do
    def save_file(df, file_path):
      df.to_csv(file_path)
    
    with Flow(‘abc’, storage=azure_store, run_configs=LocalRun()) as flow:
      df = query_db(sql_query)
      save = save_file(df, file_name=ROOT_DIR /file_name.csv’)
    It also throw error when I instantiate KubernetesRun like below
    kubernetes_run = KubernetesRun(image=os.environ.get(“IMAGE_URL”), job_template_path=os.environ.get(“JOB_TEMPLATE_PATH”, ROOT_DIR / “job_template.yaml”))
    And for KubernetesRun, the error is thrown at the flow registration step
    It works if I just replace
    ROOT_DIR / “job_template.yaml”
    with
    ”./job_template.yaml”
    however, if I have it like this, it will throw error when I do
    flow.run()
    as it can’t find the job_template.yaml file
    So I got this resolved by converting the Path object to string. It works perfectly with
    str(ROOT_DIR / “job_template.yaml”)
    Kevin Kho

    Kevin Kho

    11 months ago
    Thanks for the detailed solution. I am surprised that there was an inconsistency in the Python minor version
    s

    Scarlett King

    11 months ago
    Hey, np. Thank you very much for your help too