https://prefect.io
Join Slack
Hi, is there a way for my local agent to run flows that are stored on Azure Blob storage? I can see ...
# prefect-server
s
Hi, is there a way for my local agent to run flows that are stored on Azure Blob storage? I can see the flows are registered locally and stored on the blob storage but when I tried to run them, they are not started
k
Hey @Scarlett King, what error do you get?
s
Just no error, when I kicked off the quick run on the UI, the flow just doesn’t start
I can see that it is stored on the blob storage container that I specified and I can see it on the UI on localhost:8080
The local agent just doesn’t seem to pick up this flow
k
Is the Flow stuck in Scheduled? or does it get Submitted?
s
It’s stuck in scheduled
k
What are the labels for your flow and agent?
s
The flow is None, the agent is my machine name
k
I think None itself is a label which is why it’s not getting picked up. Can you try matching them or removing the agent one with 
prefect agent … start --no-hostname-label
?
s
Great, thank you for that. I now have an error on the agent terminal: ‘run_config’ of type ‘KubernetesRun’, only ‘LocalRun’ is supported. I have set up the flow with KubernetesRun for flow run on our AKS cluster. Do you recommend to use LocalRun when develop locally and change to KubernetesRun when pushing to the cluster or is there a way to do KubernetesRun locally so we don’t have to change the codes before pushing?
@Kevin Kho I removed the KubernetesRun part and now the flow is stopped at execute flow-run step with the error: Failed to load and execute Flow’s environment: AttributeError(‘NoneType’ object has no attribute ‘rstrip’)
I think it’s just to do with version between flow built and execution env
Is there a way to specify which python version it is for the prefect image pulling from Docker? I need to register the flow on python 3.8.10 to make it the same as my local machine since I can’t easily update my python version to 3.8.12
k
Ohh one sec I know that error
See this , there was a bug with that connection string not being serialized. Fixed in 0.15.6 i think
within the same Python major version should be fine
s
I changed to the docker image for prefect 0.14.22 as the python image it bases on is 3.8.10 and reinstalled prefect on my machine to match this version. Now it’s back to show the AttributeError that I mentioned above
Ah, I didn’t see your message above. So what does this mean? Should I upgrade everything back to latest?
k
I think there might be a workaround here ?
s
So I added the env var AZURE_STORAGE_CONNECTION_STRING in both config.toml and as —env when starting the agent
it seems to move to a new error. Now it’s saying it can’t instantiate PosixPath due to python version differences between the flow built (3.8.12) and the execution environment 3.8.10
k
Could you show me the error message? Haven’t seen this before
If you make a local environment with 3.8.10, does the same error appear when you do 
flow.run()
?
s
Hello, sorry I can’t copy the error message out as VPN blocks that. However, I did a few tests. 
flow.run()
works fine. On this flow, I’m using Path from pathlib to handle file path and such. I would have a task that take filepath as an input and would pass 
filepath= ROOT_DIR / ‘filename.csv’
where 
ROOT_DIR = Path(__file__).resolve().parent
. So I removed these and set the file path as 1 string, I believe this only occurs when I pass the value to a task function. When I run this inside a task it works fine with no error.
Copy code
def save_file(df):
  file_path = ROOT_DIR / ‘file_name.csv’
  df.to_csv(file_path)
It throws the PosixPath error if I do
Copy code
def save_file(df, file_path):
  df.to_csv(file_path)

with Flow(‘abc’, storage=azure_store, run_configs=LocalRun()) as flow:
  df = query_db(sql_query)
  save = save_file(df, file_name=ROOT_DIR / ‘file_name.csv’)
It also throw error when I instantiate KubernetesRun like below
Copy code
kubernetes_run = KubernetesRun(image=os.environ.get(“IMAGE_URL”), job_template_path=os.environ.get(“JOB_TEMPLATE_PATH”, ROOT_DIR / “job_template.yaml”))
And for KubernetesRun, the error is thrown at the flow registration step
It works if I just replace 
ROOT_DIR / “job_template.yaml”
with 
”./job_template.yaml”
however, if I have it like this, it will throw error when I do 
flow.run()
as it can’t find the job_template.yaml file
So I got this resolved by converting the Path object to string. It works perfectly with 
str(ROOT_DIR / “job_template.yaml”)
k
Thanks for the detailed solution. I am surprised that there was an inconsistency in the Python minor version
s
Hey, np. Thank you very much for your help too
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by