https://prefect.io logo
Join Slack
Powered by
Hello everyone! I have a flow that executes sql qu...
# ask-community
j
Hello everyone! I have a flow that executes sql queries, which I store in a subfolder within the repository (Gitlab). Something like that:
Copy code
flows/process_01/main.py
flows/process_01/queries/query1.sql
flows/process_01/queries/query2.sql
flows/process_01/queries/query3.sql
It works fine in my local development environment, and I have no problems running the deploy to Prefect Cloud, with a worker on aws EC2. When running it from Prefect Cloud, I get an error like this:
Copy code
FileNotFoundError: [Errno 2] No such file or directory: 'flows/process_01/queries/query1.sql'
Is there some configuration in deploy that I need to do, so that it considers non-python files? My deploy file looks like this:
Copy code
pull:
- prefect.deployments.steps.git_clone:
     id: clone-step
     repository: <https://gitlab.com/aaaaa/bbb.git>
     branch: master
     access_token: '{{ prefect.blocks.secret.gitlab-api-token }}'
- prefect.deployments.steps.pip_install_requirements:
     directory: '{{ clone-step.directory }}'
     requirements_file: requirements.txt
hello everyone! I think I found the my problem: To open the sql file, I get the current path with os.getcwd(). But my deploy file has as entrypoint the subfolder where the flow is. So in the worker, os.getcwd() returns the root folder. And in my local development environment, os.getcwd() returns the subfolder of the flow. os.getcwd() on worker:
Copy code
/tmp/tmpb10mhcdtprefect/repo-name/
os.getcwd() on local:
Copy code
C:\Dir1\repo-name\flows\process_01\
Is it possible to set the working directory as the subfolder of the flow (on the example: /repo-name/flows/process_01/), so that os.getcwd() returns that path?
j
Hi, I think you'll want to set your working directory in your deployment's pull steps, similar to this: https://prefect-community.slack.com/archives/CL09KU1K7/p1686688509519969?thread_ts=1686687970.243059&amp;cid=CL09KU1K7
j
Thanks for reply @Jake Kaplan! I had to modify the code you sent me like this (according to the documentation, since it gave me an error):
Copy code
- prefect.deployments.steps.set_working_directory:
    directory: /home/prefect
I think changing working-dir works, but now I get an error with git clone command.
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/home/prefect/.local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 122, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/home/prefect/.local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 93, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/home/prefect/.local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 291, in aresult
    return await asyncio.wrap_future(self.future)
  File "/home/prefect/.local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 315, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/home/prefect/.local/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 231, in git_clone
    raise RuntimeError(
RuntimeError: Failed to clone repository '<https://gitlab.com/repo/prefect-repo.git>' with exit code 128.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/prefect/.local/lib/python3.10/site-packages/prefect/engine.py", line 394, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/home/prefect/.local/lib/python3.10/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/home/prefect/.local/lib/python3.10/site-packages/prefect/deployments/deployments.py", line 217, in load_flow_from_flow_run
    output = await run_steps(deployment.pull_steps)
  File "/home/prefect/.local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 150, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.git_clone
Do you have any idea what I'm doing wrong? This is my code for the pull:
Copy code
pull:
- prefect.deployments.steps.set_working_directory:
    directory: /home/prefect
- prefect.deployments.steps.git_clone:
    id: clone-step
    repository: <https://gitlab.com/repo/prefect-repo.git>
    branch: master
    access_token: '{{ prefect.blocks.secret.gitlab-api-token }}'
- prefect.deployments.steps.pip_install_requirements:    
    directory: '{{ clone-step.directory }}'
    requirements_file: requirements.txt
j
128 is a pretty generic git error but usually implies an auth issue. Can you double check that your repo url and access token are right?
j
Hi @Jake Kaplan ! Credentials are fine. Because if I remove the 
prefect.deployments.steps.set_working_directory
setting, it works on Prefect Cloud. Maybe it's a permissions issue, and the process can't do the git clone in the new path. how can i validate that? since the worker is correctly configured in Prefect Cloud.
j
maybe? It's hard to say since it's based on your specific setup. You can you try using the 
run_shell_script
utility pull step to verify things: https://docs.prefect.io/latest/guides/prefect-deploy/#utility-steps I think you'll need to set stream output to True to see it but you can try things like checking the perms of the current dir
j
Thanks @Jake Kaplan, Finally I managed to execute my flow by adjusting the working directory in my code, using the file where the flow is as a reference, like this:
Copy code
CURRENT_DIR = str(Path(__file__).parent.absolute())
2 Views
Open in Slack
PreviousNext
Powered by