https://prefect.io logo
Join SlackCommunities
Powered by
:wave: Running into the below error on this step p...
# ask-community
f
👋 Running into the below error on this step prefect.deployments.steps.pip_install_requirements install step Any pointers to what I am doing wrong here ?
Copy code
StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.pip_install_requirements
Copy code
pull:
  - prefect.deployments.steps.git_clone:
      id: my-git
      repository: git@github.com:mygit/mygit.git
      branch: main
      access_token: '{{ github-prefect-dev }}'
  - prefect.deployments.steps.pip_install_requirements:
      directory: '{{ my-git }}'
      requirements_file: requirements.txt
^ excerpt from my prefect.yaml
n
do you have the whole trace?
f
Copy code
16:57:03.731 | ERROR   | Flow run 'uppish-turkey' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 154, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 125, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 293, in aresult
    return await asyncio.wrap_future(self.future)
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 355, in _run_async
    result = await coro
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/deployments/steps/utility.py", line 234, in pip_install_requirements
    async with open_process(
  File "/opt/homebrew/Cellar/python@3.9/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 181, in __aenter__
    return await self.gen.__anext__()
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/utilities/processutils.py", line 202, in open_process
    process = await anyio.open_process(command, **kwargs)
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/anyio/_core/_subprocesses.py", line 126, in open_process
    return await get_asynclib().open_process(
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1041, in open_process
    process = await asyncio.create_subprocess_exec(
  File "/opt/homebrew/Cellar/python@3.9/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/subprocess.py", line 236, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
  File "/opt/homebrew/Cellar/python@3.9/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 1676, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "/opt/homebrew/Cellar/python@3.9/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/unix_events.py", line 197, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
  File "/opt/homebrew/Cellar/python@3.9/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "/opt/homebrew/Cellar/python@3.9/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/unix_events.py", line 789, in _start
    self._proc = subprocess.Popen(
  File "/opt/homebrew/Cellar/python@3.9/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/opt/homebrew/Cellar/python@3.9/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1770, in _execute_child
    self.pid = _posixsubprocess.fork_exec(
TypeError: expected str, bytes or os.PathLike object, not dict

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/engine.py", line 414, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/deployments/deployments.py", line 239, in load_flow_from_flow_run
    output = await run_steps(deployment.pull_steps)
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 182, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.pip_install_requirements
 > Running git_clone step...
 > Running pip_install_requirements step...
n
ah i see
Copy code
directory: '{{ my-git }}'
should be
Copy code
directory: '{{ my-git.directory }}'
f
Thanks @Nate. I fixed that and running into the below now
Copy code
/opt/homebrew/Cellar/python@3.9/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py:127: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
17:14:15.740 | INFO    | prefect.deployment - Cloned repository 'git@github.com:my-git/my-git.git' into 'my-git-main'
ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
17:14:16.034 | ERROR   | Flow run 'watchful-python' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 154, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 125, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 293, in aresult
    return await asyncio.wrap_future(self.future)
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 355, in _run_async
    result = await coro
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/deployments/steps/utility.py", line 249, in pip_install_requirements
    raise RuntimeError(
RuntimeError: pip_install_requirements failed with error code 1: ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/engine.py", line 414, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/deployments/deployments.py", line 239, in load_flow_from_flow_run
    output = await run_steps(deployment.pull_steps)
  File "/Users/foobar/development/foobarlabs/fdata/venv/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 182, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.pip_install_requirements
 > Running git_clone step...
 > Running pip_install_requirements step...
n
Copy code
TypeError: expected str, bytes or os.PathLike object, not dict
the
Copy code
'{{ my-git }}'
is the whole dictionary of outputs from that step
is your requirements.txt actually pushed to your remote repo? i literally just hit the same error while making that example 🙂
yess 1
f
is there anyway I can see on the worker, what is the code downloaded ?
n
you could add a run_shell_script pull step after the clone that's just 
ls
for debugging -
f
would that run_shell_script execute after git pull ?
n
yep
well itll go in the order that you specify them, so put the run_shell_script after the clone
f
oh got it, I can put run_shell_script under 
pull
section.. let me play around that.. thanks much!
n
yep just like this one - no problem
thank you 1
f
ok, looks like it is trying to find in the current directory instead of looking inside the 
foobarlab-main
dir. Here is the ls option output
Copy code
drwx------    3 foobar  staff    96B Jan  9 17:26 .
drwx------@ 217 foobar  staff   6.8K Jan  9 17:25 ..
drwxr-xr-x    6 foobar  staff   192B Jan  9 17:26 foobarlabs-main
do I need to set any path variable somewhere ?
n
hmm you shouldnt need to, are both your prefect.yaml and the requirements.txt at the root of the repo you're cloning?
f
foobarlabs-main/prefect_flows/prefect.yaml foobarlabs-main/requirements.txt
I moved prefect.yaml to root and updated the entry point accordingly, still seeing the same errror
🤦 its a mono repo, updated the directory as below and the requirements are installed successfully
Copy code
directory: '{{ my-git }}/fdata'
but, now the prefect worker can't find the flow. I guess it assumes everything starts from root ? anyway to work around ? I tried the 
path
variable by setting to 
fdata/
, it allowed me to deploy but the worker can't resolve the location
n
at this point is your 
prefect.yaml
at repo root? i believe that's a hard requirement at this time
f
no, it is not because it is a mono repo. anyway to clone a specific folder instead of the entire repo ?
n
this monorepo example of my own might be useful reference afaik in github its not supported in a first class way to clone subsets of a repo
the requirements don't have to be at root, but the prefect.yaml does
f
got it. I will try creating separate repo for prefect flows. I really appreciate your help. I will update here, once I sort this out. thanks again!
n
no problem! catjam
5 Views
Open in Slack
PreviousNext
Powered by