Hi there Am a big user of prefect for work and am now spinni Prefect Community #prefect-getting-started

Hi there! Am a big user of prefect for work and am...

Mike Ng

02/14/2024, 5:11 AM

Hi there! Am a big user of prefect for work and am now spinning it up for some personal projects. I am replicating the quickstart exactly and run into an error related to the github repo for the create_deployment.py file:

Copy code

FileNotFoundError: [WinError 2] The system cannot find the file specified

I copied the code exactly:

Copy code

from prefect import flow

if __name__ == "__main__":
    flow.from_source(
        source="<https://github.com/discdiver/demos.git>",
        entrypoint="my_gh_workflow.py:repo_info",
    ).deploy(
        name="my-first-deployment",
        work_pool_name="my-managed-pool",
        cron="0 1 * * *",
    )

Nate

02/14/2024, 2:32 PM

hmm this example works for me on 2.14.21, do you mind sharing

prefect version

output?

Mike Ng

02/15/2024, 3:24 AM

Also 2.14.21. Maybe it has something to do with the virtual environment, path or github connection (though it's a public repo)? I also tried instead to use a .serve which did work, but then you can't specify a workpool. Below code work for example.

Copy code

import httpx
from prefect import flow


@flow(log_prints=True)
def get_repo_info(repo_name: str = "PrefectHQ/prefect"):
    url = f"<https://api.github.com/repos/{repo_name}>"
    response = httpx.get(url)
    response.raise_for_status()
    repo = response.json()
    print(f"{repo_name} repository statistics 🤓:")
    print(f"Stars 🌠 : {repo['stargazers_count']}")
    print(f"Forks 🍴 : {repo['forks_count']}")


if __name__ == "__main__":
    get_repo_info.serve(name="my-first-deployment")

Mike Ng

02/15/2024, 3:36 AM

Full version info:

Copy code

Version:             2.14.21
API version:         0.8.4
Python version:      3.11.3
Git commit:          9a059bfe
Built:               Thu, Feb 8, 2024 5:35 PM
OS/Arch:             win32/AMD64
Profile:             default
Server type:         cloud

Jeff Hale

02/15/2024, 1:40 PM

@Mike Ng did you have Docker running when you tried to create the work pool based deployment?

Michael Ng

02/15/2024, 2:21 PM

No local. Should I create a docker image first?

Jeff Hale

02/15/2024, 2:25 PM

Nevermind, I was thinking you were trying to run the flow in a Docker container. Can you share the full traceback?

Mike Ng

02/16/2024, 1:10 AM

sorry for delay. here is the full traceback:

Copy code

flow.from_source(
...     source="<https://github.com/discdiver/demos.git>",
...     entrypoint="my_gh_workflow.py:repo_info"
... ).deploy(
...     name="my-second-deployment",
...     work_pool_name="my-managed-pool",
...     cron="0 1 * * *"
... )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\utilities\asyncutils.py", line 259, in coroutine_wrapper
    return call()
           ^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 398, in __call__
    return self.result()
           ^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 284, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 168, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 355, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\flows.py", line 856, in from_source
    await storage.pull_code()
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\runner\storage.py", line 233, in pull_code
    await self._clone_repo()
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\runner\storage.py", line 257, in _clone_repo       
    await run_process(cmd)
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\anyio\_core\_subprocesses.py", line 62, in run_process     
    async with await open_process(
               ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\anyio\_core\_subprocesses.py", line 126, in open_process
    return await get_asynclib().open_process(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\anyio\_backends\_asyncio.py", line 1041, in open_process   
    process = await asyncio.create_subprocess_exec(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\asyncio\subprocess.py", line 218, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 1694, in subprocess_exec
    transport = await self._make_subprocess_transport(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\asyncio\windows_events.py", line 399, in _make_subprocess_transport      
    transp = _WindowsSubprocessTransport(self, protocol, args, shell,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\asyncio\windows_events.py", line 921, in _start
    self._proc = windows_utils.Popen(
                 ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\asyncio\windows_utils.py", line 153, in __init__
    super().__init__(args, stdin=stdin_rfd, stdout=stdout_wfd,
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\subprocess.py", line 1024, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\subprocess.py", line 1509, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] The system cannot find the file specified

Jeff Hale

02/16/2024, 2:48 PM

Looks like this is a Windows-specific error and not related to retrieving the flow code. Are you running this in a virtual environment where you are generally able to run Python files? The Windows install notes in the docs might be helpful.

Jeff Hale

02/16/2024, 2:56 PM

Stack Overflow on the issue: https://stackoverflow.com/questions/9935151/popen-error-errno-2-no-such-file-or-directory-when-calling-shell-function

Mike Ng

02/17/2024, 7:42 PM

I am yeah. Added the recommended file from prefect installation documentation to Path, tried Shell=True in the deploy function. Even created a a new virtual environment and reinstalled prefect and added venv it to Windows path (also confirmed venv is active). Still getting same error. All other normal python code works fine. Very weird

Jeff Hale

02/19/2024, 1:30 PM

Hi Mike, Sorry you are still dealing with this. I worked with a number of Windows users recently who ran a similar file and didn’t encounter this error - so I think it’s likely to be something specific to your setup. Digging a bit more, it looks like it likely is a PATH issue. Here’s a similar situation discussed on a Ray Python package forum where the issue was solved by adjusting the PATH variable.

Mike Ng

02/20/2024, 4:51 PM

Thanks, Jeff. I still can't get it to work, so I am going to try and deploy using Docker with a custom image. However, now I am running into the following error:

prefect.deployments.runner.DeploymentApplyError: Error creating deployment: <ValidationError: "'my_image:deploy-guide' is not one of ['prefecthq/prefect:2-latest', 'prefecthq/prefect:2-python3.11', 'prefecthq/prefect:2-python3.10', 'prefecthq/prefect:2-python3.9']">

I don't really understand this because I do use the 3.11 base prefect image in the Dockerfile.

Mike Ng

02/20/2024, 4:52 PM

My Dockerfile is:

Copy code

FROM prefecthq/prefect:2-python3.11

COPY requirements.txt .
RUN apt-get update
RUN pip install prefect -U
RUN pip install -r requirements.txt

and my deployment code is:

Copy code

import httpx
from prefect import flow, task
from prefect.deployments import DeploymentImage



@task(retries=2)
def get_repo_info(repo_owner: str, repo_name: str):
    """Get info about a repo - will retry twice after failing"""
    url = f"<https://api.github.com/repos/{repo_owner}/{repo_name}>"
    api_response = httpx.get(url)
    api_response.raise_for_status()
    repo_info = api_response.json()
    return repo_info


@task
def get_contributors(repo_info: dict):
    """Get contributors for a repo"""
    contributors_url = repo_info["contributors_url"]
    response = httpx.get(contributors_url)
    response.raise_for_status()
    contributors = response.json()
    return contributors


@flow(log_prints=True)
def repo_info(repo_owner: str = "PrefectHQ", repo_name: str = "prefect"):
    """
    Given a GitHub repository, logs the number of stargazers
    and contributors for that repo.
    """
    repo_info = get_repo_info(repo_owner, repo_name)
    print(f"Stars 🌠 : {repo_info['stargazers_count']}")

    contributors = get_contributors(repo_info)
    print(f"Number of contributors 👷: {len(contributors)}")


if __name__ == "__main__":
    repo_info.deploy(
        name="my-custom-dockerfile-deployment", 
        work_pool_name="my-managed-pool", 
        image=DeploymentImage(
            name="my_image",
            tag="deploy-guide",
            dockerfile="Dockerfile"
    ),
    push=False
)

Mike Ng

02/20/2024, 4:52 PM

I also tried deploying from Dockerhub instead of building locally and got the same error

Jeff Hale

02/20/2024, 5:25 PM

Ah. Prefect managed work pools don’t allow custom images at the moment. Push work pools and hybrid pools do.

Mike Ng

02/20/2024, 5:42 PM

I see. Okay thanks. I will look into ECS push pool. For the moment I just want to get something to work. I tried deploying from your

tutorial▾

(thanks for all the good content!) which deployed but when I try to run it (even after starting the work pool) I run into the following fail:

Copy code

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\deployments\steps\core.py", line 154, in run_steps
    step_output = await run_step(step, upstream_outputs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\deployments\steps\core.py", line 125, in run_step
    result = await from_async.call_soon_in_new_thread(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 294, in aresult
    return await asyncio.wrap_future(self.future)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 319, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\utilities\asyncutils.py", line 259, in coroutine_wrapper
    return call()
           ^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 407, in __call__
    return self.result()
           ^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 285, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 169, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 319, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: git_clone() missing 1 required positional argument: 'repository'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\engine.py", line 427, in retrieve_flow_then_begin_flow_run
    else await load_flow_from_flow_run(flow_run, client=client)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\client\utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\deployments\deployments.py", line 249, in load_flow_from_flow_run
    output = await run_steps(deployment.pull_steps)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python311\Lib\site-packages\prefect\deployments\steps\core.py", line 182, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.git_clone

Is this just another iteration of the original path/windows issue I had? In other words, it seems like I will need an ECS push deployment with a custom docker image to get around this?

Jeff Hale

02/20/2024, 6:20 PM

Looks like maybe the GitHub repo specified isn’t available?

TypeError: git_clone() missing 1 required positional argument: 'repository'

7 Views

Open in Slack

Previous Next