https://prefect.io
Join Slack
I am porting a python command line program to a prefect flow - I am doing something like this ```@fl...
# ask-community
j
I am porting a python command line program to a prefect flow - I am doing something like this
Copy code
@flow
def my_flow(arg1, etc.):
    ...

if __name__ == "__main__":
   # process some CLI args, etc
   deployment = my_flow.to_deployment(
       name="my_custom_name",
       version="1.2.3",
       parameters = {
            "arg1": value1,
            ..etc
       }
   )
   serve(deployment)
This was working great until I tried to run my CLI command from a place other than my source tree, then it fails in python3.10/site-packages/prefect/deployments/runner.py at line 543
Copy code
541                 # set entrypoint                                                                    
542                 entry_path = (                                                                      
543                     Path(flow_file).absolute().relative_to(Path.cwd().absolute())                   
544                 )                                                                                   
545                 deployment.entrypoint = f"{entry_path}:{flow.fn.__name__}"                          
546
Is there a hard requirement that you must run a flow this way from within your source tree? Why? If not, is there a way to set an argument to avoid this error?
n
hi @Janet Carson - it would be most helpful if you could share your entire stack trace! perhaps also the rest of your if name == main block, if possible
j
Yeah, it's nested within a complicated program, I'll have to make a smaller test case for you.
n
great thank you!
j
@Nate Since the example requires subdirectories, would it be easier to make you a tar or a zip?
Untarball this, 
make setup
to install, then 
make works
works and 
make fails
fails
example.tar
@Nate Did that test case work for you?
n
sorry, i haven't had a chance to reproduce yet do you happen to have an minimal reproduction that doesn't require a tar / zip?
j
I can paste all the files separately if that helps
They're not large, I just combined into one file
n
ahh yeah bc it requires subdirs i forgot, okay thanks. nevermind, ill try to get to it soon
j
Again, no trouble to paste the files separately or put in a different format than tarball if that helps at all
@Nate Here is the data as separate files ``````
Lay them out like this
Copy code
simple_example/
simple_example/silly_package/
simple_example/silly_package/pyproject.toml
simple_example/silly_package/silly_package/
simple_example/silly_package/silly_package/silly_flow.py
simple_example/Makefile
Makefile,pyproject.toml,silly_flow.py
Makefilepyproject.tomlsilly_flow.py
n
okay yeah i have reproduced it, give me a couple to understand what's happening
Copy code
nate :: ~/github.com/zzstoatzz/simple_example ‹main›
» make fails
mkdir sub_directory && \
        . ./silly/bin/activate && \
        cd sub_directory && \
        uv run silly_script
warning: `uv run` is experimental and may change without warning
Your flow 'flow-main' is being served and polling for scheduled runs!

To trigger a run for this flow, use the following command:

        $ prefect deployment run 'flow-main/a-silly-flow'

You can also run your flow via the Prefect UI: <https://app.prefect.cloud/account/12242a57-9f05-4bf5-8853-9bff595d4bab/workspace/cafa2ffa-f6cc-4ed6-ab76-eaa4ba1ad40e/deployments/deployment/5eb2a3dd-e884-4e46-b65c-2f638b84d331>
so you need a 
from_source
to tell us in general where the source lives which in this case is a file, but could be a repo, s3 bucket etc
j
If this were a more complicated example, would this be the site-packages directory for my virtual environment?
n
sorry, whats "this" in your question?
i have a template repo here which illustrates the structure i'd recommend in most cases
j
So, your correction is :
Copy code
path = Path(__file__)
    flow_main.from_source(
        source=str(path.parent.resolve()),
        entrypoint=f"{path.name}:flow_main",
I'm wondering if the path in the general case (where I have multiple packages, etc, if the path would be the location of "site-packages" in my virtual environment where all the everything is, or whether it is still just the path to where the entrypoint file is
n
oh, i see. yeah its the latter would not generally recommend pointing the deployment at site-packages
j
thank you
catjam 1
One more follow up question: Do I need to call "from_source" on all my flows if this first flow is going to have to create a sub-flow? Or, since tasks can create other tasks now, should I just make the second thing a "beefy" task rather than a subflow?
n
no, 
from_source
is just to tell the API where the deployment entrypoint is normal python rules once you start the entrypoint
j
thank you again!
catjam 1
5 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by