Hi all, Quick question about flow.from_source(). ...
# ask-community
p
Hi all, Quick question about flow.from_source(). Can I use this from within the same repo? As in, can I do something like this from within the same repo defined by the GitRepo url? @flow async def my_flow(name: str = "world"): print(f"Hello {name}!") if name == "__main__": DEFAULT_GH_SECRET_BLOCK_KEY = "github-secret" access_token = Secret.load(DEFAULT_GH_SECRET_BLOCK_KEY).value flow.from_source( source=GitRepository( url="https://github.com/company/my-repo.git", branch="main", credentials={"access_token": access_token}, ), entrypoint="/builtin_worklflows/paige-test.py:my_flow", ).deploy( name="my-deployment", work_pool_name="my-k8s-job", job_variables=dict(pull_policy="Never"), schedule=(CronSchedule(cron="0 0 * * *", timezone="America/Chicago")), tags=["test1","test2"], paramaters=dict(name='Paige') )
n
hi @Paige Fuller! is there a reason that you wouldn't want to do something like this? you don't need
from_source
if you already have the flow object handy
p
I guess this goes into my lack of understanding of how prefect deployments work! We currently copy all of our files into GCS bucket and then deploy from there, but we are trying to move away from there. If we do this, we werent sure where prefect would get the code from when the deployment is actually run? We are using the open source and not the cloud offering
If we changed it to your suggestion and then ran something like this in our CICD pipeline, would that properly apply our deployments to our prefect server? We are fumbling our way through our current prefect implementation and are worried it wont find the flow code..
name: Deploy to Prefect Server
on:
push:
branches:
- main
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.x
- name: Install dependencies
run: pip install prefect
- name: Deploy to Prefect Server
run: |
# Set up Prefect environment variables
export PREFECT__SERVER__API="${{ secrets.PREFECT_API_URL }}"
python paige-test.py
n
this will be helpful I think
short answer, yes*! that will work for registering the deployment the last missing piece is telling the prefect deployment where the code lives, one question for you: do you have any requirements about whether the flow code lives in the image or somewhere else? (like GitHub?)
p
I mean, we have all of the code living in GitHub so ideally we just pull from there! We just seem to be struggling to make it happen haha
but open to suggestions on what you think is the best approach? We are prefect newbies!!
n
nice! i like keeping my code in github so i dont have to rebuild images every time my code changes best approach is always different for folks, but here's what I'd say (given what I know now from chatting): • to minimize a lot of changes for you all (since you were already using
from_source
) - I would stick with
from_source
and just point it at github like your original example, this will: ā—¦ allow you to write some GHA like you did above (just set
PREFECT_API_URL
there and you should be good!) ā—¦
from_source
will make it so that the eventual
Deployment
you register with the server has the information about where the flow code lives when the worker picks up a flow run from that deployment if you're trying this
Copy code
@flow
async def my_flow(name: str = "world"):
    print(f"Hello {name}!")

if __name__ == "__main__":
    flow.from_source(
      ...
    ).deploy(...)
like above and its not working for you, what's the error look like that you're getting?
as an aside, my favorite approach is to use a `prefect.yaml` and use
prefect --no-prompt deploy --all
in CI like in this repo I'm linking, but lots of folks use
.deploy()
like it sounds like you all prefer šŸ™‚
p
ok fab, will give it a try! I havent had it approved to merge into dev yet until I can convince people its a good idea, so this chat is good ammo haha! Pointint to the github repo from source works fine when its at the same repo in which its running? Just double confirming!
n
Pointing to the github repo from source works fine when its at the same repo in which its running?
yes it does! because all you're doing at the moment you call
.deploy()
is telling your prefect server that "hey, this Deployment I'm giving you lives at • source=
<https://github.com/company/my-repo.git>
• entrpoint=
path/to/my/flow/relative/to/repo/root.py:my_flow_function
so when its time to run, that's where you should go
pull
it from!" (behind the scenes, we translate that
from_source
bit into a
pull
step for your worker that instructs it on how to retrieve your flow code at runtime)
entrypoint="/builtin_worklflows/paige-test.py:my_flow"
also just noticing that your
entrypoint
looks a little goofy, just because of that leading forward slash entrypoint should be relative to repo root
p
ahh thanks! Will change that - thank you!! Appreciate you always responding to my chaotic questions šŸ™‚
n
catjam
p
@Nate actually going to ask you one more question while I have you trapped in this thread!! If we want to deploy a flow as a k8s job, if we deploy to a kubernetes agent pool, will it automatically run as a k8s job? We had some before we defined in deployment.yamls that ran as a process