https://prefect.io logo
Join Slack
Powered by
Hi all, I am trying to deploy a multi flow setup ...
# ask-community
j
Hi all, I am trying to deploy a multi flow setup on the prefect managed worker pool using the following master flow
Copy code
from prefect import flow
from prefect.runner.storage import GitRepository
from prefect_github import GitHubCredentials
from sync_from_airbyte_flow import sync_from_airbyte

github_repo = GitRepository(
    url="<https://github.com/webbio/AirByte>",
    credentials=GitHubCredentials.load("github"),
)

@flow
def master_flow():
    sync_result = sync_from_airbyte()
    return sync_result

if __name__ == "__main__":
    master_flow.from_source(
        source=github_repo,
        entrypoint="orchestration/master_flow.py:master_flow",
    ).deploy(
        name="Sync from Airbyte and run DBT transform",
        work_pool_name="webbio-workpool",
        image="prefecthq/prefect:3-python3.12",
        push=True,
        cron="0 0 * * *",
    )
I only receive a exited code 1 when running the deployment on the 
from sync_from_airbyte_flow import sync_from_airbyte
. Am I missing something?
n
hi @Joeri Smits - this is usually because you're missing python deps in your remote runtime
prefecthq/prefect:3-python3.12
only contains prefect, so how were you trying to install the deps for 
prefect-github
?
j
Hi @Nate, thanks for the response. Should I configure 
["prefect-github"]
in the Pip Packages on the Prefect UI?
n
yep that's one way to do it, otherwise you could build an image and supply that as the 
image
in 
.deploy()
j
@Nate I’m not sure anymore. Every run I currently execute from the deployment runs endlessly without any response in the Prefect UI
Copy code
from prefect import flow, get_run_logger
from prefect.runner.storage import GitRepository
from prefect_github import GitHubCredentials
from sync_from_airbyte_flow import sync_from_airbyte

github_repo = GitRepository(
    url="<https://github.com/webbio/AirByte>",
    credentials=GitHubCredentials.load("github"),
)

@flow
def master_flow():
    logger = get_run_logger()
    logger.setLevel("DEBUG")
    <http://logger.info|logger.info>("Hello World")

if __name__ == "__main__":
    master_flow.from_source(
        source=github_repo,
        entrypoint="orchestration/master_flow.py:master_flow",
    ).deploy(
        name="Sync from Airbyte and run DBT transform",
        work_pool_name="webbio",
        image="prefecthq/prefect:3-python3.12",
        push=True,
        cron="0 0 * * *",
    )
This is my current master_flow.py and I’ve 
prefect-github
configured in the prefect managed workpool
A simple hello world runs endlessly until cancelled
Copy code
from prefect import flow
from prefect.runner.storage import GitRepository
from prefect_github import GitHubCredentials

@flow
def hello_world_flow():
    return "Hello World"

if __name__ == "__main__":
    github_repo = GitRepository(
        url="<https://github.com/webbio/AirByte.git>",
        credentials=GitHubCredentials.load("github"),
    )

    hello_world_flow.from_source(
        source=github_repo,
        entrypoint="./orchestration/hello_world.py:hello_world_flow",
    ).deploy(
        name="Hello World",
        work_pool_name="webbio",
        image="prefecthq/prefect:3-python3.12",
        push=False,
        cron="0 0 * * *",
    )
Not sure what I am doing wrong here. Been stuck at it all day
Stuck on Late
n
can you try running against a local worker 
prefect worker start --pool local-pool
instead of a managed pool? then you can see the logs of what's going wrong
j
Hi @Nate, Thank I tried that and it’s running fine on a local worker. Maybe my workspace got bricked.
n
hmm that's odd maybe its about your git creds? like the worker process probably has direct access to whatever creds you normally use for git if you have docker installed locally, if you try 
prefect worker start --pool local-docker --type docker
and add a job_variable for the dep
Copy code
hello_world_flow.from_source(
        source=github_repo,
        entrypoint="./orchestration/hello_world.py:hello_world_flow",
    ).deploy(
        name="Hello World",
        work_pool_name="webbio",
        image="prefecthq/prefect:3-python3.12",
        job_variables=dict(env=dict(EXTRA_PIP_PACKAGES="prefect-github")),
        push=False,
        cron="0 0 * * *",
    )
j
@Nate I worked with @Jake Kaplan in another thread and I wasn’t aware that another run was stuck on canceling preventing any other runs from going to pending. Thanks for your support 😄 It’s working now.
n
aha!
good to hear! catjam
6 Views
Open in Slack
PreviousNext
Powered by