https://prefect.io logo
#prefect-server
Title
# prefect-server
d

David Charles

03/23/2022, 11:23 AM
Hi - we have the “mono-repo” layout - we intend to keep all our flows in a package and leverage shared stuff in a common package. We are able to register flows with our AWS hosted instance, but only a very simple “stand-alone” flow executes, we cannot get ones that reference common code to run, details follow in thread… Any help much appreciated!
We have the “mono-repo” layout below, we intend to keep all our flows in the
flows
package and leverage shared stuff in
common
.
Copy code
. # <-- repo root
├── common
│   ├── __init__.py
│   ├── config.py              # Reusable config
│   ├── schedules              # Reusable schedules
│   │   ├── __init__.py
│   │   └── simple_schedule.py
│   └── tasks                  # Reusable tasks
│       └── __init__.py
└── flows
    ├── __init__.py
    └── simple_flow
        ├── __init__.py
        └── src
            ├── __init__.py
            ├── __main__.py
            ├── config.py
            ├── core.py
            ├── flow.py
            └── tasks.py
flow.py
looks a like this:
Copy code
from prefect import Flow, Parameter
from prefect.storage import GitLab

from flows.simple_flow.src import config
from flows.simple_flow.src.tasks import (
    get_source,
    decode_source,
    store_decoded_data
)

# just to prove import works
from common.schedules.simple_schedule import schedule_daily
data_source_url = Parameter("data_source_url", default=config.data_source_url)


storage = GitLab(
    host="<https://private-gitlab-host.com>",
    repo="the-repo",
    path="flows/simple_flow/src/flow.py",
    ref="main",
    access_token_secret="TOKEN_SECRET",
)


with Flow(name="simple_flow", storage=storage) as flow:
    data = get_source(data_source_url=data_source_url)
    decoded_data = decode_source(data=data)
    store_decoded_data(decoded_data=decoded_data)

if __name__ == "__main__":
    flow.register(project_name="default", add_default_labels=False)
This all works fine running a Prefect core server locally with a local agent. We’ve deployed into AWS and have an ECS agent. I have updated my local
~/.prefect/config.toml
as follows:
Copy code
backend = "server"
[server]
host = "<https://our-aws-prefect-apollo.domain.com>"
port = "443"
endpoint = "${server.host}:${server.port}"
  [server.ui]
  endpoint = "<https://our-aws-prefect-ui.domain.com>"
I register the flow from a local machine using (python interpreter):
Copy code
>>> from flows.simple_flow.src.flow import flow
>>> from prefect.run_configs import UniversalRun
>>> flow.name
'simple_flow'

>>> flow.storage
<Storage: GitLab>

>>> flow.run_config = UniversalRun(labels=["dev"])
>>> flow.register(project_name="default", labels=["dev"])
Flow URL: <https://our-aws-prefect-ui.domain.com/main/flow/53ae2776-36d1-4bed-8f9a-87ce95fad866>
 └── ID: 182b65b6-5ad1-42c4-98ea-eac767b3b867
 └── Project: default
 └── Labels: ['dev']
'182b65b6-5ad1-42c4-98ea-eac767b3b867'
This registers without error - then when I try to execute I see this in the flow LOGS:
Copy code
Failed to load and execute Flow's environment: KeyError("'__name__' not in globals")
I suppose main question I have is, when using GitLab storage, when we register the flow, does the runner pull in all the modules in the repo that the
flow.py
references? (in my example, the
config.py
,
core.py
and
tasks.py
modules in the same package as
flow.py
, as well as stuff in the top level
common
package?
k

Kevin Kho

03/23/2022, 1:10 PM
Hi @David Charles, only Docker storage keeps the dependencies. Gitlab and script based storage only keeps the flow file. Actually for Prefect 2.0, it is on the immediate roadmap to make packaging a lot easier. Not saying this will be the exact mechanism, but cloudpickle 2.0 recently added a way to serialize custom modules. For now though, you either need to package the dependencies in a Docker container or have them in the execution environment.
d

David Charles

03/23/2022, 1:14 PM
Thanks @Kevin Kho for the clarification, that's good to know. Appreciate you getting back to me 😀
👍 1
k

Kyle McChesney

03/23/2022, 2:07 PM
we use S3 storage, but then do an ECSRun and the container thats used has the module pip installed on it (technically it also has a duplicate copy of the flow code but thats not whats used when the flow actually runs). Overall, it works quite well for us. For local development/running, it all behaves like a normal python package similar to what you’ve laid out above.
d

David Charles

03/23/2022, 2:46 PM
Interesting, thanks @Kyle McChesney - might get back to you for deets 🙂
1
t

Tomer Cagan

03/24/2022, 10:14 AM
@Kyle McChesney - how/when are you getting the module installed on the container - is it part of an external process (e.g. CI pipeline) or something else?
k

Kevin Kho

03/24/2022, 1:38 PM
Until Kyle replies, I think he made his image ahead of time, but what I have seen other people do is use the ENTRYPOINT of the container to
git clone
and
pip install -e .
k

Kyle McChesney

03/24/2022, 2:08 PM
we have a CI process that pushes a built container version to a container repo in AWS and then registers the new flow version. So latest flow <> latest image.
d

David Charles

04/01/2022, 3:45 PM
We used Docker storage in the end. We built a
storages
module that has a
get_storage
callable. It returns an appropriate storage based on environment, so for local flow execution it defaults to
Local()
but we have options to add other storages down the line. When we register a flow (e.g. in a CI pipeline) it will get a
Docker
storage that’s been instanced with
dockerfile
and
registry_url
etc for the docker build and push that ensues. Main issue encountered doing this from a mono-repo was we wanted to have a single “parameterised” Dockerfile (i.e. using
ARG flow_name
) so we can build per-flow deps. However seems there’s a bug in
prefect/storage/docker.py
where
build_kwargs
is incorrectly passed into the Python Docker client. I’ve raised this issue: https://github.com/PrefectHQ/prefect/issues/5630
OK not a bug, just not obvious (well, to me 😂) have posted what I did wrong
k

Kevin Kho

04/01/2022, 5:05 PM
Oh I see that’s good to know
18 Views