Nikhil Jain
05/19/2022, 9:19 PMstorage.add_flow()
method. Is there a way around it?Kevin Kho
05/19/2022, 9:20 PMNikhil Jain
05/19/2022, 9:25 PMFROM python:3.9
WORKDIR /application
COPY requirements.txt .
ARG GITHUB_ACCESS_TOKEN
RUN git config --global url."https://${GITHUB_ACCESS_TOKEN}:@github.com/".insteadOf "<https://github.com/>"
RUN pip install --upgrade pip \
&& pip install -r requirements.txt
COPY . .
If I simply do: docker build myimage -t latest
and push it to my ECR repo, will that work?Kevin Kho
05/19/2022, 9:28 PMflow.register(build=False)
Nikhil Jain
05/19/2022, 9:53 PMFlow is not contained in this Storage
Traceback (most recent call last):
File "/usr/local/bin/prefect", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect/cli/execute.py", line 92, in flow_run
raise exc
File "/usr/local/lib/python3.9/site-packages/prefect/cli/execute.py", line 73, in flow_run
flow = storage.get_flow(flow_data.name)
File "/usr/local/lib/python3.9/site-packages/prefect/storage/local.py", line 88, in get_flow
raise ValueError("Flow is not contained in this Storage")
ValueError: Flow is not contained in this Storage
@task
def say_hello_task(name):
logger = prefect.context.get('logger')
<http://logger.info|logger.info>(f'Hello {name}')
with Flow('say_hello') as say_hello_flow:
people = Parameter('people', default=['Aay', 'Bee'])
say_hello_task.map(people)
say_hello_flow.storage = Local(
path=f'{/application/flows/{__file__}',
stored_as_script=True,
add_default_labels=False
)
register_flows.py
from prefect.run_configs import ECSRun
# local modules
import wordcount
import say_hello
ecs_run = ECSRun(
run_task_kwargs={'cluster': 'dev-prefect-cluster'},
task_role_arn='arn:aws:iam::<>:role/dev-task-runner-ecs-task-role',
execution_role_arn='arn:aws:iam::<>:role/dev-task-runner-ecs-task-execution-role',
image='<>.<http://dkr.ecr.us-west-1.amazonaws.com/dev-automation-scripts-ecr:latest|dkr.ecr.us-west-1.amazonaws.com/dev-automation-scripts-ecr:latest>',
labels=['ecs', 'dev']
)
flows = [
wordcount.count_words_flow,
say_hello.say_hello_flow
]
for flow in flows:
flow.run_config = ecs_run
flow.register(project_name='Dev-Test', build=False)
`Dockerfile`:
FROM python:3.9
WORKDIR /application
COPY requirements.txt .
ARG GITHUB_ACCESS_TOKEN
RUN git config --global url."https://${GITHUB_ACCESS_TOKEN}:@github.com/".insteadOf "<https://github.com/>"
RUN pip install --upgrade pip \
&& pip install -r requirements.txt
COPY . .
Kevin Kho
05/20/2022, 1:17 AMbuild=True
? Or just calling flow.register
without the build argument?Nikhil Jain
05/20/2022, 1:21 AMflows = [
wordcount.count_words_flow,
say_hello.say_hello_flow
]
for flow in flows:
flow_path = f'{common.FLOW_APPLICATION_PATH}/{flow.name}.py'
docker_storage.path = flow_path
docker_storage.add_flow(flow)
docker_storage.build()
for flow in flows:
flow.storage = docker_storage
flow.run_config = ecs_run
# flow.executor = LocalExecutor()
flow.register(project_name='Dev-Test', build=False)
Kevin Kho
05/20/2022, 1:25 AMNikhil Jain
05/20/2022, 1:27 AMKevin Kho
05/20/2022, 5:41 PMNikhil Jain
05/20/2022, 7:27 PMfrom setuptools import setup, find_packages
from os import path
import __about__
loc = path.abspath(path.dirname(__file__))
with open(loc + '/requirements.txt') as f:
requirements = f.read().splitlines()
required = []
dependency_links = []
# Do not add to required lines pointing to Git repositories
EGG_MARK = '#egg='
for line in requirements:
if line.startswith('-e git:') or line.startswith('-e git+') or \
line.startswith('git:') or line.startswith('git+'):
line = line.lstrip('-e ') # in case that is using "-e"
if EGG_MARK in line:
package_name = line[line.find(EGG_MARK) + len(EGG_MARK):]
repository = line[:line.find(EGG_MARK)]
required.append('%s @ %s' % (package_name, repository))
dependency_links.append(line)
else:
print('Dependency to a git repository should have the format:')
print('<git+ssh://git@github.com/xxxxx/xxxxxx#egg=package_name>')
else:
required.append(line)
setup(
name='automation-scripts', # Required
version=__about__.__version__,
description='Description here....', # Required
packages=find_packages(), # Required
install_requires=required,
dependency_links=dependency_links,
)
and here’s the dir structure:
automation-scripts
|-- __init__.py
|-- flows/
|-- __init__.py
|-- say_hello.py
|-- common.py
In say_hello.py
, I have the line import common
which is giving the module import error.Kevin Kho
05/20/2022, 7:29 PMNikhil Jain
05/20/2022, 7:29 PMFROM python:3.9
WORKDIR /opt/prefect
COPY . .
ARG GITHUB_ACCESS_TOKEN
RUN git config --global url."https://${GITHUB_ACCESS_TOKEN}:@github.com/".insteadOf "<https://github.com/>" \
&& pip install .
Not able to reproduce the error in local Docker run. I was able to run the flow from any directory in the container, even before I created the python package using setup.py
Kevin Kho
05/20/2022, 7:42 PMwhich python
give you?Nikhil Jain
05/20/2022, 7:47 PMwhich python
gives: /usr/local/bin/python
which points to python3.9
Checking wheel info…Kevin Kho
05/20/2022, 7:55 PMpython setup.py bdist_wheel
I think which builds a wheel fileNikhil Jain
05/20/2022, 8:00 PMbuild/
|-- bdist.macosx-10.9-x86_64/
|-- lib/
|-- flows/
|-- __init__.py
|-- common.py
|-- say_hello.py
So as far as I can tell, wheel looks okay.Kevin Kho
05/20/2022, 8:02 PMpip show package_name
inside the container to show it was installed rightNikhil Jain
05/20/2022, 10:09 PMsetuptools
. Usually when you want to build a python package for a project, all the source code is nested inside a src
or myproject
folder which wasn’t the case for me since I added setup.py
as an after-thought. It ended up creating packages for individual directories in my project (e.g. flows
became a package instead of mypackage.flows
) So I had to import like so: from flows import common
instead of from myproject.flows import common
or the usual import common
that you can do when you don’t have packages at all.Kevin Kho
05/20/2022, 10:10 PMNikhil Jain
05/20/2022, 10:14 PM~/mypackage/
pyproject.toml
setup.cfg # or setup.py
mypackage/ # or src/
__init__.py
some_code.py
somefolder/*
Kevin Kho
05/20/2022, 10:15 PMNikhil Jain
05/20/2022, 10:21 PMimport common
since common.py
was in the same folder as say_hello.py
. And I guess it worked fine because pythonpath was working correctly when I ran: prefect run -p /opt/prefect/flows/say_hello.py
I also figured out how to reproduce in local Docker run what would happen when running on ECS using this code.
fp='/opt/prefect/flows/say_hello.py'
with open(fp) as f:
contents = f.read()
exec(contents) # this throws module not found error
This way I can test my imports in local Docker run without having to push images and having to run everything on ECS.Kevin Kho
05/20/2022, 10:21 PM