https://prefect.io logo
Title
l

Leon Kozlowski

05/06/2022, 12:18 AM
Hi all - I have some odd behavior happening for a flow deployed to prefect cloud This might be hard to explain, but I will put most of the details in a thread
Agent: Kubernetes Storage: Docker
I use a jenkins pipeline to build and register my flows. This jenkins pipeline creates tmp directories to build and install dependencies in order to execute
prefect build …
and
prefect register ...
- My Dockerfiles are configured to use a
WORKDIR
called
/app
Typically in flow logs. for failures I’m able to see a traceback and see:
File "/app/src/<some_file>.py", line 28, <function_name>
etc.
The most recent flow I deployed - I am seeing tracebacks that are calling functions from
tmp
directories that my Jenkinsfile creates, which is causing flows to fail while loading environment variables (since my environment variables exist in cluster through k8s secrets)
An example traceback (blanked out function/file names):
Task 'XXXXX': Exception encountered during task execution!
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 876, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/executors.py", line 467, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/tmp/jenkins-XXXXXX/workspace/XXXXX/prefect-flows/flows/XXXXX/src/tasks/XXXXX.py", line 32, in XXXXX
  File "/app/src/helpers.py", line 28, in query_db
    engine = create_engine(url=url)
  File "<string>", line 2, in create_engine
I tried rebuilding my flow with no cache, but I’m still seeing this odd jenkins tmp filepath being called for task runs
a

Anna Geller

05/06/2022, 12:26 AM
could this import error be because the WORKDIR is not
/app/src
? looks like your flow is looking for this one rather than /app re environment variables, this is a common and understandable source of issues with respect to build-time vs. runtime that Prefect 2.0 tries to address - you need to make sure that if you use those env variables somewhere in your flow code, then they must also exist in your build environment (Jenkins server), not only in your execution layer (Kubernetes)
l

Leon Kozlowski

05/06/2022, 12:28 AM
I copy
src
into my
WORKDIR
a

Anna Geller

05/06/2022, 12:30 AM
I see - still somehow copying files over doesn't seem to add them to the PYTHONPATH to make those importable in your flow code. Have you thought about building a package to make that easier?
it could be something as simple as that, then copy and install as here
l

Leon Kozlowski

05/06/2022, 12:33 AM
hmm - I’m just sort of confused, because I havent experienced this with any of my other flows
I am setting a pythonpath in my dockerfile
ENV PYTHONPATH="$PYTHONPATH:src/"

COPY --from=builder /usr/local/lib/python3.8/site-packages/ /usr/local/lib/python3.8/site-packages/

WORKDIR /app

COPY src src
COPY job_template.yaml .
COPY VERSION .
a

Anna Geller

05/06/2022, 12:34 AM
I can understand that - Python path is peculiar 😄
l

Leon Kozlowski

05/06/2022, 12:35 AM
So, I actually just got it to work
And this might sound insane, but 1 of my tasks was a function with a decorator
a

Anna Geller

05/06/2022, 12:35 AM
wow, what did you do differently?
l

Leon Kozlowski

05/06/2022, 12:35 AM
and I changed that to a subclass of
Task
and it worked
a

Anna Geller

05/06/2022, 12:36 AM
generally,
@task
is preferred over
Task
but if it works 🙂
l

Leon Kozlowski

05/06/2022, 12:38 AM
What do you mean?
a

Anna Geller

05/06/2022, 12:46 AM
it would be good to understand why this change helped