https://prefect.io logo
Join Slack
Powered by
Hi all - I have some odd behavior happening for a ...
# prefect-community
l
Hi all - I have some odd behavior happening for a flow deployed to prefect cloud This might be hard to explain, but I will put most of the details in a thread
Agent: Kubernetes Storage: Docker
I use a jenkins pipeline to build and register my flows. This jenkins pipeline creates tmp directories to build and install dependencies in order to execute 
prefect build …
and 
prefect register ...
- My Dockerfiles are configured to use a 
WORKDIR
called 
/app
Typically in flow logs. for failures I’m able to see a traceback and see:
Copy code
File "/app/src/<some_file>.py", line 28, <function_name>
etc.
The most recent flow I deployed - I am seeing tracebacks that are calling functions from 
tmp
directories that my Jenkinsfile creates, which is causing flows to fail while loading environment variables (since my environment variables exist in cluster through k8s secrets)
An example traceback (blanked out function/file names):
Copy code
Task 'XXXXX': Exception encountered during task execution!
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 876, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/executors.py", line 467, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/tmp/jenkins-XXXXXX/workspace/XXXXX/prefect-flows/flows/XXXXX/src/tasks/XXXXX.py", line 32, in XXXXX
  File "/app/src/helpers.py", line 28, in query_db
    engine = create_engine(url=url)
  File "<string>", line 2, in create_engine
I tried rebuilding my flow with no cache, but I’m still seeing this odd jenkins tmp filepath being called for task runs
a
could this import error be because the WORKDIR is not 
/app/src
? looks like your flow is looking for this one rather than /app re environment variables, this is a common and understandable source of issues with respect to build-time vs. runtime that Prefect 2.0 tries to address - you need to make sure that if you use those env variables somewhere in your flow code, then they must also exist in your build environment (Jenkins server), not only in your execution layer (Kubernetes)
l
I copy 
src
into my 
WORKDIR
a
I see - still somehow copying files over doesn't seem to add them to the PYTHONPATH to make those importable in your flow code. Have you thought about building a package to make that easier?
it could be something as simple as that, then copy and install as here
l
hmm - I’m just sort of confused, because I havent experienced this with any of my other flows
I am setting a pythonpath in my dockerfile
Copy code
ENV PYTHONPATH="$PYTHONPATH:src/"

COPY --from=builder /usr/local/lib/python3.8/site-packages/ /usr/local/lib/python3.8/site-packages/

WORKDIR /app

COPY src src
COPY job_template.yaml .
COPY VERSION .
a
I can understand that - Python path is peculiar 😄
l
So, I actually just got it to work
And this might sound insane, but 1 of my tasks was a function with a decorator
a
wow, what did you do differently?
l
and I changed that to a subclass of 
Task
and it worked
a
generally, 
@task
is preferred over 
Task
but if it works 🙂
l
What do you mean?
a
it would be good to understand why this change helped
36 Views
Open in Slack
PreviousNext
Powered by