Leon Kozlowski
05/06/2022, 12:18 AMprefect build …
and prefect register ...
- My Dockerfiles are configured to use a WORKDIR
called /app
File "/app/src/<some_file>.py", line 28, <function_name>
etc.tmp
directories that my Jenkinsfile creates, which is causing flows to fail while loading environment variables (since my environment variables exist in cluster through k8s secrets)Task 'XXXXX': Exception encountered during task execution!
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 876, in get_task_run_state
value = prefect.utilities.executors.run_task_with_timeout(
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/executors.py", line 467, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "/tmp/jenkins-XXXXXX/workspace/XXXXX/prefect-flows/flows/XXXXX/src/tasks/XXXXX.py", line 32, in XXXXX
File "/app/src/helpers.py", line 28, in query_db
engine = create_engine(url=url)
File "<string>", line 2, in create_engine
I tried rebuilding my flow with no cache, but I’m still seeing this odd jenkins tmp filepath being called for task runsAnna Geller
05/06/2022, 12:26 AM/app/src
? looks like your flow is looking for this one rather than /app
re environment variables, this is a common and understandable source of issues with respect to build-time vs. runtime that Prefect 2.0 tries to address - you need to make sure that if you use those env variables somewhere in your flow code, then they must also exist in your build environment (Jenkins server), not only in your execution layer (Kubernetes)Leon Kozlowski
05/06/2022, 12:28 AMsrc
into my WORKDIR
Anna Geller
05/06/2022, 12:30 AMLeon Kozlowski
05/06/2022, 12:33 AMENV PYTHONPATH="$PYTHONPATH:src/"
COPY --from=builder /usr/local/lib/python3.8/site-packages/ /usr/local/lib/python3.8/site-packages/
WORKDIR /app
COPY src src
COPY job_template.yaml .
COPY VERSION .
Anna Geller
05/06/2022, 12:34 AMLeon Kozlowski
05/06/2022, 12:35 AMAnna Geller
05/06/2022, 12:35 AMLeon Kozlowski
05/06/2022, 12:35 AMTask
and it workedAnna Geller
05/06/2022, 12:36 AM@task
is preferred over Task
but if it works 🙂Leon Kozlowski
05/06/2022, 12:38 AMAnna Geller
05/06/2022, 12:46 AM