jars
06/18/2020, 3:02 AMTraceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 986, in get_task_run_state
result = self.result.write(value, filename="output", **prefect.context)
File "/usr/local/lib/python3.7/site-packages/prefect/engine/results/gcs_result.py", line 73, in write
self.gcs_bucket.blob(new.location).upload_from_string(binary_data)
File "/usr/local/lib/python3.7/site-packages/prefect/engine/results/gcs_result.py", line 35, in gcs_bucket
from prefect.utilities.gcp import get_storage_client
File "/usr/local/lib/python3.7/site-packages/prefect/utilities/gcp.py", line 6, in <module>
from google.cloud import bigquery, storage
ImportError: cannot import name 'bigquery' from 'google.cloud' (unknown location)
It seems the process cannot find a prefect core bigquery module inside of google.cloud.
I thought perhaps it had something to do with setting PYTHONPATH
to my own application & lib directories in my Flow's Dockerfile:
ENV PYTHONPATH="/app:/app/lib"
And now Prefect core cannot find it's own packages?
But experimenting with the alternative "extension" of PYTHONPATH
instead of overwrite:
ENV PYTHONPATH="/app:/app/lib:${PYTHONPATH}"
simply yields a PYTHONPATH
with a colon on the end (/app:/app/lib:
), so I opted not to extend, since there is no default.
My suspicions about PYTHONPATH
could very well be a red herring... Any ideas?Chris White
06/18/2020, 3:03 AMprefect[gcp]
into your image to ensure all google related packages are availablejars
06/18/2020, 3:05 AMFROM prefecthq/prefect:0.11.4-python3.7
Does this image have prefect[gcp]
preinstalled?Chris White
06/18/2020, 3:06 AMRUN pip install google-cloud-bigquery google-cloud-storage
and you should be good to gojars
06/18/2020, 3:08 AMChris White
06/18/2020, 3:08 AMjars
06/18/2020, 3:08 AMChris White
06/18/2020, 3:23 AMjars
06/18/2020, 3:26 AMChris White
06/18/2020, 3:28 AMLocalExecutor
(the default)
- set checkpoint=False
on the task that returns the firestore referencejars
06/18/2020, 3:29 AMChris White
06/18/2020, 3:32 AMjars
06/18/2020, 3:33 AMChris White
06/18/2020, 3:33 AM