Josh Greenhalgh
01/27/2021, 3:40 PMSystem Version check: OK
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/tasks/gcp/__init__.py", line 14, in <module>
from prefect.tasks.gcp.bigquery import (
File "/usr/local/lib/python3.7/site-packages/prefect/tasks/gcp/bigquery.py", line 4, in <module>
from google.cloud import bigquery
ImportError: cannot import name 'bigquery' from 'google.cloud' (unknown location)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/prefect/healthcheck.py", line 151, in <module>
flows = cloudpickle_deserialization_check(flow_file_paths)
File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
flows.append(cloudpickle.loads(flow_bytes))
File "/usr/local/lib/python3.7/site-packages/prefect/tasks/gcp/__init__.py", line 24, in <module>
) from err
ImportError: Using `prefect.tasks.gcp` requires Prefect to be installed with the "gcp" extra.
I tried;
storage = Docker(
registry_url="<http://gcr.io/blah/|gcr.io/blah/>",
image_name=name,
image_tag="latest",
build_kwargs={"buildargs":{"EXTRAS":"kubernetes,gcp"}}
)
With no luck šgoogle-cloud-bigquery
as a pip package?Jan Marais
01/27/2021, 3:43 PMDocker(..., python_dependencies=['google-cloud-bigquery'])
?Josh Greenhalgh
01/27/2021, 3:43 PMJan Marais
01/27/2021, 3:46 PMbuild_kwargs
to handle setting the EXTRAS var in base image dockerfilejosh
01/27/2021, 3:47 PM"prefect[gcp]"
?Jan Marais
01/27/2021, 3:51 PMbuild_kwargs={"EXTRAS": "kubernetes,gcp"}
?Josh Greenhalgh
01/27/2021, 3:54 PMbuild_kwargs
but the image being built doesn't have any ARGS its not actually building prefecthq/prefect
but a dockerfile that uses that as a base)python_dependencies
?Jan Marais
01/27/2021, 4:06 PMJosh Greenhalgh
01/27/2021, 4:18 PMEXTRAS
is a build arg to that which is not being builtJan Marais
01/27/2021, 5:14 PMI did try that (I thought you said prefecthq/prefect is not being built and therefore I suggested explicitly setting the base image to prefecthq/prefect with the EXTRAS as build argsĀ but the image being built doesn't have any ARGS its not actually buildingĀbuild_kwargs
Ā but a dockerfile that uses that as a base)prefecthq/prefect
Josh Greenhalgh
01/27/2021, 5:16 PMFROM blah
RUN some stuff
And blahs dockerfile is;
FROM foo
ARG baz
Then when building the first its not possible to set the build arg baz - as far as I am aware?Jan Marais
01/27/2021, 6:47 PM