https://prefect.io logo
Channels
Powered by
j

Josh Greenhalgh

01/27/2021, 3:40 PM
Does anyone know how to specify the prefect extras I need in the Docker storage? I am getting the following build error;
Copy code
System Version check: OK
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/tasks/gcp/__init__.py", line 14, in <module>
    from prefect.tasks.gcp.bigquery import (
  File "/usr/local/lib/python3.7/site-packages/prefect/tasks/gcp/bigquery.py", line 4, in <module>
    from google.cloud import bigquery
ImportError: cannot import name 'bigquery' from 'google.cloud' (unknown location)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 151, in <module>
    flows = cloudpickle_deserialization_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
    flows.append(cloudpickle.loads(flow_bytes))
  File "/usr/local/lib/python3.7/site-packages/prefect/tasks/gcp/__init__.py", line 24, in <module>
    ) from err
ImportError: Using `prefect.tasks.gcp` requires Prefect to be installed with the "gcp" extra.
I tried;
Copy code
storage = Docker(
    registry_url="<http://gcr.io/blah/|gcr.io/blah/>",
    image_name=name,
    image_tag="latest",
    build_kwargs={"buildargs":{"EXTRAS":"kubernetes,gcp"}}
)
With no luck šŸ˜ž
Is it perhaps a case of creating my own base image?
Or even just specifying 
google-cloud-bigquery
as a pip package?
j

Jan Marais

01/27/2021, 3:43 PM
How about 
Docker(..., python_dependencies=['google-cloud-bigquery'])
?
j

Josh Greenhalgh

01/27/2021, 3:43 PM
Yeah but that feels wrong?
Should be able to specify the extras I want no?
Its easy for this one since gcp extra only installs one library
But azure for example installs 3 in that situation I have to go to setup.py look what they are and specify them all to the python_dependencies
j

Jan Marais

01/27/2021, 3:46 PM
I see. Yeah I would also expect the 
build_kwargs
to handle setting the EXTRAS var in base image dockerfile
j

josh

01/27/2021, 3:47 PM
What happens if you install the extras from the prefect package like 
"prefect[gcp]"
?
j

Jan Marais

01/27/2021, 3:51 PM
have you tried 
build_kwargs={"EXTRAS": "kubernetes,gcp"}
?
Sorry, I'm just guessing here. Not qualified to help really. Only started playing a few days ago
j

Josh Greenhalgh

01/27/2021, 3:54 PM
I did try that (
build_kwargs
but the image being built doesn't have any ARGS its not actually building 
prefecthq/prefect
but a dockerfile that uses that as a base)
@josh As 
python_dependencies
?
I think that is likely to work but still feels a little weird
j

Jan Marais

01/27/2021, 4:06 PM
And if you manually set the base_image to prefecthq/prefect ?
j

Josh Greenhalgh

01/27/2021, 4:18 PM
That is the base image but 
EXTRAS
is a build arg to that which is not being built
j

Jan Marais

01/27/2021, 5:14 PM
I understand that, just from:
I did try that (
build_kwargs
Ā but the image being built doesn't have any ARGS its not actually buildingĀ 
prefecthq/prefect
Ā but a dockerfile that uses that as a base)
I thought you said prefecthq/prefect is not being built and therefore I suggested explicitly setting the base image to prefecthq/prefect with the EXTRAS as build args
j

Josh Greenhalgh

01/27/2021, 5:16 PM
If I have a dockerfile like;
Copy code
FROM blah

RUN some stuff
And blahs dockerfile is;
Copy code
FROM foo
ARG baz
Then when building the first its not possible to set the build arg baz - as far as I am aware?
blah doesn't get rebuilt - its just used as the base
j

Jan Marais

01/27/2021, 6:47 PM
Yes, you are right. Sorry, I should have thought it through before giving suggestions
Easiest then I bet is to take Josh's suggestion of adding prefect[gcp] to python_dependencies. Bit redundant but it won't do any harm. I would have to dig deeper into the source to determine if it is the best solution
29 Views
Powered by