Does anyone know how to specify the prefect extras I need in the Docker storage? I am getting the fo...

Josh Greenhalgh

01/27/2021, 3:40 PM

Does anyone know how to specify the prefect extras I need in the Docker storage? I am getting the following build error;

Copy code

System Version check: OK
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/tasks/gcp/__init__.py", line 14, in <module>
    from prefect.tasks.gcp.bigquery import (
  File "/usr/local/lib/python3.7/site-packages/prefect/tasks/gcp/bigquery.py", line 4, in <module>
    from google.cloud import bigquery
ImportError: cannot import name 'bigquery' from 'google.cloud' (unknown location)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 151, in <module>
    flows = cloudpickle_deserialization_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
    flows.append(cloudpickle.loads(flow_bytes))
  File "/usr/local/lib/python3.7/site-packages/prefect/tasks/gcp/__init__.py", line 24, in <module>
    ) from err
ImportError: Using `prefect.tasks.gcp` requires Prefect to be installed with the "gcp" extra.

I tried;

Copy code

storage = Docker(
    registry_url="<http://gcr.io/blah/|gcr.io/blah/>",
    image_name=name,
    image_tag="latest",
    build_kwargs={"buildargs":{"EXTRAS":"kubernetes,gcp"}}
)

With no luck 😞

Josh Greenhalgh

01/27/2021, 3:41 PM

Is it perhaps a case of creating my own base image?

Josh Greenhalgh

01/27/2021, 3:41 PM

Or even just specifying

google-cloud-bigquery

as a pip package?

Jan Marais

01/27/2021, 3:43 PM

How about

Docker(..., python_dependencies=['google-cloud-bigquery'])

Josh Greenhalgh

01/27/2021, 3:43 PM

Yeah but that feels wrong?

Josh Greenhalgh

01/27/2021, 3:44 PM

Should be able to specify the extras I want no?

Josh Greenhalgh

01/27/2021, 3:44 PM

Its easy for this one since gcp extra only installs one library

Josh Greenhalgh

01/27/2021, 3:45 PM

But azure for example installs 3 in that situation I have to go to setup.py look what they are and specify them all to the python_dependencies

Jan Marais

01/27/2021, 3:46 PM

I see. Yeah I would also expect the

build_kwargs

to handle setting the EXTRAS var in base image dockerfile

josh

01/27/2021, 3:47 PM

What happens if you install the extras from the prefect package like

"prefect[gcp]"

Jan Marais

01/27/2021, 3:51 PM

have you tried

build_kwargs={"EXTRAS": "kubernetes,gcp"}

Jan Marais

01/27/2021, 3:52 PM

Sorry, I'm just guessing here. Not qualified to help really. Only started playing a few days ago

Josh Greenhalgh

01/27/2021, 3:54 PM

I did try that (

build_kwargs

but the image being built doesn't have any ARGS its not actually building

prefecthq/prefect

but a dockerfile that uses that as a base)

Josh Greenhalgh

01/27/2021, 3:55 PM

@josh As

python_dependencies

Josh Greenhalgh

01/27/2021, 3:55 PM

I think that is likely to work but still feels a little weird

Jan Marais

01/27/2021, 4:06 PM

And if you manually set the base_image to prefecthq/prefect ?

Josh Greenhalgh

01/27/2021, 4:18 PM

That is the base image but

EXTRAS

is a build arg to that which is not being built

Jan Marais

01/27/2021, 5:14 PM

I understand that, just from:

I did try that (
build_kwargs
but the image being built doesn't have any ARGS its not actually building
prefecthq/prefect
but a dockerfile that uses that as a base)

I thought you said prefecthq/prefect is not being built and therefore I suggested explicitly setting the base image to prefecthq/prefect with the EXTRAS as build args

Josh Greenhalgh

01/27/2021, 5:16 PM

If I have a dockerfile like;

Copy code

FROM blah

RUN some stuff

And blahs dockerfile is;

Copy code

FROM foo
ARG baz

Then when building the first its not possible to set the build arg baz - as far as I am aware?

Josh Greenhalgh

01/27/2021, 5:17 PM

blah doesn't get rebuilt - its just used as the base

Jan Marais

01/27/2021, 6:47 PM

Yes, you are right. Sorry, I should have thought it through before giving suggestions

Jan Marais

01/27/2021, 6:54 PM

Easiest then I bet is to take Josh's suggestion of adding prefect[gcp] to python_dependencies. Bit redundant but it won't do any harm. I would have to dig deeper into the source to determine if it is the best solution

85 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.