Michael Ulin
12/14/2021, 9:45 PMKevin Kho
12/14/2021, 9:47 PMMichael Ulin
12/14/2021, 9:56 PMKevin Kho
12/14/2021, 10:02 PMMichael Ulin
12/14/2021, 10:03 PMcoiled.create_software_environment(
name="prefect-executor-v2",
container="njgreg/greg-smith-prefect-executor",
pip=['numpy','google-api-python-client']
)
google-cloud
with pip too - I get the same error'google-auth','google-auth-httplib2', 'google-api-python-client'
Kevin Kho
12/14/2021, 10:17 PMMichael Ulin
12/14/2021, 10:17 PMKevin Kho
12/14/2021, 10:20 PMMichael Ulin
12/14/2021, 10:23 PMKevin Kho
12/14/2021, 10:24 PMMichael Ulin
12/14/2021, 10:25 PMKevin Kho
12/14/2021, 10:37 PMMichael Ulin
12/14/2021, 10:38 PMimport prefect
from prefect import task, Flow
from prefect.executors import DaskExecutor
import coiled
from prefect.storage import Docker
coiled.create_software_environment(
name="prefect-executor-v3",
container="njgreg/greg-smith-prefect-executor",
pip=['numpy','google-auth','google-auth-httplib2', 'google-api-python-client']
)
@task
def hello_task():
from google.oauth2.service_account import Credentials
logger = prefect.context.get("logger")
<http://logger.info|logger.info>("Hello world!")
flow = Flow("test-coiled-execution", tasks=[hello_task])
executor = DaskExecutor(
cluster_class=coiled.Cluster,
cluster_kwargs={
"software": "prefect-executor-v3",
"shutdown_on_close": False,
"name": "prefect-executor",
},
)
flow.executor = executor
flow.storage = Docker(registry_url="",
python_dependencies=['coiled','pandas','s3fs','blosc','lz4','google-auth','google-auth-httplib2', 'google-api-python-client'])
# flow.result = gcs_result
flow.register('tester')
# flow.result = gcs_result
I get the import error when trying to trigger it from the Web UIKevin Kho
12/14/2021, 10:56 PMgoogle-cloud-storage
?Michael Ulin
12/15/2021, 12:19 AMKevin Kho
12/15/2021, 1:40 AMMichael Ulin
12/15/2021, 1:51 AMfrom google.oauth2.service_account import Credentials
Kevin Kho
12/15/2021, 1:54 AMwhere python
Michael Ulin
12/15/2021, 1:56 AMKevin Kho
12/15/2021, 2:02 AMMichael Ulin
12/15/2021, 2:03 AMUnexpected error: ModuleNotFoundError("No module named 'google'")
Traceback (most recent call last):
File "/opt/conda/envs/coiled/lib/python3.8/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/opt/conda/envs/coiled/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 926, in get_task_run_state
result = self.result.write(value, **formatting_kwargs)
File "/opt/conda/envs/coiled/lib/python3.8/site-packages/prefect/engine/results/gcs_result.py", line 77, in write
self.gcs_bucket.blob(new.location).upload_from_string(binary_data)
File "/opt/conda/envs/coiled/lib/python3.8/site-packages/prefect/engine/results/gcs_result.py", line 39, in gcs_bucket
from prefect.utilities.gcp import get_storage_client
File "/opt/conda/envs/coiled/lib/python3.8/site-packages/prefect/utilities/gcp.py", line 6, in <module>
from google.oauth2.service_account import Credentials
ModuleNotFoundError: No module named 'google'
Kevin Kho
12/15/2021, 2:31 AM"prefect[google]"
under the pip
section of the Coiled software env?Michael Ulin
12/15/2021, 5:25 PMimport pip
pip.main(['install', 'google-auth-oauthlib'])
pip.main(['install', 'google-api-python-client'])
pip.main(['install', 'google-auth-httplib2'])
Kevin Kho
12/15/2021, 5:32 PMMichael Ulin
12/15/2021, 5:57 PMKevin Kho
12/15/2021, 6:03 PMMichael Ulin
12/16/2021, 7:05 PMinstalled_packages = pkg_resources.working_set
installed_packages_list = sorted(["%s==%s" % (i.key, i.version)
for i in installed_packages])
<http://prefect.context.logger.info|prefect.context.logger.info>(installed_packages_list)
['aiobotocore==2.0.1', 'aiohttp==3.8.1', 'aioitertools==0.8.0', 'aiosignal==1.2.0', 'appdirs==1.4.4', 'asn1crypto==1.4.0', 'async-timeout==4.0.1', 'attrs==21.2.0', 'backcall==0.2.0', 'backoff==1.11.1', 'backports.functools-lru-cache==1.6.4', 'blosc==1.10.2', 'bokeh==2.4.2', 'boto3==1.19.8', 'botocore==1.22.8', 'brotlipy==0.7.0', 'certifi==2021.10.8', 'cffi==1.15.0', 'charset-normalizer==2.0.9', 'click==8.0.3', 'cloudpickle==2.0.0', 'coiled==0.0.58', 'colorama==0.4.4', 'commonmark==0.9.1', 'croniter==0.3.36', 'cryptography==36.0.0', 'cytoolz==0.11.2', 'dask==2021.11.2', 'dataclasses==0.8', 'decorator==5.1.0', 'distributed==2021.11.2', 'docker-pycreds==0.4.0', 'docker==5.0.3', 'frozenlist==1.2.0', 'fsspec==2021.11.1', 'future==0.18.2', 'heapdict==1.0.1', 'idna==3.1', 'importlib-resources==5.4.0', 'ipython==7.30.1', 'jedi==0.18.1', 'jinja2==3.0.3', 'jmespath==0.10.0', 'locket==0.2.0', 'lz4==3.1.10', 'markupsafe==2.0.1', 'marshmallow-oneofschema==3.0.1', 'marshmallow==3.14.1', 'matplotlib-inline==0.1.3', 'msgpack==1.0.3', 'multidict==5.2.0', 'mypy-extensions==0.4.3', 'natsort==8.0.0', 'numpy==1.21.4', 'olefile==0.46', 'packaging==21.3', 'pandas==1.3.4', 'parso==0.8.3', 'partd==1.2.0', 'pendulum==2.1.2', 'pexpect==4.8.0', 'pickleshare==0.7.5', 'pillow==8.4.0', 'pip==21.3.1', 'prefect==0.15.10', 'prompt-toolkit==3.0.23', 'psutil==5.8.0', 'ptyprocess==0.7.0', 'pycparser==2.21', 'pygments==2.10.0', 'pyopenssl==21.0.0', 'pyparsing==3.0.6', 'pysocks==1.7.1', 'python-box==5.4.1', 'python-dateutil==2.8.2', 'python-slugify==5.0.2', 'pytz==2021.3', 'pytzdata==2020.1', 'pyyaml==6.0', 'requests==2.26.0', 'rich==10.15.2', 'ruamel.yaml.clib==0.2.6', 'ruamel.yaml==0.17.17', 's3fs==2021.11.1', 's3transfer==0.5.0', 'setuptools==59.4.0', 'six==1.16.0', 'sortedcontainers==2.4.0', 'tabulate==0.8.9', 'tblib==1.7.0', 'text-unidecode==1.3', 'toml==0.10.2', 'toolz==0.11.2', 'tornado==6.1', 'traitlets==5.1.1', 'typing-extensions==4.0.1', 'unidecode==1.3.2', 'urllib3==1.26.7', 'wcwidth==0.2.5', 'websocket-client==1.2.3', 'wheel==0.37.0', 'wrapt==1.13.3', 'yarl==1.7.2', 'zict==2.0.0', 'zipp==3.6.0']
Kevin Kho
12/16/2021, 7:09 PMMichael Ulin
12/16/2021, 7:22 PMKevin Kho
12/16/2021, 7:28 PMMichael Ulin
12/16/2021, 7:29 PMKevin Kho
12/16/2021, 7:31 PMAnna Geller
12/16/2021, 7:37 PMcoiled.create_software_environment(
name="prefect-executor-v3",
container="njgreg/greg-smith-prefect-executor", # comment this line out
# put all dependencies into pip list below:
pip=['numpy','google-auth','google-auth-httplib2', 'google-api-python-client']
)
I think taking it slowly and separating dependencies, then potentially building your own Prefect image with a custom Dockerfile, will solve the issue.Michael Ulin
12/17/2021, 12:55 AM