@Marvin how to pull existing image in the

prefect.yaml

file? Iv'e found only a build section..

According to the Prefect documentation, if using CI/CD to build a Docker image outside of Prefect, you don't need to include push or pull steps in the

prefect.yaml

file. Instead, you can reference the image directly in your work pool configuration. By default, Prefect work pools that use containers reference the

2-latest

image, but you can specify a different image when creating the work pool. The specified image can also be overridden in individual deployment specifications. For further details, check the following guides: - Prefect Docker Guide - Prefect Storage Guide Please note, you will need to configure the work pool or deployment specification to use the existing Docker image accordingly. The documentation should provide specific instructions on how to set this up in your Prefect environment.

hi @Netanel Malka it depends how you want to do it • flow code in the image? ◦ just specify your

image

on your work pool, use

set_working_directory

in your pull step • flow code somewhere else? (e.g. github) ◦ still set your (base)

image

in your work pool, use something like

git_clone

in your pull step

@Nate Thanks. I want to use flow code in the image. I am trying also to understand if the following is a bug. I ran (locally with local prefect server):

prefect work-pool create test_pool --provision-infra

and get:

File "/home/netanel/.cache/pypoetry/virtualenvs/upstream-data-engineering-ptQzHj37-py3.10/lib/python3.10/site-packages/httpx/_models.py", line 762, in json
    return jsonlib.loads(self.content, **kwargs)
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

that does seem like some sort of bug, is that the full stack trace? this looks like the error you get when you try to read an empty JSON file 🙂

Full stack:

Set 'PREFECT_API_URL' to '<http://localhost:4200>'.
Updated profile 'default'.
netanel@netanel-ThinkPad-T14-Gen-2i ~/p/upstream-data-engineering (release-candidate)> prefect work-pool create -t process  test --provision-infra
Traceback (most recent call last):
  File "/home/netanel/.cache/pypoetry/virtualenvs/upstream-data-engineering-ptQzHj37-py3.10/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/home/netanel/.cache/pypoetry/virtualenvs/upstream-data-engineering-ptQzHj37-py3.10/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 259, in coroutine_wrapper
    return call()
  File "/home/netanel/.cache/pypoetry/virtualenvs/upstream-data-engineering-ptQzHj37-py3.10/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 431, in __call__
    return self.result()
  File "/home/netanel/.cache/pypoetry/virtualenvs/upstream-data-engineering-ptQzHj37-py3.10/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 317, in result
    return self.future.result(timeout=timeout)
  File "/home/netanel/.cache/pypoetry/virtualenvs/upstream-data-engineering-ptQzHj37-py3.10/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 178, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/netanel/.cache/pypoetry/virtualenvs/upstream-data-engineering-ptQzHj37-py3.10/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 388, in _run_async
    result = await coro
  File "/home/netanel/.cache/pypoetry/virtualenvs/upstream-data-engineering-ptQzHj37-py3.10/lib/python3.10/site-packages/prefect/cli/work_pool.py", line 122, in create
    await client.read_work_pool(work_pool_name=name)
  File "/home/netanel/.cache/pypoetry/virtualenvs/upstream-data-engineering-ptQzHj37-py3.10/lib/python3.10/site-packages/prefect/client/orchestration.py", line 2552, in read_work_pool
    return pydantic.parse_obj_as(WorkPool, response.json())
  File "/home/netanel/.cache/pypoetry/virtualenvs/upstream-data-engineering-ptQzHj37-py3.10/lib/python3.10/site-packages/httpx/_models.py", line 764, in json
    return jsonlib.loads(self.content, **kwargs)
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
An exception occurred.