Oscar Björhn
11/01/2022, 3:04 PMjawnsy
11/01/2022, 3:07 PMRob Freedy
11/01/2022, 3:09 PMNate
11/01/2022, 3:30 PMJohn Mizerany
11/01/2022, 3:34 PMAndrew Huang
11/01/2022, 4:26 PMEsdras Lopes Nani
11/01/2022, 4:29 PMJosh
11/01/2022, 4:35 PMAnthony Head
11/01/2022, 4:44 PMJesper Bruun Hansen
11/08/2022, 8:56 PMprefecthq/prefect:2-latest
as base image, that they can run locally? I’m having hard time defining the ENTRYPOINT
Nelson Griffiths
01/27/2023, 2:01 PMNelson Griffiths
05/06/2023, 12:15 PMprefect-docker
?Nelson Griffiths
05/11/2023, 5:44 PMJane Jeong
05/26/2023, 9:07 PMYulius
06/29/2023, 6:31 AMmerlin
07/04/2023, 3:07 PMprefect_docker.projects.steps.build_docker_image
, etc.
I am able to successfully deploy the toy log_flow.py
flow on local process, no problem.
Docker desktop is installed, the image is available locally as well as from dockerhub prefecthq/prefect: 2.10.13-python3.11
. The deployment doesn't need to build the image, just run the container.Toby Coleman
07/20/2023, 1:40 PMA
uses my-org/image1:latest
and deployment B
uses my-org/image2:latest
, then these cannot share a work pool, instead I will need to create separate pools with different images configured on each one?Aurko Routh
08/08/2023, 6:06 PMJose Neto
08/10/2023, 4:38 PMraise cls(e, response=response, explanation=explanation) from e
docker.errors.APIError: 400 Client Error for <http+docker://localhost/v1.43/containers/create?name=zealous-marten>: Bad Request ("cannot share the host's network namespace when user namespaces are enabled"
16:29:57.583 | INFO | prefect.agent - Completed submission of flow run '9b0d24b2-0584-4229-9d9d-3f8b754b9e28'
16:29:57.622 | INFO | prefect.agent - Reported flow run '9b0d24b2-0584-4229-9d9d-3f8b754b9e28' as crashed: Flow run could not be submitted to infrastructure
Max Eggers
09/03/2023, 2:20 PMJane Jeong
09/21/2023, 5:07 PMSophia Alice
09/26/2023, 9:22 PMDominic Tarro
01/02/2024, 6:14 PMDockerRegistryCredentials
to login with ECR? On the command line I would pass "AWS" as the username and the token from aws ecr get-login-password
to STDIN, but that's not really an option here.
I see a few PRs/issues related to it (notably docker-py, prefect), but no examples for the Block. docker-py
's login doesn't have any native code for authenticating with AWS, and the issue I cited suggests people write their own authentication. prefect-docker
just calls the client's login method.Kevin
01/04/2024, 12:20 AMDavid Backx
01/22/2024, 6:52 PMprefect.deployment.steps.pip_install_requirements
step.
Now the problem goes over the package prefect_soda_core which i have in my requirements. But when I google this i'm pretty sure the package is prefect-soda-core
not sure if there is really a big difference.
pull:
- prefect.deployments.steps.git_clone:
repository: <https://github.com/datarootsio/xmas-soda-prefect-duckdb.git>
branch: david/docker_deployment
access_token: '{{ prefect.blocks.secret.sodagithubaccesstoken }}'
- prefect.deployments.steps.pip_install_requirements:
requirements_file: requirements.txt
stream_output: true
deployments:
- name: soda_docker_local
version:
tags: []
description:
entrypoint: flows/schedule.py:run_soda_scan
parameters: {}
work_pool:
name: soda_docker
work_queue_name:
job_variables: {}
schedule:
is_schedule_active: true
This is my prefect.yml
file.
Its probably complaining about these imports
from prefect_soda_core.soda_configuration import SodaConfiguration
from prefect_soda_core.sodacl_check import SodaCLCheck
from prefect_soda_core.tasks import soda_scan_execute
Thanks in advance for looking into this problem, I was also looking into trying to add this package to EXTRA_PIP_PACKAGES
but not sure where to put this for a local docker workpool.
Added my requirements.txt
to the thread 😁Amruth VVKP
01/24/2024, 7:38 AMdocker-compose.yml
is starting up a docker type Prefect work pool which is resulting in this error -
2024-01-24 07:35:32 07:35:32.358 | ERROR | prefect.flow_runs.worker - Failed to submit flow run '7639a755-0d52-4d45-874d-ca7aa17e5f81' to infrastructure.
2024-01-24 07:35:32 Traceback (most recent call last):
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
2024-01-24 07:35:32 httplib_response = self._make_request(
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
2024-01-24 07:35:32 conn.request(method, url, **httplib_request_kw)
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
2024-01-24 07:35:32 super(HTTPConnection, self).request(method, url, body=body, headers=headers)
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/http/client.py", line 1294, in request
2024-01-24 07:35:32 self._send_request(method, url, body, headers, encode_chunked)
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/http/client.py", line 1340, in _send_request
2024-01-24 07:35:32 self.endheaders(body, encode_chunked=encode_chunked)
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/http/client.py", line 1289, in endheaders
2024-01-24 07:35:32 self._send_output(message_body, encode_chunked=encode_chunked)
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/http/client.py", line 1048, in _send_output
2024-01-24 07:35:32 self.send(msg)
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/http/client.py", line 986, in send
2024-01-24 07:35:32 self.connect()
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/docker/transport/unixconn.py", line 27, in connect
2024-01-24 07:35:32 sock.connect(self.unix_socket)
2024-01-24 07:35:32 FileNotFoundError: [Errno 2] No such file or directory
2024-01-24 07:35:32
2024-01-24 07:35:32 During handling of the above exception, another exception occurred:
2024-01-24 07:35:32
2024-01-24 07:35:32 Traceback (most recent call last):
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
2024-01-24 07:35:32 resp = conn.urlopen(
2024-01-24 07:35:32 ^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 799, in urlopen
2024-01-24 07:35:32 retries = retries.increment(
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 550, in increment
2024-01-24 07:35:32 raise six.reraise(type(error), error, _stacktrace)
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/urllib3/packages/six.py", line 769, in reraise
2024-01-24 07:35:32 raise value.with_traceback(tb)
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
2024-01-24 07:35:32 httplib_response = self._make_request(
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
2024-01-24 07:35:32 conn.request(method, url, **httplib_request_kw)
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
2024-01-24 07:35:32 super(HTTPConnection, self).request(method, url, body=body, headers=headers)
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/http/client.py", line 1294, in request
2024-01-24 07:35:32 self._send_request(method, url, body, headers, encode_chunked)
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/http/client.py", line 1340, in _send_request
2024-01-24 07:35:32 self.endheaders(body, encode_chunked=encode_chunked)
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/http/client.py", line 1289, in endheaders
2024-01-24 07:35:32 self._send_output(message_body, encode_chunked=encode_chunked)
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/http/client.py", line 1048, in _send_output
2024-01-24 07:35:32 self.send(msg)
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/http/client.py", line 986, in send
2024-01-24 07:35:32 self.connect()
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/docker/transport/unixconn.py", line 27, in connect
2024-01-24 07:35:32 sock.connect(self.unix_socket)
2024-01-24 07:35:32 urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
2024-01-24 07:35:32
2024-01-24 07:35:32 During handling of the above exception, another exception occurred:
2024-01-24 07:35:32
2024-01-24 07:35:32 Traceback (most recent call last):
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
2024-01-24 07:35:32 return self.version(api_version=False)["ApiVersion"]
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/docker/api/daemon.py", line 181, in version
2024-01-24 07:35:32 return self._result(self._get(url), json=True)
2024-01-24 07:35:32 ^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/docker/utils/decorators.py", line 46, in inner
2024-01-24 07:35:32 return f(self, *args, **kwargs)
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 237, in _get
2024-01-24 07:35:32 return self.get(url, **self._set_request_timeout(kwargs))
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
2024-01-24 07:35:32 return self.request("GET", url, **kwargs)
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
2024-01-24 07:35:32 resp = self.send(prep, **send_kwargs)
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
2024-01-24 07:35:32 r = adapter.send(request, **kwargs)
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 501, in send
2024-01-24 07:35:32 raise ConnectionError(err, request=request)
2024-01-24 07:35:32 requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
2024-01-24 07:35:32
2024-01-24 07:35:32 During handling of the above exception, another exception occurred:
2024-01-24 07:35:32
2024-01-24 07:35:32 Traceback (most recent call last):
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/prefect_docker/worker.py", line 498, in _get_client
2024-01-24 07:35:32 docker_client = docker.from_env()
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/docker/client.py", line 96, in from_env
2024-01-24 07:35:32 return cls(
2024-01-24 07:35:32 ^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/docker/client.py", line 45, in __init__
2024-01-24 07:35:32 self.api = APIClient(*args, **kwargs)
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 197, in __init__
2024-01-24 07:35:32 self._version = self._retrieve_server_version()
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
2024-01-24 07:35:32 raise DockerException(
2024-01-24 07:35:32 docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
2024-01-24 07:35:32
2024-01-24 07:35:32 The above exception was the direct cause of the following exception:
2024-01-24 07:35:32
2024-01-24 07:35:32 Traceback (most recent call last):
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 896, in _submit_run_and_capture_errors
2024-01-24 07:35:32 result = await self.run(
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/prefect_docker/worker.py", line 417, in run
2024-01-24 07:35:32 container, created_event = await run_sync_in_worker_thread(
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
2024-01-24 07:35:32 return await anyio.to_thread.run_sync(
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
2024-01-24 07:35:32 return await get_asynclib().run_sync_in_worker_thread(
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
2024-01-24 07:35:32 return await future
2024-01-24 07:35:32 ^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
2024-01-24 07:35:32 result = context.run(func, *args)
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/prefect_docker/worker.py", line 548, in _create_and_start_container
2024-01-24 07:35:32 docker_client = self._get_client()
2024-01-24 07:35:32 ^^^^^^^^^^^^^^^^^^
2024-01-24 07:35:32 File "/usr/local/lib/python3.11/site-packages/prefect_docker/worker.py", line 501, in _get_client
2024-01-24 07:35:32 raise RuntimeError("Could not connect to Docker.") from exc
2024-01-24 07:35:32 RuntimeError: Could not connect to Docker.
2024-01-24 07:35:32 07:35:32.364 | INFO | prefect.flow_runs.worker - Completed submission of flow run '7639a755-0d52-4d45-874d-ca7aa17e5f81'
2024-01-24 07:35:32 07:35:32.388 | INFO | prefect.flow_runs.worker - Reported flow run '7639a755-0d52-4d45-874d-ca7aa17e5f81' as crashed: Flow run could not be submitted to infrastructure
Petko
01/24/2024, 12:22 PMRicardo Schalch
01/29/2024, 6:18 PMPiotr Borkowski
02/02/2024, 11:25 AMif __name__ == "__main__":
deploy(
pb_flow.from_source(
entrypoint=entrypoint,
source="/python/pb_flow.py"
).to_deployment(
name="pb_flow_docker",
tags=default_snowflake_tags + ["test"],
enforce_parameter_schema=True,
),
print_next_steps_message=False,
work_pool_name=default_work_pool,
image="<http://362403473208.dkr.ecr.eu-west-1.amazonaws.com/pbtest:latest|362403473208.dkr.ecr.eu-west-1.amazonaws.com/pbtest:latest>",
push=False
David Backx
02/21/2024, 3:18 PMFROM prefecthq/prefect:2-python3.11
COPY requirements.txt .
RUN pip install -r requirements.txt --trusted-host <http://pypi.python.org|pypi.python.org> --no-cache-dir
RUN pip install -i <https://pypi.cloud.soda.io> soda-snowflake
RUN pip install -i <https://pypi.cloud.soda.io> soda-duckdb
COPY flows /opt/prefect/flows
CMD ["python", "flows/migration_flow.py"]
This is my docker image and this is my flow
from prefect_soda_core.tasks import soda_scan_execute
from prefect import flow, task
from prefect.artifacts import create_markdown_artifact
from prefect_soda_core.soda_configuration import SodaConfiguration
from prefect_soda_core.sodacl_check import SodaCLCheck
from prefect.blocks.system import Secret
from jinja2 import Environment, FileSystemLoader
import duckdb
from pathlib import Path
import json
motherduck_secret_block = Secret.load("sodamotherducktoken")
soda_id_secret_block = Secret.load("soda-api-key-id")
soda_secret_block = Secret.load("soda-api-key-secret")
snowflake_pass_block = Secret.load("snowflake-pass")
global_con = duckdb.connect(
f"md:my_db?motherduck_token={motherduck_secret_block.get()}"
)
def run_migration(file_name: Path):
"""Runs the data contract and stores the results in a file
Args:
file_name (str): The name of the file containing the data contract
"""
configuration_yaml_path = (
Path("./soda/config/configuration.yml").absolute().as_posix()
)
soda_config_block = SodaConfiguration(
configuration_yaml_path=configuration_yaml_path
)
soda_check_block = SodaCLCheck(sodacl_yaml_path=file_name)
scan_results = soda_scan_execute(
data_source_name="students_md",
configuration=soda_config_block,
checks=soda_check_block,
variables={
"TOKEN": motherduck_secret_block.get(),
"API_KEY": soda_id_secret_block.get(),
"API_SECRET": soda_secret_block.get(),
"SNOWPWD": snowflake_pass_block.get(),
},
verbose=True,
return_scan_result_file_content=False,
)
create_markdown_artifact_task(scan_results)
def json_to_markdown(data: str):
"""Converts json to markdown
Args:
data (str): json data
Returns:
str: markdown data
"""
data = json.loads(data)
failed_checks = []
for check in data.get("checks", []):
if check["outcome"] == "fail":
check_info = {
"name": check["name"],
"outcome": check["outcome"],
"check_value": check["diagnostics"]["value"],
"source": check["dataSource"],
"source_table": check["table"],
"column": check["column"],
}
failed_checks.append(check_info)
amount_failed = len(failed_checks)
path = Path("./markdown").absolute().as_posix()
environment = Environment(loader=FileSystemLoader(path))
template = environment.get_template("soda_report.md")
markdown = template.render(
data=data, failed_checks=failed_checks, amount_failed=amount_failed
)
return markdown
@task(name="create_markdown_artifact_migration")
def create_markdown_artifact_task(scan_output: str):
"""Builds a markdown artifact from the data contract results
Args:
data (str): file containing the data contract results
"""
markdown = json_to_markdown(scan_output)
create_markdown_artifact(
key="data-quality-report-data-contract",
markdown=markdown,
description="Soda Data Contract Results",
)
@flow
def migration_flow() -> bool:
"""Prefect flow for executing data contract
Returns:
bool: returns True
"""
raw_path = Path("./soda/checks/migration_checks.yml").absolute().as_posix()
run_migration(file_name=raw_path)
return True
if __name__ == "__main__":
migration_flow()
But i keep getting this error
TypeError: cannot pickle 'classmethod' object
It complains about my first import but this works fine when I deploy this on prefect with prefect deploy
Thanks in advance!