Mark McDonald
01/22/2021, 9:46 PMJosh
01/22/2021, 10:25 PMS3Download
and GCSUpload
tasks.
My suspicion is that the flow is not releasing the memory of the files being transferred. Is there any way to ensure the file contents are being released from memory?Giovanni
01/23/2021, 1:49 AMJacob Lester
01/23/2021, 2:02 AMAjith Kumara Beragala Acharige Lal
01/23/2021, 2:23 AMGeorge Shishorin
01/24/2021, 10:04 PMBorut Hafner
01/25/2021, 1:46 PMFelix Schran
01/25/2021, 1:59 PMJan Marais
01/25/2021, 2:51 PMciaran
01/25/2021, 4:24 PMLocalDaskExecutor
) - Does anyone know how I can visualise the Dask Diagnostics when running Prefect flows?
I know I can look at the individual mapped tasks, but I just wanted to see if there's a similar visualisation to how Dask does itBraun Reyes
01/25/2021, 6:31 PMPhilip MacMenamin
01/25/2021, 7:56 PMrequests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd0e8943640>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
^C[2021-01-25 19:55:03,534] INFO - agent | Keyboard Interrupt received: Agent is shutting down.
Fina Silva-Santisteban
01/25/2021, 11:36 PMFROM python:3.7
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
My flow like this:
(...)
storage = Docker(dockerfile="../Dockerfile")
with Flow("Generate Report", result=LocalResult(), storage=storage) as flow:
(...)
When I try to register the flow, it seems to have issues with finding the requirements.txt
file needed to create the image:
[2021-01-25 15:21:03-0800] INFO - prefect.Docker | Building the flow's Docker storage...
Step 1/14 : FROM python:3.7
---> ca194d6afe58
Step 2/14 : WORKDIR /app
---> Using cache
---> a729913b700e
Step 3/14 : COPY requirements.txt .
COPY failed: stat /var/lib/docker/tmp/docker-builder883681743/requirements.txt: no such file or directory
Traceback (most recent call last):
File "/Users/fina/Documents/github/concierge_iro_reporting/etl/register_all_flows.py", line 53, in <module>
register_locally()
File "/Users/fina/Documents/github/concierge_iro_reporting/etl/register_all_flows.py", line 21, in register_locally
report_lt_target.register(project_name=project_name)
File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/core/flow.py", line 1675, in register
idempotency_key=idempotency_key,
File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/client/client.py", line 783, in register
serialized_flow = flow.serialize(build=build) # type: Any
File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/core/flow.py", line 1450, in serialize
storage = self.storage.build() # type: Optional[Storage]
File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/storage/docker.py", line 363, in build
self._build_image(push=push)
File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/storage/docker.py", line 431, in _build_image
"Your docker image failed to build! Your flow might have "
ValueError: Your docker image failed to build! Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
My folder is structure is like this:
App
-->prefect_flows
-->my_flow.py
requirements.txt
Dockerfile
Why is it looking for the requirements.txt inside /var/lib/docker/
instead of the App’s directory? What am I missing?
(The container runs normally when I use docker-compose to build and run the app)Adam Roderick
01/26/2021, 2:45 AMShyam Meduri
01/26/2021, 8:34 AMpsimakis
01/26/2021, 9:16 AMMarion
01/26/2021, 12:11 PMSeverin Ryberg [sevberg]
01/26/2021, 2:00 PMMarc Lipoff
01/26/2021, 3:27 PMBen Wolz
01/26/2021, 4:18 PMJeff Brainerd
01/26/2021, 8:42 PMon_start
arg in order to bootstrap django when our flow runs. I’m not seeing any replacement for that in the new RunConfig world. Did I miss something… or is there another pattern for this? Thanks!Amanda Wee
01/26/2021, 10:01 PMdask_cloudprovider.aws.FargateCluster
plus adaptive scaling) with the local agent. My thinking is that I could separately build a Docker image that has all the required flow dependencies (they're the same for all the flows) and upload that to ECR during CI/CD, and then the local agent can pull from ECR and initialise the Dask executor with this image, while also initialising it with the flows retrieved from S3 storage. Is this feasible?Henry
01/26/2021, 10:23 PMJim Crist-Harif
01/26/2021, 11:45 PMprefecthq/prefect
images via setting a EXTRA_PIP_PACKAGES
environment variable
• Includes several improvements to agents, notably including a new execution_role_arn
kwarg to both `ECSAgent`/`ECSRun` run config
Big thanks to our community contributors this week (Loïc Macherel and Thomas Baldwin) 👏
The full release notes can be found here: https://github.com/PrefectHQ/prefect/releases/tag/0.14.5.
Happy Engineering!Billy McMonagle
01/27/2021, 2:21 AMKubernetesRun
... is there a direct way to set metadata labels on the flow run job/pod, or is a custom job template necessary? I'm specifically interested in tagging my jobs so that datadog is able to collect metrics.Joyce Xu
01/27/2021, 3:03 AMStartFlowRun
task to build "flow-within-a-flow" pipelines, i.e. Flow A includes the StartFlowRun
task X, and task X starts Flow B. We would like to use the graphql API to track metadata on our flows. Is there a way to query flows such that for Flow B, we can see that it corresponds to task X, or at least, that it is originated from Flow A?Nabeel
01/27/2021, 8:13 AMVitaly Shulgin
01/27/2021, 3:14 PMStartFlowRun
, when I run it locally, everything is working fine, but, when it executed in k8s container, which is running by schedule it failsJosh Greenhalgh
01/27/2021, 3:40 PMSystem Version check: OK
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/tasks/gcp/__init__.py", line 14, in <module>
from prefect.tasks.gcp.bigquery import (
File "/usr/local/lib/python3.7/site-packages/prefect/tasks/gcp/bigquery.py", line 4, in <module>
from google.cloud import bigquery
ImportError: cannot import name 'bigquery' from 'google.cloud' (unknown location)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/prefect/healthcheck.py", line 151, in <module>
flows = cloudpickle_deserialization_check(flow_file_paths)
File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
flows.append(cloudpickle.loads(flow_bytes))
File "/usr/local/lib/python3.7/site-packages/prefect/tasks/gcp/__init__.py", line 24, in <module>
) from err
ImportError: Using `prefect.tasks.gcp` requires Prefect to be installed with the "gcp" extra.
I tried;
storage = Docker(
registry_url="<http://gcr.io/blah/|gcr.io/blah/>",
image_name=name,
image_tag="latest",
build_kwargs={"buildargs":{"EXTRAS":"kubernetes,gcp"}}
)
With no luck 😞Matthew Blau
01/27/2021, 4:32 PMFROM prefecthq/prefect:0.7.1-python3.6
or similar in the Dockerfile, which I have done so. From there I am needing to create a flow that takes this Dockerfile and builds the container and that is where my understanding is weaker and not so clear. I have @task decorations added to various bits of the integration that I am currently attempting to convert over to having Prefect handle the execution of. I am stuck with the how of writing the flow in order to have it work with this docker container. Am I needing a seperate flow.py that takes the Dockerfile of the container, build it, and run the tasks denoted by "@task" within the integration in order for this to be able to be orchestrated by Prefect? If so, how would I write the flow as an example? I feel like my understanding is flawed and would appreciate some help with this. For reference I am running 0.14.1 Thank you all in advance!