matt forbes
09/18/2020, 9:02 PMmatt forbes
09/18/2020, 11:56 PMJovan Visnjic
09/21/2020, 1:54 PMLocalResult
. But then we'd like to register the same workflow on the server and run it there on a large scale, using PrefectResult
and cache to the database. We're struggling to get the LocalResult
and PrefectResult
behave in the same way, so we could just switch between one and the other. Does this make sense at all - to have a switch between offline and server run? Or should one result class handle all cases, perhaps a custom one?Julian
09/21/2020, 5:15 PMmot s
09/22/2020, 8:01 PMsark
09/23/2020, 9:07 AMwith Flow('flow', storage=docker_storage) as flow:
passs
using docker storage
docker_storage=Docker(registry_url="<http://gcr.io/repo|gcr.io/repo>", dockerfile='Dockerfile')
where the dockerfile is just
FROM prefecthq/prefect:all_extras-0.13.7
and the agent logs show
[2020-09-23 09:01:30,367] INFO - agent | Pulling image <http://gcr.io/repo/flow:2020-09-23t08-59-44-707304-00-00|gcr.io/repo/flow:2020-09-23t08-59-44-707304-00-00>...
ERROR: gcloud failed to load: No module named '_sqlite3'
gcloud_main = _import_gcloud_main()
import googlecloudsdk.gcloud_main
from googlecloudsdk.api_lib.iamcredentials import util as iamcred_util
from googlecloudsdk.core.credentials import transports
from googlecloudsdk.core.credentials import http
from googlecloudsdk.core.credentials import creds as core_creds
import sqlite3
from sqlite3.dbapi2 import *
from _sqlite3 import *
This usually indicates corruption in your gcloud installation or problems with your Python interpreter.
Please verify that the following is the path to a working Python 2.7 or 3.5+ executable:
/opt/prefect/.local/share/virtualenvs/prefect-Ta6MxEs8/bin/python3
If it is not, please set the CLOUDSDK_PYTHON environment variable to point to a working Python 2.7 or 3.5+ executable.
If you are still experiencing problems, please reinstall the Cloud SDK using the instructions here:
<https://cloud.google.com/sdk/>
so this seems like it is the agent not having proper GCR auth? how can i configure the docker agent for this? is there a way to configure the image of the agent like it is possible to configure the image for the flow?Greg
09/23/2020, 3:19 PMGreg
09/23/2020, 3:40 PMcharmatski
09/25/2020, 2:48 PMNo such file or directory: 'docker-compose': 'docker-compose'
Should docker-compose not be installed inside the image and this be a read-to go dev environment?Gitanshu Sardana
09/25/2020, 10:54 PMGitanshu Sardana
09/25/2020, 10:55 PMVipul
09/27/2020, 10:39 AMale
09/28/2020, 3:01 PMSteve Taylor
09/28/2020, 8:30 PMHagai Arad
09/29/2020, 2:29 PMAna
10/01/2020, 1:50 PMMatt Allen
10/01/2020, 3:57 PMCharles Leung
10/01/2020, 5:43 PMEmma Willemsma
10/01/2020, 8:15 PMAn error occurred (InvalidParameterException) when calling the RegisterTaskDefinition operation: Invalid 'cpu' setting for task.
The entrypoint we're using for the agent is:
["prefect","agent","start","fargate","cpu=256","memory=1024","networkConfiguration=$NETWORK_CONFIGURATION"],
Does anyone know what we should we be setting for cpu
to get it working?Luke
10/01/2020, 10:25 PMCharles Leung
10/02/2020, 7:00 PMBruce Haggerty
10/04/2020, 7:06 PMAnish Chhaparwal
10/05/2020, 5:50 PMprefect agent start docker
i am able to register the flow to the UI but it always remains in pending state and never gets executed(refer images attached).
Can someone explain where i am going wrong?Darren Forsdike
10/07/2020, 9:33 AMNewskooler
10/07/2020, 1:08 PMprefect server start
via a running docker image, can you please help me fix an error I keep getting : /
I summarized it here: https://github.com/PrefectHQ/prefect/discussions/3449tom
10/08/2020, 5:27 PMprefect server start
) and my flows are executing via the agent (prefect agent start
). I was wondering how I could get them to run on Dask? I have dask-scheduler
and a few dask-worker
processes running. When I run my flow using flow.run(executor=DaskExecutor(address="<tcp://xxx:8786>"))
everything works well, but when I run the flow via prefect server’s UI it doesn’t run on Dask. What am I missing? I tried running the server and agent with export PREFECT__ENGINE__EXECUTOR__DEFAULT_CLASS="prefect.engine.executors.DaskExecutor"
and export PREFECT__ENGINE__EXECUTOR__DASK__ADDRESS="<tcp://xxx:8786>"
set but this didn’t help.DJ Erraballi
10/08/2020, 10:41 PMDJ Erraballi
10/08/2020, 10:42 PMDJ Erraballi
10/08/2020, 10:42 PMJ.R. Carneiro
10/08/2020, 11:02 PM