Brett Jurman
07/29/2021, 9:35 PMKrapi Shah
07/30/2021, 5:39 PMKurt Rhee
07/30/2021, 7:34 PMLawrence Finn
07/30/2021, 7:48 PMEric Mauser
07/30/2021, 10:44 PMTOMAS IGNACIO ACUÑA RUZ
08/02/2021, 2:03 PMSam Cook
08/03/2021, 3:48 PMLawrence Finn
08/03/2021, 5:45 PMMatt Klein
08/03/2021, 7:44 PMCancelledError
? Any clues on how we can configure Prefect or Dask to allow flows to run past the 12 hour mark?Lawrence Finn
08/03/2021, 8:30 PMAlfie
08/04/2021, 3:34 AMRyan Sattler
08/04/2021, 6:11 AMprefect agent kubernetes start
. When I kick off the hello world flow, this indeed starts a job container on my desktop k8s which appears to start up fine and have the values that I configured (eg correct image). However it stays in a “running” state for several minutes until it errors out as follows: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5455f07710>: Failed to establish a new connection: [Errno -3] Tempor
Here’s the code for my flow:
import prefect
from prefect import task, Flow
from prefect.run_configs import KubernetesRun
@task
def hello_task():
logger = prefect.context.get("logger")
<http://logger.info|logger.info>("Hello world!")
flow = Flow("hello-flow2", tasks=[hello_task])
# flow.run() We could run our flow locally using the flow's run method but we'll be running this from Cloud!
flow.run_config = KubernetesRun(
env={"PREFECT__LOGGING__LEVEL": "DEBUG"},
image="prefecthq/prefect:latest",
labels=[]
)
flow.register(project_name="tester")
Alexander van Eck
08/04/2021, 1:41 PMDaskExecutor
with a custom cluster_class
the logs in Prefect UI show me;
15:36:12 INFO DaskExecutor Creating a new Dask cluster with `dask_jobqueue.remote_slurm.RemoteSLURMCluster.with_http_api_client`...
15:36:13 INFO DaskExecutor The Dask dashboard is available at <http://10.240.3.45:8787/status>
But I know for a fact that RemoteSlurmCluster (by dask-jobqueue) also has logger.debug statement.
How do I make those logger statements show up in either the flow logs (in the agent) or in the prefect logger UI?Adam Everington
08/04/2021, 3:27 PMKurt Rhee
08/04/2021, 4:01 PMAlexander van Eck
08/04/2021, 4:06 PMprefect agent docker start --network host
where the network is added to the container but not able to use it as the primary interface to find PREFECT__SERVER__HOST
behind a private DNS.
https://github.com/PrefectHQ/prefect/issues/4840
Posting here for visibilityBrett Jurman
08/04/2021, 4:07 PMJoe McDonald
08/04/2021, 6:00 PMmutation {
create_flow_run(
input: {
flow_id: "f667e3f3-b5eb-4c52-8f90-36bc54633ac9",
parameters: "{\"file_id\": \"42556bd0-f495-11eb-a7a0-8c8590acd4d8\"}"
}
) {
id
}
}
Which gives this output. We are getting the same output from our code that runs against graphql endpoint to trigger a flow run and has been working fine for months on 0.14.15.
{
"errors": [
{
"message": "[{'extensions': {'path': '$.variableValues.insert_on_conflict.constraint', 'code': 'validation-failed'}, 'message': 'unexpected value \"ix_flow_run_idempotency_key_unique\" for enum: \\'flow_run_constraint\\''}]",
"locations": [
{
"line": 2,
"column": 3
}
],
"path": [
"create_flow_run"
],
"extensions": {
"code": "INTERNAL_SERVER_ERROR",
"exception": {
"message": "[{'extensions': {'path': '$.variableValues.insert_on_conflict.constraint', 'code': 'validation-failed'}, 'message': 'unexpected value \"ix_flow_run_idempotency_key_unique\" for enum: \\'flow_run_constraint\\''}]"
}
}
}
],
"data": {
"create_flow_run": null
}
}
Margaret Walter
08/04/2021, 8:09 PMRuslan
08/05/2021, 5:27 AMArran
08/05/2021, 9:42 AMPierre Monico
08/05/2021, 12:55 PMStorage
in particular.
My current project is using simple flow.run()
s, all packaged in a Docker container. When switching to Prefect Server for orchestration, I have to register flows and thus define a Storage
class, RunConfig
etc. It seems like I have to write a lot of “infrastructure” code into my repo and reference different systems etc. I am not sure how to properly encapsulate this / separate these concerns from my business logic.
I just want to be able to keep developing locally and use flow.run
, but then have all the needed registration, storage configuration etc taken care of during deployment.
I tried by simply adding a separate file in which I import the flow
objects and then configure them before I register them, but e.g. with Local()
storage and save_as_script=True
, I then need to provide a path to the original flow file so I am not sure all the flow config made in the separate file will be taken into account.
Long story short: I find the step from using core to orchestration very difficult to understand and to structure in terms of code - I have the feeling it’s kind of either-or.
Positive closing: my experience with Prefect has been amazing so far 🙂patrickd
08/05/2021, 2:57 PMfrom prefect import task, Flow
import random
from time import sleep
from prefect.run_configs import KubernetesRun
from prefect.executors import DaskExecutor
from prefect.storage import GitLab
@task
def inc(x):
sleep(random.random() / 10)
return x + 1
@task
def dec(x):
sleep(random.random() / 10)
return x - 1
@task
def add(x, y):
sleep(random.random() / 10)
return x + y
@task(name="sum")
def list_sum(arr):
return sum(arr)
with Flow("dask-test") as flow:
incs = inc.map(x=range(100))
decs = dec.map(x=range(100))
adds = add.map(x=incs, y=decs)
total = list_sum(adds)
flow.storage = GitLab(<git kwargs>)
flow.run_config = KubernetesRun(image="prefecthq/prefect:latest-python3.8")
flow.executor = DaskExecutor("<tcp://dask-scheduler:8786>")
flow.register(project_name="test")
When running this flow with a LocalDaskExecutor
, it works fine. In the Kubernetes cluster, however, the job is never marked as finished on the Prefect server. I can see the job execute to completion on the Dask scheduler UI, but the flow run goes indefinitely on Prefect. Any help would be greatly appreciated!Blake List
08/05/2021, 11:14 PMFailed to load and execute Flow's environment: ValueError('No flows found in file.')
Any ideas what I am doing wrong?John T
08/06/2021, 5:50 AMArran
08/06/2021, 8:04 AMprefect server start
keeps giving me this error. Is there somewere i can just get a copy of the docker-compose file? I’ve tried this on two VMs now
ERROR:
Can't find a suitable configuration file in this directory or any
parent. Are you in the right directory?
Supported filenames: docker-compose.yml, docker-compose.yaml
i had a look in the directory mentioned in the traceback for the docker-compose.yml and it is there, so I’m not sure why it isn’t being recognised
Traceback (most recent call last):
File "******/venv/lib/python3.8/site-packages/prefect/cli/server.py", line 608, in start
subprocess.check_call(
File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker-compose', 'pull']' returned non-zero exit status 1.
Alex Furrier
08/06/2021, 5:02 PMPayloadTooLargeError: request entity too large
at readStream (/apollo/node_modules/raw-body/index.js:155:17)
at getRawBody (/apollo/node_modules/raw-body/index.js:108:12)
at read (/apollo/node_modules/body-parser/lib/read.js:77:3)
at jsonParser (/apollo/node_modules/body-parser/lib/types/json.js:135:5)
at Layer.handle [as handle_request] (/apollo/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (/apollo/node_modules/express/lib/router/index.js:317:13)
at /apollo/node_modules/express/lib/router/index.js:284:7
at Function.process_params (/apollo/node_modules/express/lib/router/index.js:335:12)
at next (/apollo/node_modules/express/lib/router/index.js:275:10)
at cors (/apollo/node_modules/cors/lib/index.js:188:7)
Any idea where to start for debugging that?Serdar Tumgoren
08/06/2021, 5:43 PMRoss Rochford
08/09/2021, 9:23 AMRoss Rochford
08/09/2021, 11:14 AM