Praveen Chaudhary
04/27/2022, 10:25 AMPraveen Chaudhary
04/27/2022, 10:25 AMPraveen Chaudhary
04/27/2022, 10:25 AMSang Young Noh
04/27/2022, 11:40 AMBaris Cekic
04/27/2022, 12:30 PMKubernetesRun
. The flow is trying to download the py
file from a local git repo in the same local K8S cluster. But I got an SSL_VERIFICATION
error even though the url/host
for git storage
is HTTP
. Is there anyone having similar issue?Vivek Kaushal
04/27/2022, 12:33 PMCould not upgrade the database!
graphql_1 | Error: HTTPConnectionPool(host='hasura', port=3000): Max retries exceeded with url: /v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f25746773d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
apollo_1 | Checking GraphQL service at <http://graphql:4201/health> ...
towel_1 | {"severity": "ERROR", "name": "prefect-server.Lazarus", "message": "Unexpected error: ValueError([{'extensions': {'path': '$.selectionSet.flow_run', 'code': 'validation-failed'}, 'message': 'field \"flow_run\" not found in type: \\'query_root\\''}])", "exc_info": "Traceback (most recent call last):\n File \"/prefect-server/src/prefect_server/services/loop_service.py\", line 60, in run\n await self.run_once()\n File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 37, in run_once\n return await self.reschedule_flow_runs()\n File \"/prefect-server/src/prefect_server/services/towel/lazarus.py\", line 89, in reschedule_flow_runs\n limit=500,\n File \"/prefect-server/src/prefect_server/database/orm.py\", line 501, in get\n as_box=not apply_schema,\n File \"/prefect-server/src/prefect_server/database/hasura.py\", line 85, in execute\n as_box=as_box,\n File \"/prefect-server/src/prefect_server/utilities/graphql.py\", line 84, in execute\n raise ValueError(result[\"errors\"])\nValueError: [{'extensions': {'path': '$.selectionSet.flow_run', 'code': 'validation-failed'}, 'message': 'field \"flow_run\" not found in type: \\'query_root\\''}]"}
Jonathan Mathews
04/27/2022, 1:24 PMSash Stasyk
04/27/2022, 1:42 PMmap
call?Marc Lipoff
04/27/2022, 1:52 PMCreating a new Dask cluster with None.fargate_cluster...
. I'm running on prefect==0.15.6. My code is attached. The trouble is that it just hangs and I have no clue what it's "trying" to do.Billy McMonagle
04/27/2022, 3:27 PMMatthew Roeschke
04/27/2022, 3:42 PMTask.map
, is there a way to customize the repr of the child task besides seeing Mapped Child n
where n is a number?
For example Task.map(x=list("abc"), ...)
, is there a way to see Mapped Child a
instead?Jonathan Mathews
04/27/2022, 3:43 PMos.environ.get
? I have also tried EnvVarSecret
but that only seems to work within a flow?Rio McMahon
04/27/2022, 4:10 PMFuETL
04/27/2022, 4:35 PMBen Fisher
04/27/2022, 4:56 PMFrederick Thomas
04/27/2022, 5:39 PMFile "/mnt/data/prefect/venv/lib/python3.8/site-packages/box/box.py", line 490, in __getattr__
raise BoxKeyError(str(err)) from None
box.exceptions.BoxKeyError: "'Config' object has no attribute 'staging_db_constring'"
However if I attempt to print out the config object the code works as expected. I haven't touched the config.toml file and Googling hasn't helped. Any help would be appreciated. ThanksAlex Cannon
04/27/2022, 5:41 PMMilly gupta
04/27/2022, 5:58 PMJai P
04/27/2022, 6:29 PMwait_for
between the subflows? i'll drop a trivial example in the threadDeepak Pilligundla
04/27/2022, 6:31 PMChris Reuter
04/27/2022, 6:45 PMMatt Delacour
04/27/2022, 7:52 PMFuETL
04/27/2022, 8:14 PMMarked Failed by a Zombie Killer process
how can i increase this tolerante and why this happening? My task only take some time but this not means that is failedDavid Haynes
04/27/2022, 8:20 PMMars
04/27/2022, 8:47 PMMatt Alhonte
04/28/2022, 1:34 AMRunning
state but also has an End Time
and the Duration
isn't ticking up? Example:Jonathan Mathews
04/28/2022, 8:39 AMJames Phoenix
04/28/2022, 10:12 AMJames Phoenix
04/28/2022, 10:47 AMDavid Evans
04/28/2022, 10:50 AMprefect
CLI to push flows from GitHub Actions.
But where we're hitting problems is with dependency management (both internal code which is shared between multiple tasks/flows, and external dependencies). From what I've seen, Prefect doesn't really support this at all (flows are expected to be self-contained single files), with the implication being that the agent itself has to have any shared dependencies pre-installed (which in our case would mean that any significant changes require re-building and re-deploying the agent image - a slow process and not very practical if we have long-lived tasks or multiple people testing different flows at the same time). I tried looking around for Python bundlers and found stickytape, but that seems a bit too rough-and-ready for any real use.
This seems to be a bit of a known problem: 1, 2 and specifically I see:
V2 supports virtual and conda environment specification per flow run which should help someAnd I found some documentation for this (which seems to tie it to the new concept of deployments), but I'm still a bit confused on the details: • would the idea be to create a deployment for every version of every flow we push? Will we need to somehow tidy up the old deployments ourselves? • can deployments be given other internal files (i.e. common internal code), or is it limited to just external dependencies? Relatedly, do deployments live on the server or in the configured Storage? • is there any way to use zipapp bundles? • ideally we want engineers to be able to run flows in 3 ways: entirely locally; on a remote runner triggered from their local machine (with local code, including their latest local dependencies); and entirely remotely (pushed to the cloud server via an automated pipeline and triggered or scheduled - basically "push to production") — I'm not clear on how I should be thinking about deployments vs flows to make these 3 options a reality. I also wonder if I'm going down a complete rabbit hole and there is an easier way to do all of this?