bitsofinfo
08/22/2022, 9:15 PMui.apolloApiUrl
is = <http://apollo-api.mydomain.net/graphql|apollo-api.mydomain.net/graphql>
yet its reporting localhost in the errorbitsofinfo
08/22/2022, 9:16 PMbitsofinfo
08/22/2022, 9:29 PMapi.ingress
labels?harsh patel
08/23/2022, 3:42 PMJonathan Pou
08/25/2022, 1:16 AMNumline1
08/25/2022, 7:56 PMKeith Hickey
08/27/2022, 2:12 PMPrefectFuture
from a prior task and pass it into `next_task.submit(wait_for=future_for_prior_task)`*_
Hello. Regarding the Prefect 2.0 Radar View available from the orion server ... it's neat, and provides a unique dimension of data-traceability and drill-down/fan-out ... however I feel like it has over-rotated from the Prefect 1.0 Schematic to strictly visualize explicit point-to-point data propagation between tasks, and loses the logical sequence of task execution that is equally if not more important.
Put another way: Radar highlights data flow and hides control flow.
I would argue control flow is more important, as you can typically infer how data is flowing by tracing the flow of control, but inferring dependency and temporality from the subset of tasks that explicitly exchange Python data in a tightly-coupled way will lose sight of implicit data dependencies among tasks. It would seem that getting away from DAGs (directed, acyclic graphs) is a driving principle of Prefect 2.0 ("Goodbye, DAGs"). But I still find it immensely helpful to know progress of the flow execution, i.e. where it is in the overall control graph, or where it failed.
Part of my context is that many of our DAGs / pipelines / ETLs / data processing is using external data stores (databases, files, lakehouse tables on cloud storage, etc.) with very large datasets being processed. Maybe this is more batch-processing vs. record streaming and the Radar UI is not as well suited for that? But it's not feasible to pull those large datasets into memory of a python process to be handed as python objects from task to task, or serialized from process to process. Therefore with only this implicit data-dependency, and not an explicit code-level one in the flow, no relationships is mapped between those very-much-related Tasks in the Radar UI. So questions:
1. Is there a way to code-in (via the Prefect API for tasks/flows/Futures) implicit data-dependencies from task-to-task? ... So that those tasks are laid out and connected on the Radar UI in expanding concentric circles?
a. i.e. Task do_a
pre-processes dataset A
in DB_A
... Task do_b
transports resulting data in DB_A
to external API_B
... so do_a
must happen before do_b
and yields control to it only after done.
2. Even if there's not an implicit (or explicit) data-dependency, but just a matter of sequential control flow (e.g. as governed by the SequentialTaskRunner
) can this be expressed via the Python Task/Flow API? When using the SequentialTaskRunner, tasks are laid out on the same Radar ring with no obvious order, even though they clearly are run in-sequential-ordering (not concurrent).
3. I_*s there hope for a DAG-like schematic? And if so would that be built further into Radar, or another forthcoming UI?*_
4. There seemed like some API constructs when Tasks returned PrefectFuture (or Task?) objects, which allowed some explicit functions to chain things together. But the recent shift to strip return objects down to their results makes that less possible. Is that Python API and how tasks can be tied together still under design?
I saw a few other posts where people may be struggling with the same conceptual issue, linked below.
• https://prefect-community.slack.com/archives/CL09KU1K7/p1649011482174799
• https://prefect-community.slack.com/archives/C0192RWGJQH/p1642702304024400Angel Acosta
08/30/2022, 8:36 PMPascal Holy
08/31/2022, 9:14 AMAndy Thurgood
08/31/2022, 4:52 PMupm
, data
, sdata
and reserved.
Is there a way to debug this at all? Should we be landing at a signup or set credentials type page?
ThanksPablo Beltran
08/31/2022, 8:24 PMYoussef Ben Farhat
09/01/2022, 3:15 PMYusuf
09/01/2022, 8:48 PMRajvir Jhawar
09/02/2022, 1:34 AMYusuf
09/02/2022, 2:25 AMAdam Brusselback
09/02/2022, 2:50 PMLucas Cavalcanti Rodrigues
09/02/2022, 7:21 PMgql(
'''
query LatestFlowByName($name: String) {
flow(
where: {name: {_eq: $name}},
order_by: {version: desc},
limit: 1,
)
{
id
}
}
''',
)
But how can I have all flow ids regardless of the name?Adam Brusselback
09/02/2022, 8:23 PMTan Li
09/04/2022, 4:41 PMAndreas Nord
09/07/2022, 1:50 PMMarc-Antoine Bélanger
09/08/2022, 12:41 AM{"data": {
"title": "data",
"default": {},
"type": "object",
},
"train": {
"title": "train",
"default": {},
"type": "object",
}
}
This is what it is showing in the UI:
But the run fails:
Validation of flow parameters failed with error: ParameterTypeError('Flow run received invalid parameters:\n - data: value is not a valid dict\n - train: value is not a valid dict')
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 284, in retrieve_flow_then_begin_flow_run
parameters = flow.validate_parameters(flow_run.parameters)
File "/usr/local/lib/python3.10/site-packages/prefect/flows.py", line 269, in validate_parameters
raise ParameterTypeError.from_validation_error(exc) from None
prefect.exceptions.ParameterTypeError: Flow run received invalid parameters:
- data: value is not a valid dict
- train: value is not a valid dict
They seem to be interpreted as strings in the Parameters tab:
{
"data": "{}",
"train": "{}"
}
Any idea what I am doing wrong?Marc-Antoine Bélanger
09/08/2022, 12:43 AMAmelie Froessl
09/08/2022, 9:58 AMclass Parameters_1(BaseModel, extra=Extra.forbid):
a: int
b: int
c: str
class Parameters_2(BaseModel, extra=Extra.forbid):
d: int
e: int
f: str
class Parameters_3(BaseModel, extra=Extra.forbid):
g: int
h: int
i: str
class TestConfig(BaseModel, extra=Extra.forbid):
parameters_1: Parameters_1
parameters_2: Parameters_2
parameters_3: Parameters_3
On the UI it shows up like in the attached screenshot. The input would be a dictionary of the values in the respecitve BaseModel like : {"a":1, "b":2, "c": "hello"}
. ... which is not the nicest I guess. I was wondering if there is a way to nicely have the parameters of a sub BaseModel show up/ editable in the UI? Or if somebody has a suggestion on how this could be solved in a nicer way?
Thanks in advance!davzucky
09/10/2022, 12:42 AMLocal File System
has for slug
local-file-system
Tomás Emilio Silva Ebensperger
09/12/2022, 8:03 PMSteph Clacksman
09/13/2022, 10:41 AMDavid George
09/14/2022, 2:14 PMVaikath Job
09/17/2022, 2:55 AMvk
09/19/2022, 8:08 PMfrom prefect import flow, task
from prefect import get_run_logger
import time
@task
def run_computation():
logger = get_run_logger()
<http://logger.info|logger.info>("run some computation")
time.sleep(30)
return 123
@task
def dependent_task(x):
logger = get_run_logger()
<http://logger.info|logger.info>("run dependent task")
time.sleep(10)
@task
def independent_task(i):
logger = get_run_logger()
<http://logger.info|logger.info>(f"run independent task: {i}")
time.sleep(3)
@flow(name="orion2")
def f():
x = run_computation()
independent_task(0)
for i in range(1, 10):
independent_task(i)
dependent_task(x)
if __name__ == '__main__':
f()
Alix Cook
09/21/2022, 7:09 PMAlix Cook
09/21/2022, 7:09 PMJeff Hale
09/21/2022, 8:02 PM