Bartek
05/13/2020, 1:11 PMUI
as it not showing any flows
and runs
. I register flow with success and see in agent logs and server logs that flow runs when is scheduled but I have no information about flows and runs in UI
.Troy Sankey
05/13/2020, 2:27 PMKostas Chalikias
05/13/2020, 2:38 PMMatthias
05/13/2020, 3:02 PMLarge object of size 5.49 MB detected in task graph:
Am I doing something wrong? This is the simplest example I could come up with, that shows this behaviour.Scott Zelenka
05/13/2020, 3:03 PMversion_group_id
. The execution of the Flow includes a step to write data back to a Prod API of an external system.
The challenge comes when making iterations to this Flow in development. The external system's Dev instance still triggers our Flow through GraphQL, but is expecting it to write data back to a Dev API. We could update and deploy the Flow to Cloud, but then the version_group_id
picks up our Dev Flow, rather than our Prod Flow.
The only thing I can think of is to have two different Flows deployed on Cloud (one for Prod and another for Dev). But in that case, promoting from Dev to Prod has a bunch of manual steps prone to human error.
Interested in the communities thoughts on how you handle deploying multiple versions of the same Flow between Dev and Prod environments, where the configuration between Dev and Prod are different.Darragh
05/13/2020, 3:06 PMJohn Ramirez
05/13/2020, 3:12 PM.map()
or submit a single job and manage running the different models within the single spark job.Emmanuel Klinger
05/13/2020, 3:41 PMKaz
05/13/2020, 4:13 PMAndy Waugh
05/13/2020, 6:59 PMJeremiah
matta
05/13/2020, 11:20 PMgspread
package, which has you do everything from method calls to an authentication object. So like you go gc = gspread.service_account(filename=<filename>)
and point it at a special credentials file, then everything is through that. Should I just pass it to Secrets? Is there any risk of sensitive credentials being cached somewhere if I define the gc
object within the Flow itself?Darragh
05/14/2020, 10:20 AMSimon Basin
05/14/2020, 2:46 PMChristopher Harris
05/14/2020, 4:26 PMbatch_size
increments, and map the individual data packets across the remaining DAG.
In a way this kind of seems like a āworkflow loopā with the parameters for the first node constantly updating.Julia Eskew
05/14/2020, 5:46 PMtkanas
05/14/2020, 7:26 PMLocalResult
but I'm wondering if there are Prefect features that are particularly appropriate for this use case.Matthias
05/14/2020, 8:05 PMDan DiPasquo
05/14/2020, 9:23 PMMatthew Maldonado
05/14/2020, 10:02 PMMatthew Maldonado
05/14/2020, 11:27 PMJoe Schmid
05/15/2020, 3:05 AM@task
def times_two(x):
return x * 2
@task
def add(items):
return sum(items)
class SimpleFlow(SRMPrefectFlow):
@property
def flow(self) -> Flow:
with Flow("SimpleFlow", environment=env) as flow:
x = Parameter("x", default=[1, 2, 3])
times_two_task_result = times_two.map(x)
flow_result = add(times_two_task_result)
return flow
And a simple test to run the flow & check the last task's result:
def test_flow_run_result():
flow = SimpleFlow().flow
fr = flow.run()
assert list(fr.result.values())[2].result == 12
The list(fr.result.values())[2].result
works, but is fragile. We'd rather fr.result[flow_result].result
but flow_result
isn't available outside of the function that defines the flow. Is there a better approach that people have used?Barry Roszak
05/15/2020, 8:07 AMtags
? If I us map in one task and I want to limit operation besed on input to the task?Cab Maddux
05/15/2020, 2:03 PMPierre CORBEL
05/15/2020, 3:54 PMprefect 0.11.0
,
you can't use the enviuronment variable PREFECT__CLOUD__AGENT__LABELS=["hello"]
anymore but you have to wrap the values in single quote
like PREFECT__CLOUD__AGENT__LABELS='["hello"]'
I know it is documented with the right format in the doc but it was working with a "bad" format before and so it can broke your flow after upgrading to v0.11.0 šWill Milner
05/15/2020, 4:47 PMapollo_1 | 2020-05-15T16:42:41.530Z {"message":"Cannot query field \"setFlowRunStates\" on type \"Mutation\". Did you mean \"set_flow_run_states\" or \"set_task_run_states\"?","locations":[{"line":2,"column":5}],"extensions":{"code":"GRAPHQL_VALIDATION_FAILED"}}
apollo_1 | 2020-05-15T16:42:41.547Z {"message":"Unknown type \"writeRunLogsInput\". Did you mean \"write_run_logs_input\", \"write_run_log_input\", \"create_flow_input\", or \"archive_flow_input\"?","locations":[{"line":1,"column":18}],"extensions":{"code":"GRAPHQL_VALIDATION_FAILED"}}
Not sure how I go about debugging this, any tips?Laura Lorenz (she/her)
05/15/2020, 7:42 PMBrad
05/16/2020, 12:41 AMPedro Machado
05/16/2020, 2:34 AMNicolas Michel
05/16/2020, 6:26 PM