Dilip Thiagarajan
01/07/2021, 11:04 PMflow.run
method isn’t called. Does anyone know if that’s expected?Sai Srikanth
01/07/2021, 11:07 PMVerun Rahimtoola
01/07/2021, 11:23 PMitay livni
01/08/2021, 2:04 AMFailed to load and execute Flow's environment: SyntaxError("invalid or missing encoding declaration for '/home/.../MOC/etl_moc.py'")
-After the flow began and executed some tasks successfully. The flow was run locally. Is this a bug or something not configured correctly on my end? ThanksJoël Luijmes
01/08/2021, 12:32 PMDan Corbiani
01/08/2021, 2:31 PMJeff Williams
01/08/2021, 2:39 PMMarwan Sarieddine
01/08/2021, 2:58 PMVipul
01/08/2021, 3:27 PMMatt
01/08/2021, 8:28 PMmatta
01/09/2021, 2:46 AMFailed to retrieve task state with error: ValidationError({'_schema': 'Invalid data type: None'},)
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/prefect/engine/cloud/task_runner.py", line 193, in initialize_run
map_index=map_index,
File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 1387, in get_task_run_info
state = prefect.engine.state.State.deserialize(task_run_info.serialized_state)
File "/usr/local/lib/python3.6/site-packages/prefect/engine/state.py", line 362, in deserialize
state = StateSchema().load(json_blob)
File "/usr/local/lib/python3.6/site-packages/marshmallow_oneofschema/one_of_schema.py", line 144, in load
raise exc
marshmallow.exceptions.ValidationError: {'_schema': 'Invalid data type: None'}
Using 0.13.19 right now
Looks like some kind of serialization problem?Dr. Jones
01/09/2021, 11:26 AM[2021-01-09 11:20:53,118] ERROR - My_Agent | 400 Client Error: Bad Request for url: <http://apollo:4200/>
The following error messages were provided by the GraphQL server:
GRAPHQL_VALIDATION_FAILED: Cannot query field "run_config" on type "flow_run".
The GraphQL query was:
query {
flow_run(where: { id: { _in: ["84b340e4-9f74-425b-9abd-27247fdc3759"] }, _or: [{ state: { _eq: "Scheduled" } }, { state: { _eq: "Running" }, task_runs: { state_start_time: { _lte: "2021-01-09T11:20:50.061124+00:00" } } }] }) {
id
version
state
serialized_state
parameters
scheduled_start_time
run_config
flow {
core_version
name
storage
id
environment
version
}
task_runs(where: { state_start_time: { _lte: "2021-01-09T11:20:50.061124+00:00" } }) {
serialized_state
version
task_id
id
}
}
}
The passed variables were:
null
Prash S
01/09/2021, 6:41 PMjeff n
01/10/2021, 12:02 AMJosh Pitts
01/10/2021, 2:17 AMJosh Pitts
01/10/2021, 2:17 AM@task(target="func_task_target.txt", checkpoint=True, result=LocalResult(dir="~/.prefect"))
def func_task():
return 99
Josh Pitts
01/10/2021, 2:17 AMJosh Pitts
01/10/2021, 2:18 AMJosh Pitts
01/10/2021, 2:30 AMPREFECT__FLOWS__CHECKPOINTING=true
, what is actually happening regarding target
, checkpoint=True
, and result=LocalTarget()
? or are those just ignored?Josh Pitts
01/10/2021, 3:12 AMDanny Vilela
01/10/2021, 11:35 PMPedro Machado
01/11/2021, 4:04 AMDbtShellTask
, it doesn't show output why it's running or when it succeeds. The only time I see the detailed output is when there is a failure.
I tried passing these args and it still doesn't show streamed output in the logs, not even after the task completes.
return_all=True,
log_stdout=True,
log_stderr=True,
I also set the PREFECT__LOGGING__LEVEL=DEBUG
everywhere I could think of: prefect server, agent, docker storage, run config. I see DEBUG level messages on Prefect Server, but nothing from the task.
I am running 0.14.2
Any ideas?Maurits de Ruiter
01/11/2021, 9:54 AMJarek Piotrowski
01/11/2021, 1:27 PMVincent
01/11/2021, 2:24 PMdistributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)
...
distributed.core - INFO - Event loop was unresponsive in Scheduler for 7.07s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
...
distributed.comm.tcp - INFO - Connection closed before handshake completed
...
distributed.scheduler - INFO - Close client connection: Client-worker-3c6d8642-53b5-11eb-800e-32b98c347770
When I scale the job down to 250 nodes and 3 threads per worker, I still get 100% utilization, but it is slightly more stable. where the only warning messages is
distributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)
Thanks for any advice!Equipe AI HOC
01/11/2021, 3:06 PMKevin Weiler
01/11/2021, 5:08 PMBrett Naul
01/11/2021, 5:08 PMRiley Hun
01/11/2021, 6:43 PMBilly McMonagle
01/11/2021, 7:18 PMprefect agent <AGENT TYPE> start --env KEY=VALUE --env KEY2=VALUE2
Instead, is it possible to do this by setting an environment variable on the agent, similar to how labels are set?
- name: PREFECT__CLOUD__AGENT__LABELS
value: "[]"
Billy McMonagle
01/11/2021, 7:18 PMprefect agent <AGENT TYPE> start --env KEY=VALUE --env KEY2=VALUE2
Instead, is it possible to do this by setting an environment variable on the agent, similar to how labels are set?
- name: PREFECT__CLOUD__AGENT__LABELS
value: "[]"
Kyle Moon-Wright
01/11/2021, 7:35 PMBilly McMonagle
01/11/2021, 7:39 PMAny lowercase Prefect configuration key can be set by environment variable. In order to do so, prefix the variable withand use two underscores (PREFECT__
) to separate each part of the key.__
For example, if you setBut it doesn't look like the flow environment variables are ever set like, thenPREFECT__TASKS__DEFAULTS__MAX_RETRIES=4
.prefect.config.tasks.defaults.max_retries == 4
prefect.config.agent.env
--env Key=VALUE
thing.Kyle Moon-Wright
01/11/2021, 7:49 PMPREFECT__TASKS__DEFAULTS__MAX_RETRIES=4
will apply to all your flows globally, rather than be delegated to an individual Agent, whose env vars would need to be configured independently by one of the means mentioned above.