Julien Allard
08/20/2020, 1:51 PMUnexpected error: TypeError("Cannot map over unsubscriptable object of type <class 'NoneType'>: None...")
. The problems seems to come from a mapped task that outputs a pandas dataframe.
Anyone has any ideas on how to fix this problem or how to debug it further?Richard Hughes
08/20/2020, 2:02 PMRichard Hughes
08/20/2020, 2:14 PMNo heartbeat detected from the remote task; marking the run as failed.
Marwan Sarieddine
08/20/2020, 2:23 PMPREFECT_FLOWS_CHECKPOINTING
and I use a pd.DataFrame
as a parameter - I am using prefect v0.13.3
export PREFECT_FLOWS_CHECKPOINTING=true
In [1]: import pandas as pd
In [2]: from prefect import Parameter, task, Flow
In [3]: @task
...: def simple(df):
...: return df
...:
In [4]: with Flow("test") as flow:
...: df = Parameter(name="df")
...: simple(df)
...: flow.run(df=pd.DataFrame())
[2020-08-20 14:20:38] ERROR - prefect.TaskRunner | Unexpected error: TypeError('Object of type DataFrame is not JSON serializable')
Traceback (most recent call last):
File "~/.pyenv/versions/3.7.7/envs/infima/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "~/.pyenv/versions/3.7.7/envs/infima/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 838, in get_task_run_state
result = self.result.write(value, **formatting_kwargs,)
File "~/.pyenv/versions/3.7.7/envs/infima/lib/python3.7/site-packages/prefect/engine/results/prefect_result.py", line 60, in write
new.location = self.serializer.serialize(new.value).decode("utf-8")
File "~/.pyenv/versions/3.7.7/envs/infima/lib/python3.7/site-packages/prefect/engine/serializers.py", line 100, in serialize
return json.dumps(value).encode()
File "~/.pyenv/versions/3.7.7/lib/python3.7/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "~/.pyenv/versions/3.7.7/lib/python3.7/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "~/.pyenv/versions/3.7.7/lib/python3.7/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "~/.pyenv/versions/3.7.7/lib/python3.7/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type DataFrame is not JSON serializable
Out[4]: <Failed: "Some reference tasks failed.">
when I disable PREFECT_FLOWS_CHECKPOINTING
flow runs successfully
export PREFECT_FLOWS_CHECKPOINTING=false
In [1]: import pandas as pd
In [2]: from prefect import Parameter, task, Flow
In [3]: @task
...: def simple(df):
...: return df
...:
In [4]: with Flow("test") as flow:
...: df = Parameter(name="df")
...: simple(df)
...: flow.run(df=pd.DataFrame())
Out[4]: <Success: "All reference tasks succeeded.">
William Smith
08/20/2020, 2:29 PMWilliam Smith
08/20/2020, 3:10 PMBrian Mesick
08/20/2020, 3:20 PMPrefectResult
work for a task, presuming I’m using Docker storage for the flow?
• Does adding a PrefectResult
to a task somehow transparently change the labels of a Flow?Adam
08/20/2020, 5:24 PMRuben Silva
08/20/2020, 9:38 PMJimmy Le
08/20/2020, 10:18 PMEric
08/21/2020, 3:01 AMHoward Cornwell
08/21/2020, 11:00 AMgraphql_1 |
graphql_1 | Running Alembic migrations...
graphql_1 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
graphql_1 | INFO [alembic.runtime.migration] Will assume transactional DDL.
graphql_1 |
graphql_1 | Could not upgrade the database!
graphql_1 | Error: Can't locate revision identified by '7e65dadba625'
Michael Ludwig
08/21/2020, 11:58 AMFailed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID 0c1f4327-19fe-4522-8f8c-ad4c2fb7dc45: provided a running state but associated flow run c4d445b2-8d4e-4863-9ba5-b89b5addeba3 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
File "/opt/app/.venv/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 112, in call_runner_target_handlers
state = self.client.set_task_run_state(
File "/opt/app/.venv/lib/python3.8/site-packages/prefect/client/client.py", line 1295, in set_task_run_state
result = self.graphql(
File "/opt/app/.venv/lib/python3.8/site-packages/prefect/client/client.py", line 287, in graphql
raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID 0c1f4327-19fe-4522-8f8c-ad4c2fb7dc45: provided a running state but associated flow run c4d445b2-8d4e-4863-9ba5-b89b5addeba3 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
Miecio
08/21/2020, 1:13 PMMichael Ludwig
08/21/2020, 1:57 PMResults
like S3Result
do I need to use them for all tasks to be able to restart failed flows? Not totally clear to meMichael Reeves
08/21/2020, 3:43 PMJared
08/21/2020, 5:39 PMlate_work_seconds
setting shown in the [prefect server config](https://github.com/PrefectHQ/server/blob/master/src/prefect_server/config.toml) actually do anything? As far as I can tell there's nothing referencing config.services.sla
in the open-source server code, but this would be really useful to have access tobral
08/21/2020, 9:52 PMMarwan Sarieddine
08/21/2020, 10:38 PM[5:57 PM] $ kubectl logs pod/prefect-job-e74114b4-26n6v
[2020-08-21 21:55:08] INFO - prefect.S3 | Downloading simple-flow-8/2020-08-21t21-54-33-484226-00-00 from infima-etl-flows
Traceback (most recent call last):
File "/usr/local/bin/prefect", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", line 80, in cloud_flow
raise exc
File "/usr/local/lib/python3.8/site-packages/prefect/cli/execute.py", line 69, in cloud_flow
flow = storage.get_flow(storage.flows[flow_data.name])
File "/usr/local/lib/python3.8/site-packages/prefect/environments/storage/s3.py", line 101, in get_flow
return cloudpickle.loads(output)
TypeError: an integer is required (got type bytes)
I am wondering if anyone else faced something similarMaxwell Dylla
08/21/2020, 11:38 PMKyle McEntush
08/22/2020, 12:07 AMKyle McEntush
08/22/2020, 1:41 AMKyle McEntush
08/22/2020, 4:38 PMValueError: Could not infer an active Flow context.
Jack Sundberg
08/22/2020, 6:27 PMbral
08/22/2020, 8:57 PMx062Wyhdolq
08/23/2020, 10:38 AMAvi A
08/23/2020, 11:13 AMLocalEnvironment
/ DaskCluster
to run my flow on a local cluster. Where does prefect report the LocalCluster provisioning logs? I’m interested in what port the UI is served (port 8787 is already taken on the server)Kyle McEntush
08/24/2020, 1:14 AMJacob Blanco
08/24/2020, 3:11 AMMikael
08/24/2020, 7:38 AMexecutor=DaskExecutor()
i get RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
. Anyone having the same issue?Mikael
08/24/2020, 7:38 AMexecutor=DaskExecutor()
i get RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
. Anyone having the same issue?Kyle Moon-Wright
08/24/2020, 3:25 PMMikael
08/24/2020, 9:27 PM