itay livni
04/01/2020, 2:49 AMFlow
with a schedule to run once a day at a specific time on the cloud (AWS). The size of the data is sub 1 mb. Should the ETL flow have a scheduler? -- Thanks in advance.Kyle Foreman (Convoy)
04/01/2020, 7:33 AMPierre CORBEL
04/01/2020, 11:28 AMServer Start UI Flow Run
one)
In fact, there is only one public video in the channel (Prefect Cloud Demo)
Would you mind sharing the others one? I'm pretty interested in your videos / tutorials and I think it could greatly benefit the community 😊John Ramirez
04/01/2020, 12:04 PMUnexpected error: ClientError([{'message': "('Connection aborted.', OSError(0, 'Error'))", 'locations': [{'line': 2, 'column': 9}], 'path': ['secretValue'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'errors': [{'message': "('Connection aborted.', OSError(0, 'Error'))", 'locations': [], 'path': ['secretValue']}]}}}])
Is this a error on the prefect side?Daniel Sali
04/01/2020, 12:09 PMArmin
04/01/2020, 1:30 PMHenry Cohen
04/01/2020, 2:56 PMScott Zelenka
04/01/2020, 4:23 PMPreston Marshall
04/01/2020, 4:39 PMDavid N
04/01/2020, 5:56 PMBrian Bergeron
04/01/2020, 6:14 PMJohn Ramirez
04/01/2020, 6:58 PMFailed to set task state with error: KeyError('endpoint_resolver')
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/prefect/engine/cloud/task_runner.py", line 113, in call_runner_target_handlers
cloud_state = prepare_state_for_cloud(new_state)
File "/opt/conda/lib/python3.7/site-packages/prefect/engine/cloud/utilities.py", line 21, in prepare_state_for_cloud
res.store_safe_value()
File "/opt/conda/lib/python3.7/site-packages/prefect/engine/result.py", line 92, in store_safe_value
value = self.result_handler.write(self.value)
File "/opt/conda/lib/python3.7/site-packages/prefect/engine/result_handlers/s3_result_handler.py", line 103, in write
self.client.upload_fileobj(stream, Bucket=self.bucket, Key=uri)
File "/opt/conda/lib/python3.7/site-packages/prefect/engine/result_handlers/s3_result_handler.py", line 67, in client
self.initialize_client()
File "/opt/conda/lib/python3.7/site-packages/prefect/engine/result_handlers/s3_result_handler.py", line 60, in initialize_client
aws_secret_access_key=aws_secret_access_key,
File "/opt/conda/lib/python3.7/site-packages/boto3/__init__.py", line 91, in client
return _get_default_session().client(*args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/boto3/session.py", line 263, in client
aws_session_token=aws_session_token, config=config)
File "/opt/conda/lib/python3.7/site-packages/botocore/session.py", line 824, in create_client
endpoint_resolver = self._get_internal_component('endpoint_resolver')
File "/opt/conda/lib/python3.7/site-packages/botocore/session.py", line 697, in _get_internal_component
return self._internal_components.get_component(name)
File "/opt/conda/lib/python3.7/site-packages/botocore/session.py", line 923, in get_component
del self._deferred[name]
KeyError: 'endpoint_resolver'
Dylan
David Haines
04/01/2020, 7:23 PMScott Zelenka
04/01/2020, 7:28 PMLocalDaskExecutor
locally via flow.run(executor=LocalDaskExecutor())
, to get it to spawn multiple threads to execute in parallel, but when I register the same flow to Cloud, it seems to execute the mapped tasks in sequence.George Coyne
04/01/2020, 7:51 PMGeorge Coyne
04/01/2020, 7:51 PMTraceback (most recent call last):
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/prefect/client/client.py", line 591, in register
prefect.serialization.flow.FlowSchema().load(serialized_flow)
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/prefect/utilities/serialization.py", line 141, in load
return super().load(data, **kwargs)
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow/schema.py", line 723, in load
data, many=many, partial=partial, unknown=unknown, postprocess=True
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow/schema.py", line 861, in _do_load
unknown=unknown,
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow/schema.py", line 669, in _deserialize
index=index,
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow/schema.py", line 493, in _call_and_store
value = getter_func(data)
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow/schema.py", line 662, in <lambda>
val, field_name, data, **d_kwargs
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow/fields.py", line 342, in deserialize
output = self._deserialize(value, attr, data, **kwargs)
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow/fields.py", line 597, in _deserialize
return self._load(value, data, partial=partial)
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow/fields.py", line 580, in _load
valid_data = self.schema.load(value, unknown=self.unknown, partial=partial)
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow_oneofschema/one_of_schema.py", line 122, in load
data, partial=partial, unknown=unknown
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow_oneofschema/one_of_schema.py", line 176, in _load
return schema.load(data, many=False, partial=partial, unknown=unknown)
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/prefect/utilities/serialization.py", line 141, in load
return super().load(data, **kwargs)
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow/schema.py", line 723, in load
data, many=many, partial=partial, unknown=unknown, postprocess=True
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow/schema.py", line 861, in _do_load
unknown=unknown,
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow/schema.py", line 669, in _deserialize
index=index,
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow/schema.py", line 493, in _call_and_store
value = getter_func(data)
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow/schema.py", line 662, in <lambda>
val, field_name, data, **d_kwargs
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow/fields.py", line 342, in deserialize
output = self._deserialize(value, attr, data, **kwargs)
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow/fields.py", line 597, in _deserialize
return self._load(value, data, partial=partial)
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow/fields.py", line 580, in _load
valid_data = self.schema.load(value, unknown=self.unknown, partial=partial)
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow_oneofschema/one_of_schema.py", line 131, in load
result = self._load(item, partial=partial)
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/marshmallow_oneofschema/one_of_schema.py", line 148, in _load
raise ValidationError({"_schema": "Invalid data type: %s" % data})
TypeError: not all arguments converted during string formatting
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "knack_facts.py", line 708, in <module>
flow.register("PROJECT_FLOW")
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/prefect/core/flow.py", line 1372, in register
no_url=no_url,
File "/Users/gcoyne/miniconda3/envs/FLOW_NAME/lib/python3.7/site-packages/prefect/client/client.py", line 595, in register
repr(exc)
ValueError: Flow could not be deserialized successfully. Error was: TypeError('not all arguments converted during string formatting')
Louis Guitton
04/01/2020, 8:25 PMgit clone --depth 1 <https://github.com/PrefectHQ/prefect.git>
cd prefect/examples
sed -i '' 's/flow.run/flow.register/g' etl.py
sed -i '' 's/flow.run/flow.register/g' spacy_nlp.py
prefect server start
python etl.py
python spacy_nlp.py
Manuel Aristarán
04/01/2020, 8:50 PMBrett Naul
04/02/2020, 12:03 AMcloud
from cli.run
to start runs since it handles a lot of the boilerplate of checking status / fetching logs for us. but it doesn't really seem like it was designed for programmatic use: in particular there's no meaningful return value or exit code, so it can't be used to see if a flow run succeeded or failed or anything else about it. would changing that function to return more info (say flow run ID and state) make sense?Jeremy Yeo
04/02/2020, 7:03 AMpip install prefect
.
2. prefect backend server
.
3. In a new terminal: prefect server start
- this allows UI to be accessible locally (no flows yet).
4. In another new terminal, start an agent to run flows: prefect agent start
.
5. Create a simple new flow in `demo-flow.py`:
import prefect
from prefect import task, Flow
@task
def hello_task():
logger = prefect.context.get("logger")
<http://logger.info|logger.info>("Hello, Cloud!")
@task
def another_task():
logger = prefect.context.get("logger")
<http://logger.info|logger.info>("Our second flow!")
flow = Flow("hello-flow", tasks=[hello_task])
flow.register()
flow_2 = Flow("second-flow", tasks=[another_task])
flow_2.register()
6. Register the new flows: python demo-flow.py
.
7. Navigate to the UI (localhost:8080
) and start a manual run of the "hello-flow" flow.
8. Output is visible in the log in the UI.
---
Is this how one should "register" a flow (step 6) and why would a run of such a simple flow take 3 minutes to complete? Thanks :)Milos Tomic
04/02/2020, 8:24 AMScott Zelenka
04/02/2020, 1:48 PMinquisitive-wildcat
Flow Run. And the downstream Tasks all indicate they had a failure.
However, when I navigate to that Flow Run and look at the mapped Task, I cannot locate the exact mapped instance where these failures happened. It seems the only way to jump to that specific error message is to navigate from the main Flow page?
1. Shouldn't the failed mapped Task show up as "FAILED" in the UI?
2. Is the Task considered "FAILED" if there was a problem in the Result Handler?
On the immediate next Task that consumes the output of the mapped Task, it seems Prefect sent a None
object, which then caused an exception and finally failed the Flow Run.
3. Why is Prefect sending a None
to a downstream Task of a mapped Task output that had a failure?Manuel Mourato
04/02/2020, 2:36 PMManuel Mourato
04/02/2020, 2:43 PMJohn Ramirez
04/02/2020, 2:43 PM0.10.0
Unexpected error: AttributeError("'S3ResultHandler' object has no attribute '_client'")
I thought this was fixed in the 0.10.0
releaseKostas Chalikias
04/02/2020, 3:34 PMChris Hart
04/02/2020, 7:02 PMwith Flow("trainer", schedule=None) as flow:
rand = Parameter("rand", default=params["rand"])
limit = Parameter("limit", default=params["limit"])
class_type = Parameter("class_type", default=params["class_type"])
# tasks
training_data = fetch_training_data(rand, limit, class_type)
test_data = fetch_test_data(rand, limit, class_type)
trained_model = train_model(training_data)
test_model(trained_model, test_data)
flow_state = flow.run(executor=DaskExecutor(), parameters=params)
Chris Hart
04/02/2020, 8:08 PMMark McDonald
04/02/2020, 8:11 PM