Dnyaneshwar07/10/2020, 1:38 PM
. This gives me
error. I am not able to log more data as
option also doesn't add more information. However, when I try the same tasks on
, I do not get any error. What am I missing?
nicholas07/10/2020, 1:41 PM
sounds like something misconfigured on the Hadoop end
Dnyaneshwar07/10/2020, 1:51 PM
the flow executes as expected without any errors or warnings. (I really liked the way logs were structured, so I could actually look how each worker performed. Thanks 🙂) When I use the same YarnCluster in python (without any flow), it runs as expected. However, before even the tasks are mapped, I get the
nicholas07/10/2020, 1:52 PM
Dnyaneshwar07/10/2020, 2:21 PM
function with the argument
Jim Crist-Harif07/10/2020, 6:07 PM
outside of prefect? You might try:
from dask_yarn import YarnCluster from dask.distributed import Client cluster = YarnCluster(...) # Create a cluster, with whatever configuration you want cluster.scale(1) # Scale to one worker client = Client(cluster) client.submit(lambda x: x + 1, 1).result() # Should return 2
I am not able to log more data asTheoption also doesn't add more information.
option only applies when running with a local cluster (which happens if
). It would be helpful to get the logs from the failed cluster. You can do this with:
yarn logs -applicationId <your cluster application id>
Dnyaneshwar07/12/2020, 2:41 PM
from dask to map and reduce the tasks. It is working without any error. Whenever I use the
inside Prefect, I am getting KilledWorker error.
Jim Crist-Harif07/12/2020, 3:15 PM
Dnyaneshwar07/14/2020, 12:23 PM
. This error I am getting for both
ERROR:prefect.FlowRunner:Unexpected error: TypeError("can't pickle _mysql_connector.MySQL objects",)
Jim Crist-Harif07/14/2020, 1:11 PM
object from one task, probably as a database connection used by other tasks? One option would be to recreate the connection in every task that needs it, and close the connection after use. We're still working on good patterns for these use cases.