Darragh05/01/2020, 10:20 AM
Zviri05/01/2020, 12:09 PM
) the memory consumption of my Dask Workers gradually grows until they run out of memory. When I run the same flow from a script everything finishes correctly and there are no memory issues what so ever. I am using the latest
packages. Has anyone experienced something similar? Is there anything that could be done about it? Thanks a lot.
Jacques05/01/2020, 1:46 PM
Jacob (he/him)05/01/2020, 4:00 PM
John Ramirez05/01/2020, 6:00 PM
Troy Köhler05/01/2020, 6:01 PM
John Ramirez05/01/2020, 6:19 PM
tiz.io05/01/2020, 6:49 PM
Crawford Collins05/01/2020, 7:18 PM
I'm trying to test a @task that returns a dataframe, but assert statement errors on
with Flow("test_target_transformer") as f: te = fit_target_transformer(problem, target, train_data) df = target_encoder_transform(te, imputed_categorical_df) assert df.isna().sum() == 0
. How do I return this as a DataFrame and not a task?
AttributeError: 'FunctionTask' object has no attribute 'isna'
Matt05/02/2020, 7:22 AM
David Haines05/02/2020, 12:25 PM
has been integrated within the imperative api yet?
Florian05/03/2020, 12:31 PM
leads to a blocking process until the flow has finished. Is there a way of submitting jobs to the dask cluster in an almost fire and forget scenario? Especially since my outputs are written to file or DB I really don’t need the return/result object. I basically want to dynamically create workflows submit them to the dask cluster and then have them finish based on available resources. Could be that I am overlooking something super simple or using prefect for something it is not build for 🤷♂️ but would be nice to know 🙂
Pierre CORBEL05/03/2020, 12:44 PM
in the UI but not in the logs tab (see attached image n°2 and n°3) For info, I'm using RemoteEnvironment with LocalDaskExecutor on Prefect 0.10.5
Last State Message
Nate Atkins05/04/2020, 12:37 AM
Chris O'Brien05/04/2020, 3:56 AM
that is yesterday UTC time. I would love to use the prefect context to do this and this worked for me, but is there a better way?
Thomas La Piana05/04/2020, 6:41 AM
Adisun Wheelock05/04/2020, 1:16 PM
prefect auth login -t my_generated_token
has anyone else ran into this? The server start up brought up all services just fine.
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: <http://localhost:4200/graphql/alpha>
Avi A05/04/2020, 2:45 PM
Ketan Bhikadiya05/04/2020, 4:38 PM
Pedro Machado05/04/2020, 6:39 PM
. I am trying to figure out how I could implement this in Prefect. It seems to me that I could have a single parametrized flow but I am not sure how the parameters would be provided and the flow scheduled. Thanks!
tiz.io05/05/2020, 12:38 AM
Brad05/05/2020, 2:14 AM
Pedro Machado05/05/2020, 3:53 AM
Sainath05/05/2020, 11:31 AM
Thomas La Piana05/05/2020, 11:35 AM
locally, it automatically defaults to the local executor instead of using the environment that i set for it, but when i register it and a kube runner picks it up it runs it with the correct environment. does anyone know why?
Avi A05/05/2020, 7:45 PM
itay livni05/05/2020, 10:06 PM
error using Docker Storage (local image), prefect cloud and the
NoCredentialsError('Unable to locate credentials')
When the flow is run locally. There are no credential issues initializing the handler
To resolve the issue I created a Prefect Secret called
s3_handler = S3ResultHandler(bucket='some-bucket')
What is the best way to resolve aws credential error? Thanks
s = Secret("AWS_CREDENTIALS") aws_scrts = s.get() s3_handler = S3ResultHandler(bucket='some-bucket', aws_credentials_secret=aws_scrts)
David Ojeda05/06/2020, 9:17 AM
I could come up with a plumbing hack like this:
ValueError: A task with the slug "limit" already exists in this flow.
which works, but it seems very hackish and far from elegant. Is there a cleaner alternative to this? (other than renaming the parameter, of course)
flow = Flow(name) local_flow = build_local_flow() # a function that returns a flow quetzal_flow = build_quetzal_flow() # idem # plumbing: both local and quetzal flows have a limit parameter limit_parameter = local_flow.get_tasks(name='limit') other_parameter = quetzal_flow.get_tasks(name='limit') for edge in quetzal_flow.edges: if edge.upstream_task == other_parameter: edge.upstream_task = limit_parameter quetzal_flow.tasks = set(t for t in quetzal_flow.tasks if t != other_parameter) flow.update(local_flow) flow.update(quetzal_flow) ...