Dmitry Dorofeev08/31/2019, 9:20 PM
Jonah Benton09/01/2019, 4:19 PM
Feliks Krawczyk09/02/2019, 5:18 AM
Feliks Krawczyk09/02/2019, 5:36 AM
Mikhail Akimov09/03/2019, 12:18 PM
emre09/04/2019, 11:51 AM
and loads some files in and s3 bucket, under prefix=
. I wanted to add a
that deletes any leftover file from the failed task, should my task go in a
state. The problem is that the files I want to delete are identifiable by a task input. I have failed to find a way to access input parameters from the state_handler callback. I thought
would get me what I wanted, but that had only type informations. Any suggestions?
Jan Therhaag09/04/2019, 3:07 PM
Jan Therhaag09/04/2019, 3:10 PM
Aakarsh Nadella09/05/2019, 8:53 PM
Mikhail Akimov09/06/2019, 8:42 PM
>>> from example_flow import flow >>> from prefect.engine.executors import DaskExecutor >>> flow.run() # this works [2019-09-06 20:38:53,240] INFO - prefect.FlowRunner | Beginning Flow run for 'example_flow' [2019-09-06 20:38:53,241] INFO - prefect.FlowRunner | Starting flow run. [2019-09-06 20:38:53,243] INFO - prefect.TaskRunner | Task 'Task1': Starting task run... ......... >>> flow.run(executor=DaskExecutor(address="<tcp://127.0.0.1:8786>")) # this doesn't [2019-09-06 20:39:52,658] INFO - prefect.FlowRunner | Beginning Flow run for 'example_flow' [2019-09-06 20:39:52,659] INFO - prefect.FlowRunner | Starting flow run. [2019-09-06 20:39:52,758] ERROR - prefect.FlowRunner | Unexpected error: ModuleNotFoundError("No module named 'example_flow'",)
oliland09/07/2019, 1:44 PM
oliland09/07/2019, 2:30 PM
oliland09/07/2019, 2:46 PM
KJ09/07/2019, 3:51 PM
Akash09/08/2019, 7:43 AM
. Would it not make more sense for it to be of the form
prefect <VERB> <OBJECT>
? The above is based on my experience with the gcloud and aws cli tools. Usually, the user is sure of the object they want to interact with but not necessarily the verb to be used.
prefect <OBJECT> <VERB>
Feliks Krawczyk09/09/2019, 4:07 AM
With the 3a and 3b having either
Send Ping -> Try Something -> Send Ping (Success) -> Send Ping (Failed)
. What is the correct way to do this with Prefect? I see references to handlers but I’m not quite sure what the right way to do what I want is?
Feliks Krawczyk09/09/2019, 4:09 AM
But I don’t exactly know the correct way of doing the triggers in Prefect land?
with Flow('dummy prefect job') as flow: SendStatusTask('running') DoSomthethingTask() SendStatusTask('finished') SendStatusTask('failed')
Feliks Krawczyk09/09/2019, 4:11 AM
Feliks Krawczyk09/09/2019, 4:12 AM
Feliks Krawczyk09/09/2019, 4:15 AM
Feliks Krawczyk09/09/2019, 5:53 AM
Dmitriy09/09/2019, 4:46 PM
when trying to create a BQ table with
ValueError('Could not deserialize key data.')
task? I've got my credentials in a
environment variable as a JSON string and my
I'm not passing anything for
[cloud] use_local_secrets = true [context.secrets] GOOGLE_APPLICATION_CREDENTIALS = "$GOOGLE_APPLICATION_CREDENTIALS"
to the task itself (since it already defaults to
). I'm using Python 3.7.4 and
. Below is the exact output.
If anyone has any suggestions it's much appreciated.
[2019-09-09 16:42:23,995] INFO - prefect.TaskRunner | Task 'Create Table': Starting task run... [2019-09-09 16:42:23,996] INFO - prefect.TaskRunner | Unexpected error: ValueError('Could not deserialize key data.') [2019-09-09 16:42:23,996] INFO - prefect.TaskRunner | Task 'Create Table': finished task run for task with final state: 'Failed'
Ryan Abernathey09/09/2019, 6:26 PM
objects. In Pangeo, our I/O stack is something like Google Cloud Storage <- GCSFS <- Zarr -< Xarray. I would like a Prefect task to write data to GCS. The normal way I would do this (without Prefect) is:
Obviously I can do that from within a Prefect task, but it kind of seems like I should be using a
python ds = # ... create xarray Dataset gcfs_w_token = gcsfs.GCSFileSystem(project='pangeo-181919', token=token) gcsmap = gcsfs.GCSMap(path, gcs=gcfs_w_token) ds.to_zarr(gcsmap)
. Can you point me to any examples of custom handlers? (Bonus points if they show how to use secure credentials.) Thanks again for an awesome tool.
Ryan Abernathey09/09/2019, 7:42 PM