Dmitry Dorofeev
08/31/2019, 9:20 PMJonah Benton
09/01/2019, 4:19 PMFeliks Krawczyk
09/02/2019, 5:18 AMFeliks Krawczyk
09/02/2019, 5:36 AMMikhail Akimov
09/03/2019, 12:18 PMemre
09/04/2019, 11:51 AMtable_names
and loads some files in and s3 bucket, under prefix= table_name
. I wanted to add a state_handler
that deletes any leftover file from the failed task, should my task go in a Failed
state.
The problem is that the files I want to delete are identifiable by a task input. I have failed to find a way to access input parameters from the state_handler callback. I thought task.inputs()
would get me what I wanted, but that had only type informations. Any suggestions?Jan Therhaag
09/04/2019, 3:07 PMJan Therhaag
09/04/2019, 3:10 PMAakarsh Nadella
09/05/2019, 8:53 PMMikhail Akimov
09/06/2019, 8:42 PM>>> from example_flow import flow
>>> from prefect.engine.executors import DaskExecutor
>>> flow.run() # this works
[2019-09-06 20:38:53,240] INFO - prefect.FlowRunner | Beginning Flow run for 'example_flow'
[2019-09-06 20:38:53,241] INFO - prefect.FlowRunner | Starting flow run.
[2019-09-06 20:38:53,243] INFO - prefect.TaskRunner | Task 'Task1': Starting task run...
.........
>>> flow.run(executor=DaskExecutor(address="<tcp://127.0.0.1:8786>")) # this doesn't
[2019-09-06 20:39:52,658] INFO - prefect.FlowRunner | Beginning Flow run for 'example_flow'
[2019-09-06 20:39:52,659] INFO - prefect.FlowRunner | Starting flow run.
[2019-09-06 20:39:52,758] ERROR - prefect.FlowRunner | Unexpected error: ModuleNotFoundError("No module named 'example_flow'",)
Mikhail Akimov
09/06/2019, 8:45 PMoliland
09/07/2019, 1:44 PMoliland
09/07/2019, 2:30 PMoliland
09/07/2019, 2:46 PMKJ
09/07/2019, 3:51 PMAkash
09/08/2019, 7:43 AMprefect <VERB> <OBJECT>
. Would it not make more sense for it to be of the form prefect <OBJECT> <VERB>
?
The above is based on my experience with the gcloud and aws cli tools. Usually, the user is sure of the object they want to interact with but not necessarily the verb to be used.Feliks Krawczyk
09/09/2019, 4:07 AMSend Ping -> Try Something -> Send Ping (Success)
-> Send Ping (Failed)
With the 3a and 3b having either trigger_rule="all_success"
or trigger_rule="one_failed"
. What is the correct way to do this with Prefect? I see references to handlers but I’m not quite sure what the right way to do what I want is?Feliks Krawczyk
09/09/2019, 4:09 AMwith Flow('dummy prefect job') as flow:
SendStatusTask('running')
DoSomthethingTask()
SendStatusTask('finished')
SendStatusTask('failed')
But I don’t exactly know the correct way of doing the triggers in Prefect land?Feliks Krawczyk
09/09/2019, 4:11 AMFeliks Krawczyk
09/09/2019, 4:12 AMChris White
09/09/2019, 4:12 AMFeliks Krawczyk
09/09/2019, 4:13 AMChris White
09/09/2019, 4:14 AMFeliks Krawczyk
09/09/2019, 4:15 AMChris White
09/09/2019, 4:15 AMFeliks Krawczyk
09/09/2019, 5:53 AMDmitriy
09/09/2019, 4:46 PMValueError('Could not deserialize key data.')
when trying to create a BQ table with CreateBigQueryTable
task? I've got my credentials in a GOOGLE_APPLICATION_CREDENTIALS
environment variable as a JSON string and my ~/.prefect/config.toml
as:
[cloud]
use_local_secrets = true
[context.secrets]
GOOGLE_APPLICATION_CREDENTIALS = "$GOOGLE_APPLICATION_CREDENTIALS"
I'm not passing anything for credentials
to the task itself (since it already defaults to GOOGLE_APPLICATION_CREDENTIALS
). I'm using Python 3.7.4 and prefect==0.6.1
. Below is the exact output.
[2019-09-09 16:42:23,995] INFO - prefect.TaskRunner | Task 'Create Table': Starting task run...
[2019-09-09 16:42:23,996] INFO - prefect.TaskRunner | Unexpected error: ValueError('Could not deserialize key data.')
[2019-09-09 16:42:23,996] INFO - prefect.TaskRunner | Task 'Create Table': finished task run for task with final state: 'Failed'
If anyone has any suggestions it's much appreciated.Ryan Abernathey
09/09/2019, 6:26 PMResultHandler
objects. In Pangeo, our I/O stack is something like Google Cloud Storage <- GCSFS <- Zarr -< Xarray.
I would like a Prefect task to write data to GCS. The normal way I would do this (without Prefect) is:
python
ds = # ... create xarray Dataset
gcfs_w_token = gcsfs.GCSFileSystem(project='pangeo-181919', token=token)
gcsmap = gcsfs.GCSMap(path, gcs=gcfs_w_token)
ds.to_zarr(gcsmap)
Obviously I can do that from within a Prefect task, but it kind of seems like I should be using a ResultHandler
. Can you point me to any examples of custom handlers? (Bonus points if they show how to use secure credentials.)
Thanks again for an awesome tool.Ryan Abernathey
09/09/2019, 7:42 PMAkash
09/10/2019, 12:46 PMAkash
09/10/2019, 12:46 PMChris White
09/10/2019, 3:40 PM