tash lai
04/14/2021, 10:06 AMKevin Kho
tash lai
04/14/2021, 4:23 PM@task
def download_and_process_big_json(url, mongo_url):
client = MongoClient(mongo_url)
db = client.get_default_database()
data = db['big_json'].find_one({'url': url})
if data:
<set task state to Cached>
else:
data = requests.get(url).json()
db['big_json'].insert_one({'url': url, **data})
client.close()
process_big_json(data)
So i want to simulate output caching using mongo but without actually returning anything from the task. Of course i can live without changing the task state to Cached and it will work fine but i kinda like to see if the data did already exist in the database via the prefect UI, you know
Cached: 198 Success: 300
etcKevin Kho
Kevin Kho
tash lai
04/14/2021, 4:37 PMKevin Kho
Kevin Kho
raise SUCCESS
with a custom message.Zanie
Zanie
Success
state subclasses"Marvin
04/14/2021, 5:54 PM