https://prefect.io logo
#prefect-community
Title
# prefect-community
s

Samuel Hinton

03/13/2023, 1:17 AM
Hi all! Does anyone know the best practise for catching custom exceptions within prefect tasks? I have an exception which isnt a code failure, its just data not being available, and I’d like to send a slack warning when this happens but not report flow failure. I hoped that this would work:
Copy code
try:
        data = task.submit(start, end, logger=logger)
    except DataWarning as e:
        logger.warning(e)
        send_incoming_webhook_message(
            slack_webhook,
            f"Warning raised for {task_name} for {start} to {end} due to {e}",
        )
But no dice, doesnt play well with the future object. Im reworking all of this to not use exceptions and just pass dictionaries around, but thought knowing the best practise might be useful
t

Tim Galvin

03/13/2023, 1:36 AM
Try it without the
.submit
. Just call
task
so it runs synchronously. This will cause the remainder to the flow to block until that
task
function has been evaluated.
s

Samuel Hinton

03/13/2023, 1:36 AM
Wont that stop execution via dask workers though and cause it to run in the agent itself?
t

Tim Galvin

03/13/2023, 1:45 AM
Oh good question. The answer is I don't know
Maybe look into the
.result()
attribute of the prefect futures object
s

Samuel Hinton

03/13/2023, 1:46 AM
Good idea!
t

Tim Galvin

03/13/2023, 1:46 AM
I would naively hope that that would have it execute on the worker, yet will block until it is evaluated
s

Samuel Hinton

03/13/2023, 4:39 AM
I ended up solving this by passing around a pydantic object instead that contains the actual data, warnings, and exceptions, so I can access them in other tasks. It seems like a bad way to do state handling, but it works I guess
t

Tim Galvin

03/24/2023, 5:33 PM
I may have not appreciated the desired to run the test with the
with_option
. Is there something from this method you need for your tests? I feel like I am missing the point of your question. If the intent is to test your codes with some unit test framework, I would have though separating the funcion from the prefect task / flow annotations would be the ideal path But I say this having missed the intent of your question
s

Samuel Hinton

03/26/2023, 11:15 PM
It wasnt about tests. I have a giant collection of functions that correspond to code to hit different endpoints to gather data. How those functions are inserted into tasks changes depending on the a few factors (are we getting data for right now, for a day a week ago, or for the last six months). I wanted proper alerts/messages for errors in our code, but only warnings for errors in the endpoint (endpoint is down, unavailable, returns no data, returns malformed data etc). So now each data endpoint returns an object that contains the final data, all warnings generated, etc. I can then figure out what sort of alerts/messages to send in a generic callback