Hi all! Does anyone know the best practise for catching custom exceptions within prefect tasks? I have an exception which isnt a code failure, its just data not being available, and I’d like to send a slack warning when this happens but not report flow failure. I hoped that this would work:

try:
        data = task.submit(start, end, logger=logger)
    except DataWarning as e:
        logger.warning(e)
        send_incoming_webhook_message(
            slack_webhook,
            f"Warning raised for {task_name} for {start} to {end} due to {e}",
        )

But no dice, doesnt play well with the future object. Im reworking all of this to not use exceptions and just pass dictionaries around, but thought knowing the best practise might be useful

Try it without the

.submit

. Just call

task

so it runs synchronously. This will cause the remainder to the flow to block until that

task

function has been evaluated.

Wont that stop execution via dask workers though and cause it to run in the agent itself?

Oh good question. The answer is I don't know

Good idea!

I would naively hope that that would have it execute on the worker, yet will block until it is evaluated

I ended up solving this by passing around a pydantic object instead that contains the actual data, warnings, and exceptions, so I can access them in other tasks. It seems like a bad way to do state handling, but it works I guess

I may have not appreciated the desired to run the test with the

with_option

. Is there something from this method you need for your tests? I feel like I am missing the point of your question. If the intent is to test your codes with some unit test framework, I would have though separating the funcion from the prefect task / flow annotations would be the ideal path But I say this having missed the intent of your question

It wasnt about tests. I have a giant collection of functions that correspond to code to hit different endpoints to gather data. How those functions are inserted into tasks changes depending on the a few factors (are we getting data for right now, for a day a week ago, or for the last six months). I wanted proper alerts/messages for errors in our code, but only warnings for errors in the endpoint (endpoint is down, unavailable, returns no data, returns malformed data etc). So now each data endpoint returns an object that contains the final data, all warnings generated, etc. I can then figure out what sort of alerts/messages to send in a generic callback