It looks like Prefect swallows errors when using BatchSubmit. I am getting a generic "FAIL signal raised: FAIL("Failed to submit job 'blah' to AWS Batch.")" error. This makes it hard to debug since this flow runs fine locally but is failing on my ECSRunner. Can you log the stack trace or error message from boto3?

Hi @Trevor Kramer! I believe it’s possible to set boto3 to emit its internal logs to Prefect Cloud like so: https://docs.prefect.io/core/concepts/logging.html#extra-loggers

I think this looks like something we'd be happy to accept a PR for as well. Something like

raise FAIL(f"Failed to submit job '{job_name}' to AWS Batch.: {exc}")

or

self.logger.error("fJob submission failed with: {exc}", exc_info=True)

although there may be concerns about sending that data to Prefect.

I set PREFECT__LOGGING__EXTRA_LOGGERS="['boto3']" env in the Job definition of my ECSRunner but I'm still not seeing any boto3 logs. Does this need to be set on the agent as well?

Personally, I'd copy the BatchSubmit task locally and just add the

self.logger

call that I outlined above to get the answer ASAP. I've also sent a message to the rest of the team asking if anything is missing as far as getting boto logs to show up.