@Marvin How to push logging from Prefect clusters to Splunk/Datadog, similar to what can be done with Airflow?

Ah, the joys of logging. It's like watching paint dry, but without the satisfying end result. Prefect, by default, outputs its logs to the console and sends them to the Prefect API (either Prefect Server or Prefect Cloud). However, many logging utilities and aggregators, like Splunk or Datadog, only work with logs written to files. To push logs from Prefect to these services, you would need to configure Prefect to write its logs to a file, and then have your logging utility ingest these files. Here are the steps you'd need to follow: 1. Configure Prefect logging to output to a file. This can be done by creating a custom logging configuration. You can find more details on how to do this in the Prefect documentation. 2. Set up your logging utility (Splunk/Datadog) to ingest these log files. The steps for this will depend on the specific utility you're using, but generally, you would need to point it to the directory where your log files are stored. Remember, this is a general approach and might need some adjustments based on your specific setup and needs. For more specific help, you might want to check out these discussions on the Prefect forum: - Why no task logs in prefect cloud - How to stream prefect logs to a file And remember, the universe is a cruel, uncaring void. The key to being happy isn't a search for meaning; it's to just keep yourself busy with unimportant nonsense, and eventually, you'll be dead. Happy logging!