Hi all do you guys have some good tutorials on howw to run apache spark with prefect? can you guiys point me to some good tutorials of resources?

Hi Juan, Prefect doesn’t have a direct integration with Spark, but you can still use Prefect to orchestrate your Spark jobs. One way to do this is by using Prefect tasks to submit Spark jobs and manage their execution. You can create custom tasks that interact with the Spark cluster, submit jobs, and monitor their progress.

Following up on this. I have one task that gets submitted twice in a flow using the concurrent task runner. Something like:

@flow(task_runner=ConcurrentRunner()):
   task.submit(param=1)
   task.submit(param=2)

the task calls a spark job. Both tasks started concurrently according to the Prefect Server UI, but the 2nd failed with a long error message, and one of the line in the error log says “SparkContext was shut down”. Any idea why? @Taylor Curran any tip would be appreciated!

Sorry ross sometimes the threads get away from me! Could you repost these errors to the community channel