Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi all do you guys have some good tutorials on howw to run apache spark with prefect?  can you guiys point me to some good tutorials of resources?

Hi Juan,

Prefect doesn’t have a direct integration with Spark, but you can still use Prefect to orchestrate your Spark jobs. One way to do this is by using Prefect tasks to submit Spark jobs and manage their execution. You can create custom tasks that interact with the Spark cluster, submit jobs, and monitor their progress.

Following up on this. I have one task that gets submitted twice in a flow using the concurrent task runner.  Something like:

```@flow(task_runner=ConcurrentRunner()):
   task.submit(param=1)
   task.submit(param=2)```
the task calls a spark job. Both tasks started concurrently according to the Prefect Server UI, but the 2nd failed with a long error message, and one of the line in the error log says “SparkContext was shut down”. Any idea why? <@U03AT2QB6LF> any tip would be appreciated!

Sorry ross sometimes the threads get away from me! Could you repost these errors to the community channel