Join Slack

whats a right way to submit a spark job from prefect task?

# ask-community

Sonny

01/16/2021, 1:44 AM

whats a right way to submit a spark job from prefect task?

Riley Hun

01/16/2021, 2:31 AM

I personally use the

subprocess

python module to submit spark jobs and you should be able to use

asyncio.create_subprocess_shell

to execute multiple jobs simultaneously. Prefect also has the built-in

ShellTask

task

Sonny

01/16/2021, 3:47 AM

how to handle success / fail of job run?

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.