https://prefect.io logo
s

Sonny

01/16/2021, 1:44 AM
whats a right way to submit a spark job from prefect task?
r

Riley Hun

01/16/2021, 2:31 AM
I personally use the
subprocess
python module to submit spark jobs and you should be able to use
asyncio.create_subprocess_shell
to execute multiple jobs simultaneously. Prefect also has the built-in
ShellTask
task
s

Sonny

01/16/2021, 3:47 AM
how to handle success / fail of job run?