I spin up a prefect local docker-compose service, having a few containers inside (local-spark, prefect-server, prefect-agent), I found that it’s hard to trigger spark job from local-spark container within prefect-server container as it has no spark lib installed, is there a way that I can trigger spark job from prefect-server container without installing extra libs?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
