Thread
#prefect-community
    Aqib Fayyaz

    Aqib Fayyaz

    6 months ago
    Can i use docker-storage for flow with ubtuntu 20.04 as a base image not prefect in Dockerfile?
    Matthias

    Matthias

    6 months ago
    You sure can! As long as the proper dependencies and entrypoint are set that it. Note however that installing python in a “raw” ubuntu is not as straightforward as you might think (been there done that). So why would you want to do that?
    Aqib Fayyaz

    Aqib Fayyaz

    6 months ago
    I need to install spark and when using ubuntu as base image while registering flow it fails health checks and does not register the flow
    Matthias

    Matthias

    6 months ago
    But you can use a Python image, install JDK and pip install spark. It should install all it's dependencies
    Maybe you can even start from Prefect’s base image for that?
    Anna Geller

    Anna Geller

    6 months ago
    To add to Matthias excellent answer, if you are running Spark just within a Docker image, you are not taking advantage of Spark right? Where is your Spark cluster running? Usually, you should be able to just run a "Spark submit" command with e.g. ShellTask to submit the job to your cluster Also, we've already had this conversation about Spark and packaging it into a Docker image, let me check if I can find it