Hi all, I want to run prefect on a docker container and then spin up

prefect ui+server

as well. My question is

prefect server

requires

docker and docker-compose

as mentioned in the docs. So I can't run it inside the docker container for prefect, right? How would I go about doing this then? Any help would be appreciated. TIA

Hi @Ankit - a few others in the community have spent some time on docker-in-docker setups with Prefect, but this isn't something we natively support; perhaps some of them can chime in with what they've found (there are a few threads about this). My recommendation would instead be to run Prefect in a container alongside Server as normal and configure Prefect from within its container.

Hey @nicholas thanks for the quick reply. I am trying to run prefect in its own container and run server + ui in their own separate container, by tweaking the original

docker-compose.yml

file from the repo. Would put all the containers in the same network, and hopefully that should work. I am not sure if what I am doing is exactly what you suggested.

I'm sure that'll work just fine!

@Ankit - I will be doing the same thing in the not too distant future. I think it is a reasonable deployment model to have the prefect components deployed as containers as opposed to part of a compose.

Hey @Greg Desmarais I finished up with the changes to the docker-compose to deploy all prefect ui+server dependencies in one go, just working on running my scripts in a container on the same network and trying to automate the process to register all flows as soon as the containers are restarted. If you do find some better approach, do let me know, would be happy to see as many ways as possible to do this.

I'm probably behind you, and might take a different approach. Currently I'm trying to get DockerStorage to ECR working - can't seem to get a valid authentication with ECR.

@Greg Desmarais how are you trying to authenticate with ECR?

Mostly poorly.

😆

Here is a mid stream state of the code I'm using for the storage creation:

class RightsizeFlow(Flow):
    def __init__(self, *args, **kwargs):
        super(RightsizeFlow, self).__init__(*args, **kwargs)

        ############## Storage ecr docker flow ##############
        # aws configuration
        aws_ecr_repo_name = "prefect_flows"
        aws_region = DEFAULT_REGION
        # See <https://github.com/awslabs/amazon-ecr-credential-helper>

        # 1. Reset Auth (hackish)
        # dkr_ecr_scrt = PrefectSecret("docker_ecr_login").run()

        # get_ecr_auth_token = ShellTask(helper_script="cd ~")
        # ecr_auth_token = get_ecr_auth_token.run(command=dkr_ecr_scrt)

        ecr_client = boto3.client('ecr', region_name=aws_region)
        ecr_token = ecr_client.get_authorization_token()

        # # Decode the aws token
        username, password = base64.b64decode(ecr_token['authorizationData'][0]['authorizationToken'])\
            .decode().split(':')
        ecr_registry_url = ecr_token['authorizationData'][0]['proxyEndpoint']

        # # Registry URL for prefect or docker push
        flow_registry_url = os.path.join(ecr_registry_url.replace('https://', ''), aws_ecr_repo_name)

        # see <https://docs.prefect.io/api/latest/environments/storage.html#docker>
        image_name = re.sub(r'[^a-zA-Z0-9_.-]', '_', self.name).lower()
        # docker tag prefect_flows:latest ########.<http://dkr.ecr.us-east-1.amazonaws.com/prefect_flows:latest|dkr.ecr.us-east-1.amazonaws.com/prefect_flows:latest>
        # registry_url = "########.<http://dkr.ecr.us-east-1.amazonaws.com/prefect_flows|dkr.ecr.us-east-1.amazonaws.com/prefect_flows>"

        # local_image (bool, optional): an optional flag whether or not to use a local docker image,
        # if True then a pull will not be attempted
        use_local_image = True

        storage = Docker(
            base_image='celsiustx/rightsize_99_standard_py37',
            image_name=image_name,
            local_image=use_local_image,
            registry_url=flow_registry_url
        )
        self.storage = storage

Hm, you should only need to authenticate the daemon on whatever machine this is executing, this might be overcomplicating it. Is your local daemon able to push/pull to your ECR?

Do you mean the DockerAgent?

Nope, the Docker Daemon, external to Prefect, should be authenticated

One thing that is a bit tough with Prefect is the number of processes/agents that need to be running for different things.

Totally understood @Greg Desmarais, that's something we're actively working on 🙂

Ah - so yes, the docker daemon is running.