prefect-community #pacc-london-2023

I ran into an issue with the docker deployment, my image eventually built fine and I deployed it to the prefect cloud fine, had worker pool setup and worker running but when I tried to run it from prefect cloud it got an error that the working directory /opt/prefect/prefect_training didn't exist.

Jovan Sakovic

06/22/2023, 9:39 AM

@Nate trace of the Docker access denied ❌🧵

Gemma

06/22/2023, 9:55 AM

For those who are Docker familiar, why would this Dockerfile not be installing the requirements.txt file? If I specify

python -m pip install my_module

it works fine so it’s not pip, but the requirements.txt isn’t being picked up.

Copy code

# DockerFile
FROM prefecthq/prefect:2.10.16-python3.10
COPY requirements.txt /opt/prefect/prefect_training/requirements.txt
RUN python -m pip install -r requirements.txt
COPY . /opt/prefect/prefect_training/
WORKDIR /opt/prefect/prefect_training/

Mike O'Connor

06/22/2023, 10:23 AM

Issue for getting prefect-aws added to conda forge upvote https://github.com/PrefectHQ/prefect-aws/issues/287

Jeff Hale

06/22/2023, 10:39 AM

Prefect Terraform Provider: https://github.com/PrefectHQ/terraform-provider-prefect

Gemma

06/22/2023, 10:52 AM

Reading the docs about the GHA, it looks like you can’t have anything requiring input in the CLI. However, as I have

schedule: null

in my deployments, it always asks me

? Would you like to schedule when this flow runs? [y/n] (y):

. Is there a way of permanently disabling this? i.e. say “this is never going to have a schedule”. Using

schedule: null

schedule: false

and

schedule: {}

in the deployment yaml don’t work.

Niall O'Riordan

06/22/2023, 10:54 AM

Are there any recommend links/documentation about deploying with k8s?

Taylor Curran

06/22/2023, 11:55 AM

@Jovan Sakovic ecs default command https://github.com/PrefectHQ/prefect-aws/blob/f287c23787d19e4c71bd39618d2c171857d7a753/prefect_aws/workers/ecs_worker.py#L82

👀 1

incoming-webhook

06/22/2023, 12:04 PM

Hello from Prefect! Prefect Notifications

Taylor Curran

06/22/2023, 12:20 PM

@Mamatha https://github.com/PrefectHQ/prefect-demos/blob/main/flows/specialty_flows/partitions/partition_examples.py#L70

Copy code

@flow
def partition_ingestion_unique_pods(dataset_size, start_date, end_date, n_partitions):

    date_delta = end_date - start_date

    sub_flow()

    partition_period = date_delta / n_partitions

    for i in range(n_partitions):

        end_date = start_date + partition_period
        start_date = start_date + partition_period

        run_deployment(
            deployment="ingest_raw_orders/k8s_deployment",
            parameters={
                "start_date": start_date,
                "end_date": end_date,
                "dataset_size": dataset_size,
            },
        )

Interpreting results from run_deployments:

Copy code

c = run_deployment("child-flow-c/dep-child-c")
value = c.state.result()

https://github.com/taylor-curran/flow-patterns/blob/main/flows/subflows/task_wrapped_deployments.py#L46

ale

06/22/2023, 12:29 PM

May I suggest to not delete the auto-generated Dockerfile when using

dockerfile: auto

? It might be useful for debugging. Or maybe provide an optional flag to explicitly tell

prefect deploy

to keep the file

➕ 1

Adam Stone

06/22/2023, 12:40 PM

@Nate this is the issue for parsing new-style Slack webhook urls: https://github.com/caronc/apprise/issues/891

🙏 1

Taylor Curran

06/22/2023, 12:46 PM

@Giorgio Basile https://github.com/anna-geller/prefect-dataplatform/blob/main/dataplatform/blocks/dbt.py#L11 example of custom block

Taylor Curran

06/22/2023, 12:53 PM

@grandimk helm charts for k8s self hosting and use worker helm instead of old agent prefect service start as an ECS service I think Set up the ECS worker and workpool using the following guide: https://prefecthq.github.io/prefect-aws/ecs_guide/

Copy code

deployments:
- name: dep-sync-subflows
  version: null
  tags: [parent, sync]
  entrypoint: flows/sync_subflows.py:sync_subflows
  schedule: null
  work_pool:
    name: "my-ecs-workpool"
    job_variables:
      image: "{{ image_name }}"

🙌 1

Taylor Curran

06/22/2023, 1:10 PM

@Shivam Bhatnagar https://medium.com/the-prefect-blog/gcp-and-prefect-cloud-from-docker-container-to-cloud-vm-on-google-compute-engine-2dffa026d16b https://github.com/anna-geller/prefect-cloud-gcp/tree/2784e23b16ec1a59d48ba4730e0015eea00150c4

Taylor Curran

06/22/2023, 1:28 PM

@Gemma https://discourse.prefect.io/t/how-can-i-tell-if-there-is-an-issue-with-my-api-key/2185

Jeff Hale

06/22/2023, 1:55 PM

We’ll get back together at 3pm!

Jeff Hale

06/22/2023, 1:55 PM

Please take 2 minutes to fill out the feedback survey.

🙌 1

🦜 1

🎉 1

Jovan Sakovic

06/22/2023, 1:55 PM

Copy code

ecr-login:
	aws ecr get-login-password --region ${AWS_REGION} \
		| docker login --username AWS \
		--password-stdin ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.<http://amazonaws.com|amazonaws.com>

Taylor Curran

06/22/2023, 1:56 PM

@Gemma aws ecr get-login-password --region region | docker login --username AWS --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com

David Miller

06/22/2023, 1:56 PM

has anyone seen something like this?

Gemma

06/22/2023, 2:14 PM

@Taylor Curran This is the error I’m now getting:

Copy code

botocore.errorfactory.ClientException: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Fargate requires task definition to have execution role ARN to support ECR images.

Cong CHen

06/22/2023, 2:34 PM

Are there any Amazon Sagemaker examples that can be shared?

Taylor Curran

06/22/2023, 2:38 PM

So great to meet you all! Would love to connect with you all on LinkedIn https://www.linkedin.com/in/taylorcurran/

🎉 1

Gemma

06/22/2023, 2:48 PM

Copy code

build:
      - prefect_docker.deployments.steps.build_docker_image:
          id: build_image
          requires: prefect-docker>=0.3.0
          image_name: <http://724930167716.dkr.ecr.eu-west-2.amazonaws.com/prefect-training-2023|724930167716.dkr.ecr.eu-west-2.amazonaws.com/prefect-training-2023>
          tag: 0.0.1
          dockerfile: Dockerfile
          push: true
          platform: linux/amd64