https://prefect.io logo
Powered by
Title
r

Ramzi

05/15/2022, 2:44 AM
I am in the process of building CI/CD pipeline using prefect 2.0 for cloud. I am running into an issue where i get the error: 
You have not configured default storage on the server
or set a storage to use for this deployment but this deployment is using a 
Kubernetes flow runner which requires remote storage.
I have already defined the s3 bucket as the storage in prior steps and make sure to even reset it as the default before hand. I have no problem creating the deployment locally only an issue when running it on github actions.
Wondering if there is a way to define the storage in the spec.
m

Malthe Karbo

05/15/2022, 10:43 AM
We ended up setting up storage in the cicd pipeline (first checking if storage is already set) - moreover, for kubernetes external storage is required - we created an aws iam user which has no other access than read/write and used that for the storage definition in prefect cloud
We use this script in our cicd pipeline 
#!/usr/bin/bash

set -e
# The below is empty on purpose - it is interactive CLI answers
# ENV variables are set in CI/CD github action
prefect storage create <<EOF
5
$FLOW_BUCKET
${AWS_REGION}

${AWS_ACCESS_KEY}

${AWS_SECRET_KEY}
cicd
y
EOF
and then we call it in github actions (here is a reduced script without company specific logic) 
- name: Setup default storage account in prefect cloud
        id: create-storage
        env:
          PREFECT_API_KEY: ${{ secrets.PREFECT_ACCOUNT_TOKEN }}
          PREFECT_API_URL: ${{ env.PREFECT_API_URL }} # <-- set in earlier step
          AWS_ACCESS_KEY: ${{ secrets.AWS_PROD_ACCESS_KEY }}
          AWS_SECRET_KEY: ${{ secrets.AWS_PROD_SECRET_KEY }}
          FLOW_BUCKET: mycompany-flows
          AWS_REGION: eu-central-1
        working-directory: ${{ env.working-directory }}
        run: |
          source .venv/bin/activate
          set -e
          prefect storage ls
          if ! $(prefect storage ls | grep cicd -q) ; then
            bash scripts/deployment/storage_create.sh
          else
            PREFECT_STORAGE_UUID=$(prefect storage ls | grep cicd | sed 's/│ //g' | sed -e 's/\(\S*\).*/\1/g')
            prefect storage set-default $PREFECT_STORAGE_UUID
          fi
(we then run one to create deployments and then we run one to create work queue - also first checking if it already exist)
🙌 2
a

Anna Geller

05/15/2022, 11:37 AM
@Ramzi there is a way of defining that in the spec, here is how you can do it: 
from prefect.blocks.storage import FileStorageBlock

@flow
def hello_flow():
    hi = say_hi()
    print_platform_info(wait_for=[hi])


DeploymentSpec(
    name="dev",
    flow=hello_flow,
    tags=["local"],
    flow_storage=FileStorageBlock(base_path="<s3://prefect-orion/flows>"),
)
make sure before you do that, you install s3fs: 
pip install s3fs
r

Ramzi

05/15/2022, 10:48 PM
Thank you @Anna Geller & @Malthe Karbo this is very helpful!
👍 1
#prefect-communityJoin Slack
Powered by