Ramzi
05/15/2022, 2:44 AMYou have not configured default storage on the server
or set a storage to use for this deployment but this deployment is using a
Kubernetes flow runner which requires remote storage.
I have already defined the s3 bucket as the storage in prior steps and make sure to even reset it as the default before hand. I have no problem creating the deployment locally only an issue when running it on github actions.Malthe Karbo
05/15/2022, 10:43 AM#!/usr/bin/bash
set -e
# The below is empty on purpose - it is interactive CLI answers
# ENV variables are set in CI/CD github action
prefect storage create <<EOF
5
$FLOW_BUCKET
${AWS_REGION}
${AWS_ACCESS_KEY}
${AWS_SECRET_KEY}
cicd
y
EOF
and then we call it in github actions (here is a reduced script without company specific logic)
- name: Setup default storage account in prefect cloud
id: create-storage
env:
PREFECT_API_KEY: ${{ secrets.PREFECT_ACCOUNT_TOKEN }}
PREFECT_API_URL: ${{ env.PREFECT_API_URL }} # <-- set in earlier step
AWS_ACCESS_KEY: ${{ secrets.AWS_PROD_ACCESS_KEY }}
AWS_SECRET_KEY: ${{ secrets.AWS_PROD_SECRET_KEY }}
FLOW_BUCKET: mycompany-flows
AWS_REGION: eu-central-1
working-directory: ${{ env.working-directory }}
run: |
source .venv/bin/activate
set -e
prefect storage ls
if ! $(prefect storage ls | grep cicd -q) ; then
bash scripts/deployment/storage_create.sh
else
PREFECT_STORAGE_UUID=$(prefect storage ls | grep cicd | sed 's/│ //g' | sed -e 's/\(\S*\).*/\1/g')
prefect storage set-default $PREFECT_STORAGE_UUID
fi
(we then run one to create deployments and then we run one to create work queue - also first checking if it already exist)Anna Geller
from prefect.blocks.storage import FileStorageBlock
@flow
def hello_flow():
hi = say_hi()
print_platform_info(wait_for=[hi])
DeploymentSpec(
name="dev",
flow=hello_flow,
tags=["local"],
flow_storage=FileStorageBlock(base_path="<s3://prefect-orion/flows>"),
)
make sure before you do that, you install s3fs:
pip install s3fs
Ramzi
05/15/2022, 10:48 PM