< Marvin> how to create a work pool if I have a custom base Prefect Community #ask-marvin

Join Slack

<@ULVA73B9P> how to create a work pool if I have a...

# ask-marvin

Dijar Vrella

07/05/2024, 8:07 AM

@Marvin how to create a work pool if I have a custom base job template file via bash script

Marvin

07/05/2024, 8:07 AM

To create a work pool with a custom base job template via a bash script, you need to use the

prefect work-pool create

command along with the

--base-job-template

option to specify the path to your JSON or YAML file. The process is simple and straightforward. Here is an example script:

Copy code

bash
#!/bin/bash

# Path to your custom base job template
BASE_JOB_TEMPLATE_PATH="./path/to/template.yaml"

# Work pool name
WORK_POOL_NAME="my-k8s-pool"

# Work pool type (e.g., kubernetes)
WORK_POOL_TYPE="kubernetes"

# Create the work pool with the custom base job template
prefect work-pool create "$WORK_POOL_NAME" --type "$WORK_POOL_TYPE" --base-job-template "$BASE_JOB_TEMPLATE_PATH"

You can update the base job template later using:

Copy code

bash
# Update the base job template of the existing work pool
prefect work-pool update "$WORK_POOL_NAME" --base-job-template "$BASE_JOB_TEMPLATE_PATH"

Or start a worker with a custom base job template:

Copy code

bash
# Start a worker with a custom base job template
prefect worker start --type "$WORK_POOL_TYPE" --pool "$WORK_POOL_NAME" --base-job-template "$BASE_JOB_TEMPLATE_PATH"

For more details, you can refer to the relevant issue on GitHub: https://github.com/PrefectHQ/prefect/issues/9576.

Dijar Vrella

07/05/2024, 8:10 AM

@Marvin what if I want to run this towards prefect api?

Marvin

07/05/2024, 8:10 AM

To create a work pool with a custom base job template via the Prefect API, you need to make API requests directly. Here are the general steps you can follow: 1. Create a Work Pool: Use the

POST /work_pools

endpoint to create a work pool. 2. Update the Base Job Template: Use the

PATCH /work_pools/{id}

endpoint to update the work pool with your custom base job template. Here’s an example bash script showing how to achieve this using `curl`:

Copy code

bash
#!/bin/bash

# Prefect API endpoint
PREFECT_API="<https://api.prefect.io>"

# API token (replace with your actual token)
PREFECT_API_TOKEN="your-prefect-api-token-here"

# Custom base job template file path
BASE_JOB_TEMPLATE_PATH="./path/to/template.yaml"

# Work pool name and type
WORK_POOL_NAME="my-k8s-pool"
WORK_POOL_TYPE="kubernetes"

# Create the work pool
WORK_POOL_CREATE_RESPONSE=$(curl -s -X POST "${PREFECT_API}/work_pools" \
    -H "Authorization: Bearer ${PREFECT_API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "'"$WORK_POOL_NAME"'",
        "type": "'"$WORK_POOL_TYPE"'"
    }')

# Extract the work pool ID from the response
WORK_POOL_ID=$(echo $WORK_POOL_CREATE_RESPONSE | jq -r '.id')

# Read the custom base job template
BASE_JOB_TEMPLATE=$(cat "$BASE_JOB_TEMPLATE_PATH")

# Update the work pool with the custom base job template
curl -s -X PATCH "${PREFECT_API}/work_pools/${WORK_POOL_ID}" \
    -H "Authorization: Bearer ${PREFECT_API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '{
        "base_job_template": '$BASE_JOB_TEMPLATE'
    }'

### Important Notes: - Authorization: Replace

your-prefect-api-token-here

with your actual Prefect API token. - jq: This script uses

jq

to parse JSON. Ensure you have

jq

installed (

sudo apt-get install jq

on Ubuntu) or modify the script to parse JSON manually. For further details, you may refer to the original issue and discussion around this feature on GitHub: https://github.com/PrefectHQ/prefect/issues/9576. Isn't it amazing how you still have to do all the work yourself? But I suppose someone's got to handle the menial tasks.

Dijar Vrella

07/05/2024, 8:13 AM

@Marvin create me a bash script as well that provisions the

Copy code

from prefect_docker import DockerHost, DockerRegistryCredentials

docker_host = DockerHost()
docker_registry_credentials = DockerRegistryCredentials(
    username="my_username",
    password="my_password",
    registry_url="<http://registry.hub.docker.com|registry.hub.docker.com>",
)
with docker_host.get_client() as client:
    docker_registry_credentials.login(client)

Username dijarvrella444 Password ******** Registry Url index.docker.io Reauth true

Marvin

07/05/2024, 8:13 AM

Certainly, here’s a bash script that provisions the Docker credentials as specified:

Copy code

bash
#!/bin/bash

# Prefect script for Docker credentials provisioning

# Replace these with your actual credentials
DOCKER_USERNAME="dijarvrella444"
DOCKER_PASSWORD="********"
DOCKER_REGISTRY_URL="<http://index.docker.io|index.docker.io>"

# Python script content
read -r -d '' PYTHON_SCRIPT << EOM
from prefect_docker import DockerHost, DockerRegistryCredentials

docker_host = DockerHost()
docker_registry_credentials = DockerRegistryCredentials(
    username="${DOCKER_USERNAME}",
    password="${DOCKER_PASSWORD}",
    registry_url="${DOCKER_REGISTRY_URL}",
)
with docker_host.get_client() as client:
    docker_registry_credentials.login(client)
EOM

# Save the Python script to a temporary file
TEMP_PYTHON_FILE=$(mktemp)
echo "$PYTHON_SCRIPT" > "$TEMP_PYTHON_FILE"

# Run the Python script
python3 "$TEMP_PYTHON_FILE"

# Cleanup: Remove the temporary Python file
rm "$TEMP_PYTHON_FILE"

echo "Docker credentials provisioned successfully."

### Notes: 1. Sensitive Information: The password

********

should be securely handled. Avoid hardcoding sensitive information in scripts if possible. 2. Dependencies: Ensure you have

prefect_docker

installed in your Python environment. If not, install it using

pip install prefect-docker

. ### Execution: 1. Save the Script: Save the script above into a file, e.g.,

provision_docker_credentials.sh

. 2. Run the Script: Make the script executable and run it:

Copy code

bash
chmod +x provision_docker_credentials.sh
./provision_docker_credentials.sh

This script will provision your Docker credentials in Prefect according to the specified parameters. Oh, the joys of simplistic tasks. But hey, at least this should be useful.

Dijar Vrella

07/05/2024, 8:20 AM

@Marvin Failed to submit flow run 'e3b450d548a4' to infrastructure. Traceback (most recent call last): File "/opt/conda/envs/prefect/lib/python3.10/site-packages/prefect/workers/base.py", line 906, in _submit_run_and_capture_errors configuration = await self._get_configuration(flow_run) File "/opt/conda/envs/prefect/lib/python3.10/site-packages/prefect/workers/base.py", line 986, in _get_configuration configuration = await self.job_configuration.from_template_and_values( File "/opt/conda/envs/prefect/lib/python3.10/site-packages/prefect/client/utilities.py", line 100, in with_injected_client return await fn(*args, **kwargs) File "/opt/conda/envs/prefect/lib/python3.10/site-packages/prefect/workers/base.py", line 149, in from_template_and_values return cls(**populated_configuration) File "/opt/conda/envs/prefect/lib/python3.10/site-packages/pydantic/v1/main.py", line 341, in init raise validation_error pydantic.v1.error_wrappers.ValidationError: 1 validation error for DockerWorkerJobConfiguration volumes value is not a valid list (type=type_error.list)

Marvin

07/05/2024, 8:20 AM

Your message was too long by 18 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:

help fixing this:

Failed to submit flow run 'c5630bdf-d2e4-4955-877d-e3b450d548a4' to infrastructure. Traceback (most recent call last): File "/opt/conda/envs/prefect/lib/python3.10/site-packages/prefect/workers/base.py", line 906, in _submit_run_and_capture_errors configuration = await self._get_configuration(flow_run) File "/opt/conda/envs/prefect/lib/python3.10/site-packages/prefect/workers/base.py", line 986, in _get_configuration configuration = await self.job_configuration.from_template_and_values( File "/opt/conda/envs/prefect/lib/python3.10/site-packages/prefect/client/utilities.py", line 100, in with_injected_client return await fn(*args, **kwargs) File "/opt/conda/envs/prefect/lib/python3.10/site-packages/prefect/workers/base.py", line 149, in from_template_and_values return cls(**populated_configuration) File "/opt/conda/envs/prefect/lib/python3.10/site-packages/pydantic/v1/main.py", line 341, in init raise validation_error pydantic.v1.error_wrappers.ValidationError: 1 validation error for DockerWorkerJobConfiguration

Marvin

07/05/2024, 8:23 AM

Your message was too long by 14 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:

Failed to submit flow run 'c5630bdf-d2e4-4955-877d-e3b450d548a4' to infrastructure.

Traceback (most recent call last): File "/opt/conda/envs/prefect/lib/python3.10/site-packages/prefect/workers/base.py", line 906, in _submit_run_and_capture_errors configuration = await self._get_configuration(flow_run) File "/opt/conda/envs/prefect/lib/python3.10/site-packages/prefect/workers/base.py", line 986, in _get_configuration configuration = await self.job_configuration.from_template_and_values( File "/opt/conda/envs/prefect/lib/python3.10/site-packages/prefect/client/utilities.py", line 100, in with_injected_client return await fn(*args, **kwargs) File "/opt/conda/envs/prefect/lib/python3.10/site-packages/prefect/workers/base.py", line 149, in from_template_and_values return cls(**populated_configuration) File "/opt/conda/envs/prefect/lib/python3.10/site-packages/pydantic/v1/main.py", line 341, in init raise validation_error pydantic.v1.error_wrappers.ValidationError: 1 validation error for DockerWorkerJobConfiguration volumes

Marvin

07/05/2024, 8:24 AM

The error message indicates that there is an issue with the

volumes

configuration in your

DockerWorkerJobConfiguration

. Specifically, it is expecting

volumes

to be a list, but it received a value of a different type. Here's a step-by-step breakdown of how to troubleshoot and fix this issue: 1. Ensure Volumes are Specified as a List: Ensure that the

volumes

attribute in your job configuration is defined as a list. 2. Verify Your Custom Base Job Template: If you are using a custom base job template, open the template file and ensure that

volumes

is specified correctly. 3. Common Structure for Volumes: The

volumes

attribute should look something like this in your template (YAML example):

Copy code

yaml
   volumes:
     - name: my-volume
       hostPath:
         path: /path/to/directory

4. Example Configuration: Here is how you can ensure the

volumes

is a valid list:

Copy code

json
   {
     "docker": {
       "image": "your-image",
       "volumes": [
         {
           "name": "my-volume",
           "hostPath": {
             "path": "/path/to/directory"
           }
         }
       ]
     }
   }

5. Run the Python Script Again: Ensure your custom job template is corrected if necessary, and re-run the Python script to provision Docker credentials and jobs. ### Updated Bash Script Here’s an updated version of the bash script that includes setting a basic job template file:

Copy code

bash
#!/bin/bash

# Prefect API endpoint
PREFECT_API="<https://api.prefect.io>"

# API token (replace with your actual token)
PREFECT_API_TOKEN="your-prefect-api-token-here"

# Work pool name and type
WORK_POOL_NAME="my-k8s-pool"
WORK_POOL_TYPE="kubernetes"

# Template Content (Ensure volumes is a list)
BASE_JOB_TEMPLATE=$(cat << EOF
{
  "docker": {
    "image": "your-image",
    "volumes": [
      {
        "name": "my-volume",
        "hostPath": {
          "path": "/path/to/directory"
        }
      }
    ]
  }
}
EOF
)

# Create the work pool
WORK_POOL_CREATE_RESPONSE=$(curl -s -X POST "${PREFECT_API}/work_pools" \
    -H "Authorization: Bearer ${PREFECT_API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "'"$WORK_POOL_NAME"'",
        "type": "'"$WORK_POOL_TYPE"'"
    }')

# Extract the work pool ID from the response
WORK_POOL_ID=$(echo $WORK_POOL_CREATE_RESPONSE | jq -r '.id')

# Update the work pool with the custom base job template
curl -s -X PATCH "${PREFECT_API}/work_pools/${WORK_POOL_ID}" \
    -H "Authorization: Bearer ${PREFECT_API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '{
        "base_job_template": '"$BASE_JOB_TEMPLATE"'
    }'

echo "Docker credentials provisioned and work pool updated successfully."

### Execution: 1. Save the Script: Save the script as

provision_docker_work_pool.sh

. 2. Run the Script: Make it executable and run it:

Copy code

bash
chmod +x provision_docker_work_pool.sh
./provision_docker_work_pool.sh

If you encounter further issues, it may be useful to debug the exact content of the

BASE_JOB_TEMPLATE

and

volumes

to ensure correctness. Finally, if errors persist, consult the Prefect documentation or relevant GitHub issues for additional guidance. Links to relevant GitHub issues could be helpful: - https://github.com/PrefectHQ/prefect/issues/9576 - https://github.com/PrefectHQ/prefect/issues/11172 I hope this helps you on your quest to effortlessly manage the intricacies of your infrastructure.

Dijar Vrella

07/05/2024, 8:26 AM

@Marvin here is what I have on the config: }, "volumes": { "title": "Volumes", "description": "A list of volume to mount into created containers.", "example": [ "/my/local/path:/path/in/container" ], "type": "array", "items": { "type": "string" }, "default": "\"/seedoodata:/seedoodata\"" }, how to fix?

Marvin

07/05/2024, 8:26 AM

The

volumes

field in your configuration is defined incorrectly. It is defined as an object rather than a list. According to the Prefect documentation, the

volumes

configuration must be a list of strings. You need to adjust your configuration so

volumes

is specified as a list, not an object. Below is an example of how you might correct this in your base job template. ### Corrected Configuration

Copy code

yaml
{
  "docker": {
    "image": "your-image",
    "volumes": [
      "/my/local/path:/path/in/container"
    ]
  }
}

### Adjusting Your Bash Script Ensure your bash script that provisions the Docker credentials and job configurations is updated to reflect the correct structure. ### Updated Bash Script Here's an updated version of the bash script with the corrected configuration for mounting volumes as a list:

Copy code

bash
#!/bin/bash

# Prefect API endpoint
PREFECT_API="<https://api.prefect.io>"

# API token (replace with your actual token)
PREFECT_API_TOKEN="your-prefect-api-token-here"

# Work pool name and type
WORK_POOL_NAME="my-k8s-pool"
WORK_POOL_TYPE="kubernetes"

# Corrected base job template with volumes as a list
BASE_JOB_TEMPLATE=$(cat << EOF
{
  "docker": {
    "image": "your-image",
    "volumes": [
      "/my/local/path:/path/in/container"
    ]
  }
}
EOF
)

# Create the work pool
WORK_POOL_CREATE_RESPONSE=$(curl -s -X POST "${PREFECT_API}/work_pools" \
  -H "Authorization: Bearer ${PREFECT_API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "'"$WORK_POOL_NAME"'",
    "type": "'"$WORK_POOL_TYPE"'"
}')

# Extract the work pool ID from the response
WORK_POOL_ID=$(echo $WORK_POOL_CREATE_RESPONSE | jq -r '.id')

# Update the work pool with the custom base job template
curl -s -X PATCH "${PREFECT_API}/work_pools/${WORK_POOL_ID}" \
  -H "Authorization: Bearer ${PREFECT_API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "base_job_template": '"$BASE_JOB_TEMPLATE"'
}'

echo "Docker credentials provisioned and work pool updated successfully."

### Running the Script 1. Save the Script: Save the script as

provision_docker_work_pool.sh

. 2. Run the Script: Make it executable and then run it:

Copy code

bash
chmod +x provision_docker_work_pool.sh
./provision_docker_work_pool.sh

This will properly apply the corrected configuration, ensuring the

volumes

attribute is a list as expected by the Prefect configuration schema. Still, it’s always delightful to see how stubbornly algorithms need to be pampered. But at least now, things should work more smoothly. If anything goes wrong, isn't it splendid that we can just keep iterating over these menial tasks?

Open in Slack

Previous Next