<@ULVA73B9P> when I try to run the flow below, som...
# ask-marvin
m
@Marvin when I try to run the flow below, some tasks are created but the sheet_id is just a list of one str: @flow(timeout_seconds=1800, log_prints=True) async def smartsheet_data_pipeline(): api_token = await load_credential("smartsheet-api-token") sheet_id = await load_credential("smartsheet-sheets-id") gcp_bucket = await load_credential("gcp-bucket") sa_project_id = await load_credential("sa-project-id-dev") bq_project_id = await load_credential("bq-project-id-dev") env = "dev" sheet_data = get_sheet_data.map(unmapped(api_token), sheet_id)
m
thought for 76.3 seconds
Based on the research and your code snippet, I can identify the issue. The problem is likely related to how the credential is being loaded and passed to the mapping operation. Let me explain the issue and provide a solution: 1. When you call
await load_credential("smartsheet-sheets-id")
, it's likely returning a single string that contains multiple sheet IDs, rather than a list of individual sheet IDs. 2. To fix this, you'll need to ensure that
sheet_id
is properly formatted as a list before using it with
.map()
. Here are two potential solutions: Solution 1 - If your credential is stored as a comma-separated string:
Copy code
python
@flow(timeout_seconds=1800, log_prints=True)
async def smartsheet_data_pipeline():
    api_token = await load_credential("smartsheet-api-token")
    sheet_id_str = await load_credential("smartsheet-sheets-id")
    # Convert the string to a list by splitting on commas
    sheet_ids = [id.strip() for id in sheet_id_str.split(',')]
    
    # Rest of your credentials
    gcp_bucket = await load_credential("gcp-bucket")
    sa_project_id = await load_credential("sa-project-id-dev")
    bq_project_id = await load_credential("bq-project-id-dev")
    env = "dev"

    sheet_data = get_sheet_data.map(unmapped(api_token), sheet_ids)
Solution 2 - If you can store the credential as a JSON array:
Copy code
python
import json

@flow(timeout_seconds=1800, log_prints=True)
async def smartsheet_data_pipeline():
    api_token = await load_credential("smartsheet-api-token")
    sheet_id_json = await load_credential("smartsheet-sheets-id")
    # Parse the JSON string into a list
    sheet_ids = json.loads(sheet_id_json)
    
    # Rest of your credentials
    gcp_bucket = await load_credential("gcp-bucket")
    sa_project_id = await load_credential("sa-project-id-dev")
    bq_project_id = await load_credential("bq-project-id-dev")
    env = "dev"

    sheet_data = get_sheet_data.map(unmapped(api_token), sheet_ids)
To implement this fix, you'll need to: 1. First, check how your sheet IDs are currently stored in your credentials. You can add a print statement to see the format:
Copy code
python
sheet_id = await load_credential("smartsheet-sheets-id")
print(f"Sheet ID type: {type(sheet_id)}")
print(f"Sheet ID value: {sheet_id}")
2. Then, modify how you store the credential in Prefect: - For Solution 1: Store it as a comma-separated string (e.g., "123,456,789") - For Solution 2: Store it as a JSON array string (e.g., '["123", "456", "789"]') 3. Update your code using one of the solutions above based on your chosen storage format. The key is to ensure that
sheet_id
is actually a list of values when it's passed to the
.map()
function. The
unmapped(api_token)
correctly handles the API token that should be the same for all iterations, while the sheet IDs should be a proper list that
.map()
can iterate over. Would you like me to help you verify which format your credential is currently stored in, or would you like more details about either of these solutions?