Thread
#prefect-community
    a

    Adam Roderick

    2 years ago
    I have a flow failing in cloud that runs fine locally. I suspect something is up with the secrets. I have local secrets enabled, and I simply pasted the value into cloud secrets. The error is
    Exception raised while calling state handlers: TypeError('string indices must be integers')
    . I don't really now how to begin troubleshooting this in cloud. Any ideas?
    Chris White

    Chris White

    2 years ago
    Hey @Adam Roderick! What is most likely happening is that your Cloud Secret is not JSON-formatted correctly. Here is a simple example that produces that same error message:
    x = [1, 2, 3]
    x["0"]
    # TypeError: list indices must be integers or slices, not str
    There are two paths forward:
    a

    Adam Roderick

    2 years ago
    That error makes sense. I just can't repro it locally. And the secret is TOML formatted, which is a little different than json:
    jira = { server = "<https://datateer.atlassian.net/>", project = "DSD" }
    Chris White

    Chris White

    2 years ago
    1.) You can always pass local secrets to Cloud Flow runs through environment variables (
    PREFECT__CONTEXT__SECRETS__NAME="foo"
    ). You can provide these environment variables to your agent with the
    -e
    flag and your agent will then provide this env var to each flow run it submits. This sometimes allows for easier debugging. 2.) You can see what value your Cloud Secret is returning by doing the following: - start a python session with local secrets turned off:
    PREFECT__CLOUD__USE_LOCAL_SECRETS=false ipython
    - load your secret locally to see what is being returned:
    from prefect.client.secrets import Secret
    
    Secret("my_secret").get()
    you can also set secrets using the Python client, and then quickly iterate between setting your secret and retrieving it until it returns the value you expect
    a

    Adam Roderick

    2 years ago
    Thanks, Chris. That all works as expected. So I can rule out secrets. I'll keep looking at code to see what jumps out at me. Anything else to watch for between cloud and local?
    Chris White

    Chris White

    2 years ago
    Interesting 🧐 the only difference I can think of is the formatting of Secrets
    a

    Adam Roderick

    2 years ago
    What part of the state handler is actually running when a flow start up? Cloud is telling me the flow failed to start, so I assumed the state_handler hadn't run at all
    Chris White

    Chris White

    2 years ago
    Is your state handler attached to your Flow or an individual task?
    a

    Adam Roderick

    2 years ago
    The flow
    I just confirmed the flow runs in cloud if I remove the state handler
    Chris White

    Chris White

    2 years ago
    so each time your Flow undergoes a state transition your handler will be called. In the case of a Flow, there are very few state transitions. Usually: Scheduled --(state handler)--> Running --(state handler)--> Some Finished state
    a

    Adam Roderick

    2 years ago
    The flow gets scheduled according to cloud
    After 30-60 seconds, it fails with the exception
    The gantt chart tab says "Flow Run Did Not Start Successfully"
    The logs say "Handling state change from Scheduled to Running"
    Chris White

    Chris White

    2 years ago
    yea so the state handler is failing on the first state transition
    a

    Adam Roderick

    2 years ago
    Then the exception happens.
    Okay, yeah I wasn't sure if there was a state transition going into scheduled state
    Chris White

    Chris White

    2 years ago
    can you share your state handler code?
    a

    Adam Roderick

    2 years ago
    Yeah, can I DM?
    Chris White

    Chris White

    2 years ago
    yup
    a

    Adam Roderick

    2 years ago
    For future reference, I had a local secret specified in TOML format:
    name = {prop = "value"}
    . Because this was valid TOML syntax, it loaded as a dict and I was able to use my secret like this
    name["prop"]
    But in Cloud, the secrets are not stored in a .toml file, they are stored as strings. So my secret loaded in as a string, and
    name["prop"]
    failed. I changed my local secret to be a JSON string in the TOML file.
    name = '{"prop": "value"}'
    , and ensured the same format was in my Cloud secret. Both local and cloud now load consistently as a dict. Cloud logging did not give me the detail I needed--I needed a full traceback to clue me in to what line was failing. By building my docker container locally and running an interactive session, then starting a python session and running
    flow.run()
    , I was able to see the full traceback. Thanks much @Chris White for helping troubleshoot this
    Chris White

    Chris White

    2 years ago
    Thanks for the extremely helpful post-mortem!
    @Marvin archive “Issue with the type of Prefect Cloud Secrets”
    Marvin

    Marvin

    2 years ago