I have a flow failing in cloud that runs fine locally. I suspect something is up with the secrets. I have local secrets enabled, and I simply pasted the value into cloud secrets. The error is

Exception raised while calling state handlers: TypeError('string indices must be integers')

. I don't really now how to begin troubleshooting this in cloud. Any ideas?

Hey @Adam Roderick! What is most likely happening is that your Cloud Secret is not JSON-formatted correctly. Here is a simple example that produces that same error message:

x = [1, 2, 3]
x["0"]
# TypeError: list indices must be integers or slices, not str

There are two paths forward:

That error makes sense. I just can't repro it locally. And the secret is TOML formatted, which is a little different than json:

jira = { server = "<https://datateer.atlassian.net/>", project = "DSD" }

1.) You can always pass local secrets to Cloud Flow runs through environment variables (

PREFECT__CONTEXT__SECRETS__NAME="foo"

). You can provide these environment variables to your agent with the

-e

flag and your agent will then provide this env var to each flow run it submits. This sometimes allows for easier debugging. 2.) You can see what value your Cloud Secret is returning by doing the following: - start a python session with local secrets turned off:

PREFECT__CLOUD__USE_LOCAL_SECRETS=false ipython

- load your secret locally to see what is being returned:

from prefect.client.secrets import Secret

Secret("my_secret").get()

Thanks, Chris. That all works as expected. So I can rule out secrets. I'll keep looking at code to see what jumps out at me. Anything else to watch for between cloud and local?

Interesting 🧐 the only difference I can think of is the formatting of Secrets

What part of the state handler is actually running when a flow start up? Cloud is telling me the flow failed to start, so I assumed the state_handler hadn't run at all

Is your state handler attached to your Flow or an individual task?

The flow

so each time your Flow undergoes a state transition your handler will be called. In the case of a Flow, there are very few state transitions. Usually: Scheduled --(state handler)--> Running --(state handler)--> Some Finished state

The flow gets scheduled according to cloud

yea so the state handler is failing on the first state transition

Then the exception happens.

can you share your state handler code?

Yeah, can I DM?

yup

For future reference, I had a local secret specified in TOML format:

name = {prop = "value"}

. Because this was valid TOML syntax, it loaded as a dict and I was able to use my secret like this

name["prop"]

But in Cloud, the secrets are not stored in a .toml file, they are stored as strings. So my secret loaded in as a string, and

name["prop"]

failed. I changed my local secret to be a JSON string in the TOML file.

name = '{"prop": "value"}'

, and ensured the same format was in my Cloud secret. Both local and cloud now load consistently as a dict. Cloud logging did not give me the detail I needed--I needed a full traceback to clue me in to what line was failing. By building my docker container locally and running an interactive session, then starting a python session and running

flow.run()

, I was able to see the full traceback. Thanks much @Chris White for helping troubleshoot this

Thanks for the extremely helpful post-mortem!