https://prefect.io logo
Title
a

Adam Roderick

03/05/2020, 11:17 PM
I have a flow failing in cloud that runs fine locally. I suspect something is up with the secrets. I have local secrets enabled, and I simply pasted the value into cloud secrets. The error is
Exception raised while calling state handlers: TypeError('string indices must be integers')
. I don't really now how to begin troubleshooting this in cloud. Any ideas?
c

Chris White

03/05/2020, 11:20 PM
Hey @Adam Roderick! What is most likely happening is that your Cloud Secret is not JSON-formatted correctly. Here is a simple example that produces that same error message:
x = [1, 2, 3]
x["0"]
# TypeError: list indices must be integers or slices, not str
There are two paths forward:
a

Adam Roderick

03/05/2020, 11:22 PM
That error makes sense. I just can't repro it locally. And the secret is TOML formatted, which is a little different than json:
jira = { server = "<https://datateer.atlassian.net/>", project = "DSD" }
c

Chris White

03/05/2020, 11:24 PM
1.) You can always pass local secrets to Cloud Flow runs through environment variables (
PREFECT__CONTEXT__SECRETS__NAME="foo"
). You can provide these environment variables to your agent with the
-e
flag and your agent will then provide this env var to each flow run it submits. This sometimes allows for easier debugging. 2.) You can see what value your Cloud Secret is returning by doing the following: - start a python session with local secrets turned off:
PREFECT__CLOUD__USE_LOCAL_SECRETS=false ipython
- load your secret locally to see what is being returned:
from prefect.client.secrets import Secret

Secret("my_secret").get()
you can also set secrets using the Python client, and then quickly iterate between setting your secret and retrieving it until it returns the value you expect
a

Adam Roderick

03/05/2020, 11:28 PM
Thanks, Chris. That all works as expected. So I can rule out secrets. I'll keep looking at code to see what jumps out at me. Anything else to watch for between cloud and local?
c

Chris White

03/05/2020, 11:32 PM
Interesting 🧐 the only difference I can think of is the formatting of Secrets
a

Adam Roderick

03/06/2020, 12:01 AM
What part of the state handler is actually running when a flow start up? Cloud is telling me the flow failed to start, so I assumed the state_handler hadn't run at all
c

Chris White

03/06/2020, 12:07 AM
Is your state handler attached to your Flow or an individual task?
a

Adam Roderick

03/06/2020, 12:07 AM
The flow
I just confirmed the flow runs in cloud if I remove the state handler
c

Chris White

03/06/2020, 12:08 AM
so each time your Flow undergoes a state transition your handler will be called. In the case of a Flow, there are very few state transitions. Usually: Scheduled --(state handler)--> Running --(state handler)--> Some Finished state
a

Adam Roderick

03/06/2020, 12:11 AM
The flow gets scheduled according to cloud
After 30-60 seconds, it fails with the exception
The gantt chart tab says "Flow Run Did Not Start Successfully"
The logs say "Handling state change from Scheduled to Running"
c

Chris White

03/06/2020, 12:14 AM
yea so the state handler is failing on the first state transition
a

Adam Roderick

03/06/2020, 12:14 AM
Then the exception happens.
Okay, yeah I wasn't sure if there was a state transition going into scheduled state
c

Chris White

03/06/2020, 12:14 AM
can you share your state handler code?
a

Adam Roderick

03/06/2020, 12:14 AM
Yeah, can I DM?
c

Chris White

03/06/2020, 12:14 AM
yup
a

Adam Roderick

03/06/2020, 1:32 AM
For future reference, I had a local secret specified in TOML format:
name = {prop = "value"}
. Because this was valid TOML syntax, it loaded as a dict and I was able to use my secret like this
name["prop"]
But in Cloud, the secrets are not stored in a .toml file, they are stored as strings. So my secret loaded in as a string, and
name["prop"]
failed. I changed my local secret to be a JSON string in the TOML file.
name = '{"prop": "value"}'
, and ensured the same format was in my Cloud secret. Both local and cloud now load consistently as a dict. Cloud logging did not give me the detail I needed--I needed a full traceback to clue me in to what line was failing. By building my docker container locally and running an interactive session, then starting a python session and running
flow.run()
, I was able to see the full traceback. Thanks much @Chris White for helping troubleshoot this
💯 1
c

Chris White

03/06/2020, 1:33 AM
Thanks for the extremely helpful post-mortem!
@Marvin archive “Issue with the type of Prefect Cloud Secrets”