Hey there I have a problem with prefect server My setup used

Question

Hey there I have a problem with prefect server My setup used to work but now there might be something wrong with the version or something I m getting this error in the UI when running a flow ``` 3 18pm Exception raised while calling state handlers HTTPError 400 Client Error Bad Request for url <http localhost 4200 graphql alpha> ``` Any idea on how to start debugging

nicholas · Answer

Hi < Avi A> sounds like your agent might not be able to communicate with the server where do you have Prefect Server deployed

Avi A · Answer

I have it deployed on one server and the agent is on the other On the agent I have set up an SSH tunnel between them and `prefect agent start` works and communicates with the server It then receives the job but ends up like this On the agent itself it says nothing

nicholas · Answer

When you say the agent itself says nothing you mean you re not getting any output from the agent when you start it

Avi A · Answer

it says this ``` 2020 06 03 12 15 36 746 INFO agent | Waiting for flow runs 2020 06 03 12 18 33 032 INFO agent | Found 1 flow run s to submit for execution 2020 06 03 12 18 33 066 INFO agent | Deploying flow run 679fe6ae b245 47a0 97c7 a413468096ef``` but nothing further

Avi A · Answer

it actually does so for every flow is there a way to see logs on the agent end

nicholas · Answer

< Avi A> you can start your agent with a ` verbose` flag to get some more output this is an interesting problem I haven t seen before

Jeremiah · Answer

< Avi A> is that original message at 3 18 something you re seeing in the logs in the UI

Jeremiah · Answer

Just trying to suss out where the communication break is happening if so sounds like your agent can pull work white check mark and runs can communicate logs back to the server white check mark but your state handler is having an issue x

Avi A · Answer

yes here s an example screenshot

Jeremiah · Answer

And are you using any custom state handlers or just the automatic ones for setting states

Avi A · Answer

at the moment I can t restore the issue b c the communication is off entirely task stuck on `Submitted for execution` will update

Avi A · Answer

I m not customizing the state handlers

Avi A · Answer

AFAIK

Jeremiah · Answer

Ok

Avi A · Answer

ohhh wait I have set the Slack state handler It used to work fine but maybe that s causing the problem Where should the WEBHOOK secret be defined On the agent or on the server

Jeremiah · Answer

I think it ll need to be on the agent since the flows it launches will inherit its config

Avi A · Answer

any idea on how to get the webhook URL secret for the app after installing or do I have to reinstall and get a new URL

Avi A · Answer

I m talking about the slack app I somehow lost the URL

Jeremiah · Answer

grimacing I m really sorry I dont know the answer to that maybe someone else will

Avi A · Answer

nm I ll just reinstall the app

Avi A · Answer

btw the URL can t be restored It says so upon installation slightly smiling face ```Please store it securely If you lose your webhook URL you will need to uninstall and reinstall the Prefect slack integration ```

Avi A · Answer

< Jeremiah> now I m using ` verbose` and getting this on the agent when submitting the job ``` 2020 06 03 13 55 02 998 DEBUG agent | Querying for flow runs 2020 06 03 13 55 03 064 DEBUG agent | Found flow runs 14e59135 b8fb 4d35 9140 d5eda3a709aa 2020 06 03 13 55 03 065 DEBUG agent | Querying flow run metadata 2020 06 03 13 55 03 098 INFO agent | Found 1 flow run s to submit for execution 2020 06 03 13 55 03 099 DEBUG agent | Updating states for flow run 14e59135 b8fb 4d35 9140 d5eda3a709aa 2020 06 03 13 55 03 101 DEBUG agent | Flow run 14e59135 b8fb 4d35 9140 d5eda3a709aa is in a Scheduled state updating to Submitted 2020 06 03 13 55 03 103 DEBUG agent | Next query for flow runs in 0 25 seconds 2020 06 03 13 55 03 137 INFO agent | Deploying flow run 14e59135 b8fb 4d35 9140 d5eda3a709aa 2020 06 03 13 55 03 141 DEBUG agent | Submitted flow run 14e59135 b8fb 4d35 9140 d5eda3a709aa to process PID 26768 2020 06 03 13 55 03 160 DEBUG agent | Completed flow run submission id 14e59135 b8fb 4d35 9140 d5eda3a709aa ``` On the UI different server it still says `Submitted for execution`

Jeremiah · Answer

Ok so far that s all expected My guess is that when the flow spins up and attempts to enter a `Running` state the state handler fails due to the slack config and prevents progress This is an issue we ve seen in the past since the state handler is responsible for communicate state updates back to the server including errors errors in the state handler itself get tricky to handle

Avi A · Answer

I ve set the secret now shouldn t it work

Avi A · Answer

I ll disable it for now see what happens

Avi A · Answer

I ve disabled slack and getting the same behavior

Avi A · Answer

I m using `LocalDaskExecutor` as a remote environment perhaps that s related But it worked fine before I mean it failed before but it reported the logs ok to the UI

nicholas · Answer

< Avi A> this is looking like it s probably something unique to your setup with the SSH tunnel would you mind putting together a minimum code repro and opening a GitHub issue

Avi A · Answer

I don t think it s related to the tunnel I ve set it up before and it worked perfectly I m setting a tunnel on ports 8080 and 4200 In any case I ve changed the config and using it without the ssh tunnel now In `~ prefect config toml` I put ``` server host = <http prefect> ``` Since the agent connects well to the server and receives the job submission I figured this was a good setup Is there anything else to configure for this to work

Jeremiah · Answer

I think you are encountering a different issue now Originally your flow was able to communicate with the server at least enough to set a failed state when the slack handler failed Now it sounds like it can t set states at all which means either the flow isn t starting or once the flow gets to the dask cluster it s unable to communicate with the server In both cases the most likely culprit is an inability to reach the API It s going to be very hard for us to diagnose a communication issue from here I suggest opening a GitHub issue with a description of the environment for visibility and testing

Avi A · Answer

I hear you but I still think that opening an issue would be a waste of time for you guys How could you reproduce the issue with this amount of info btw I also changed the environment back to the default so it s probably not an issue with Dask

nicholas · Answer

< Avi A> on Github you ll have access to a broader support base for the issue this is equally as difficult to diagnose here It seems that your configuration is set correctly given that your setup was working correctly previously

Avi A · Answer

ok I ve now run the agent with the flag ` f show flow logs` to output the logs from the run itself and indeed it reports problems with the state ``` 2020 06 03 15 06 48 INFO prefect CloudFlowRunner | Beginning Flow run for Smart Groups 2020 06 03 15 06 49 DEBUG prefect CloudFlowRunner | Failed to retrieve flow state with error RegistryError Multiple classes with name StateSchema were found Please use the full module qualified path ```

Avi A · Answer

< nicholas> I know why I was getting these errors Our internal piece of code was also registering an object named `State` under the name `StateSchema` and it caused a collision with Prefect s `StateSchema` Question is what can I do with it

Jeremiah · Answer

< Avi A> thanks so much for following up on this I ve got a growing list of reasons that Prefect serialization is getting too complex and I ll add this to the list I just opened a PR that might solve your issue would you mind giving it a look These are the only two places that the StateSchema appears to be referenced as a class name reference

Avi A · Answer

Thanks a lot < Jeremiah> I installed from that branch and it works for my case

Jeremiah · Answer

Glad to hear it