Tracking Down a Server Log Entry

All day long, 24/7, a server log is recording “Server Connection Failed (1 of 5)” with no more information aside from the date and time. There is no session number. When the server is idle, they occur as often 30 seconds apart. When there’s activity they’re much further apart.

Just now, with the server idle, they were recorded at 6, 9 and 10 minute intervals.

Is there some way to track this down? Could it be a timer? There are two running but they’re at 300 and 3600 intervals.

I’m not seeing it on other servers.

That log entry is generated by the robot that is checking to make sure that Panorama Server is still running.

You can specify how often the robot will check the server, what the timeout should be for each check, and how many failures in a row you’ll tolerate before the server must be rebooted.

What is happening in your case is that the server is running some task that takes longer than the connection timeout time, but takes less time than the server check interval. If your server was set up like the one above, a task on the server that takes between 10 and 30 seconds would cause this log entry. So the robot ran the check but it failed because it took more than 10 seconds to complete. However, the next check worked fine - we know that because ther is no Server Connection Failed (2 of 5) in the log.

There’s no more information because the robot has no idea why it couldn’t complete it’s check. It could be a long running task, or it could even be that Panorama Server has frozen. If it’s the latter, eventually it will get to (5 of 5) and kill and relaunch the server.

Since you say this is happening when the server is idle, then yeah, it’s probably a timer. You might want to put some performance monitoring code into your timer routines.

If it was some web procedure, the Web Performance Analysis log may be helpful for tracking this down. In fact, this is the reason this log was added.

With this log, it should be possible to track down slow web procedures. But it may still take some sleuthing and patience.

And in the end - you may decide that this is fine. If you could look at the logs on the server you would see some of these log entries. Sometimes there is simply a web procedure that needs to run a bit longer to finish what it needs to do. So these log entries are a warning that you may want to investigate further, but not any kind of emergency or error. If you’re not expecting that some code running on the server may be running for longer than usual, then you should definitely look into what might be going on.

That resolves the mystery.

Aside from that, the logs have always been very valuable in Pan 6 as well as now. When debugging or tracking certain activities, they involve some review and sleuthing, but certainly offer a host of clues. I can’t imagine running a server without them.