Page 1 of 2

strange network peaks

Posted: 19 May 2017, 18:59
by Tatanan
Hi,

Suddenly we register strange network peaks the makes user not receive messages or they receive the messages after many many seconds, and the extension queue grows to strange values. It seems that during some time the server can not send any message.
Any idea? How can I monitor if an extension handler is slow?
I Attach some graphs from the Admin Tool.
screenshot network.png
(46.42 KiB) Not downloaded yet

screenshot queue.png
(52.46 KiB) Not downloaded yet

screenshot cpu.png
(40.7 KiB) Not downloaded yet

Re: strange network peaks

Posted: 20 May 2017, 09:37
by Lapo
Hi,
one instance is not enough to understand if this is a real issue or a temporary "hiccup".
The spike might have been anything, from a garbage collection cycle to something external to the JVM, such as another process suddenly reducing the available CPU and slowing everything down.

Does this happen at regular times? Or is it dependent on server load?

thanks

Re: strange network peaks

Posted: 31 May 2017, 08:52
by Tatanan
Solved. It was a problem with the database.

Thanks you anyway and
sorry for the inconvenience

Re: strange network peaks

Posted: 31 May 2017, 13:45
by Lapo
No problem, glad you find out what caused it. :)

cheers

Re: strange network peaks

Posted: 12 Apr 2018, 14:34
by gabrielvpy
I'm having the same issue. I managed to find most of the DB processes that were causing this issue and solved them. But I'm still getting some peaks. It was kind of easy to find the issues at first because all the workers (SFSWorker:Ext:1, 2, 3 and 4) stoped working if one of them was waiting for de DB to respond.

But now is different, I get peaks but there is no worker waiting for the DB to respond. Is there a another way to monitor this. What else could be causing this issue if it's not the DB?

Re: strange network peaks

Posted: 12 Apr 2018, 15:09
by Lapo
What peaks do you mean exactly?
Peaks in the usage of the CPU? Can you show me a graph taken from the admin tool?

Thanks

Re: strange network peaks

Posted: 12 Apr 2018, 15:18
by gabrielvpy
t like the first image Tatanan posted in the first post of this thread. It's a network peak.

Re: strange network peaks

Posted: 12 Apr 2018, 15:38
by Lapo
Network peaks can be normal, depending on how frequent and wide they are.
Also they can depend on what's going on in the server's queues.

If you have screenshot we can take a look. Would be cool to actually see the screenshots of both views in the Dashboard (network and queues)

Thanks

Re: strange network peaks

Posted: 12 Apr 2018, 19:12
by gabrielvpy
Here's a screenshot that we just took.

Image
Image

Re: strange network peaks

Posted: 13 Apr 2018, 07:25
by Lapo
Thanks,
it's difficult to speculate here. One isolated spike in 10 minutes is not sign of anything bad, unless this happens often and with similar magnitude. There could be all kinds of reasons for that occurrence.

The fact that both the green and red lines overlap indicates that the server received a sudden "big input" (i.e. many requests) and immediately sent the relative responses (hence the overlap) which in turn suggests the server is doing its job correctly. Also the CPU does not show a higher use correlated with the spike, so everything looks fine from the SFS side.

It looks very much like the spike was generated from outside the server.

Re: strange network peaks

Posted: 04 Jul 2018, 14:02
by gabrielvpy
Hi, Lapo.

I have a new question regarding this situation. Yesterday morning we were having an unusual amount of spikes and not that many players were playing at that moment. Also, we realized we were having internet issues at the office. So a question roused, could it be that the sampling in the dashboard is being done locally (our office in Paraguay) and the fact that we have an awful internet connection is what is causing these spikes in our dashboard but not on the server (AWS in Virginia).

So I went back home and opened a dashboard there and came back to the office and started monitoring from the office and from my place. Then I noticed that we still get somes spikes here in the office's dashboard but not in my place's dashboard.

My conclusion is that the sampling is done locally and that's why in my house (1 computer using a 10Mbps connection) has no spikes, unlike at the office (10 computers using a 10Mbps connection) where we do get spikes.

Is that the case?

Re: strange network peaks

Posted: 04 Jul 2018, 16:11
by Lapo
gabrielvpy wrote:Hi, Lapo.

I have a new question regarding this situation. Yesterday morning we were having an unusual amount of spikes and not that many players were playing at that moment. Also, we realized we were having internet issues at the office. So a question roused, could it be that the sampling in the dashboard is being done locally (our office in Paraguay) and the fact that we have an awful internet connection is what is causing these spikes in our dashboard but not on the server (AWS in Virginia).

When you say "spikes" what you are you referring to exactly? Which parameter(s) in the server?
Generally speaking, the AdminTool simply put into graphs samples calculated by the server, based on server side parameters.

So I went back home and opened a dashboard there and came back to the office and started monitoring from the office and from my place. Then I noticed that we still get somes spikes here in the office's dashboard but not in my place's dashboard.

My conclusion is that the sampling is done locally and that's why in my house (1 computer using a 10Mbps connection) has no spikes, unlike at the office (10 computers using a 10Mbps connection) where we do get spikes.

No, there is no sampling going on in the AdminTool. The Admin just renders graphs from samples coming from the server side.

Cheers

Re: strange network peaks

Posted: 04 Jul 2018, 17:17
by gabrielvpy
When I say spikes I mean spikes in outgoing and incoming data. Like the ones in the last image I posted.

Re: strange network peaks

Posted: 05 Jul 2018, 07:10
by Lapo
gabrielvpy wrote:When I say spikes I mean spikes in outgoing and incoming data. Like the ones in the last image I posted.

The AdminTool adds a little bit of extra in/out traffic to the server but in the order of Kilobytes, certainly not Megabytes.

As I mentioned earlier a traffic spike (even a large one) does not represent any particular problem. It can happen for example if the garbage collector pauses the JVM for a moment (can be few ms. or hundreds of them, depending on the load). When this happens data that should be sent out is held into a queue and sent when the GC is done. This can result in a spike whose size depends on the current load of the server.

Cheers

Re: strange network peaks

Posted: 05 Jul 2018, 12:35
by gabrielvpy
I don't think that's the case. As I mentioned, I'm monitoring our server from 2 seperate locations. In my house there is only 1 machine online and I see no spikes there. But in the office where we have at least 10 machines online and the same bandwith as my house we get the spikes. So our conclusion is that we are seeing this spikes because too many machines are compiting for the intenet connection.
We made another experiment. We started monitoring another server (from the office), one that has no traffic at all, and we got spikes at the same time we were having spikes in our busy server. We think that proves our theory wich is, our internet conecction is what is causing the spikes and the server is running smoothly.
I just wanted to add that in case anyone else runs into this situation.