System memory always reaching 100% after 24 hours

[foxboy]

Hi SFS2X Team,

I have a very weird problem on my SFS2X server. I am using an Ubuntu server with 24GB of RAM. I have allocated 8GB for SFS2X. Things are working fine but I found a very strange behavior on my system memory after 24 hours of operation. As you can see below, the system memory is reaching almost 100% while the memory usage of JVM is just around 8GB as configured. If I stop SFS2X process then the total system memory usage falls back again to less than 1GB. At first, I suspected my SFS Extension base classes. Maybe it is not getting cleared by the garbage collector and persists for one time, so I made sure that the temporary classes to save my game state are not used anymore after operation. This didn't work so I decided to put them into individual jar files and drop them on the __lib__ folder. But still I am experiencing this problem. Do you have any similar case from other users?

By the way, on my SFS2X admin tool, it is reaching 100% usage sometime and then goes back again to around 20% and my application is still running fine. Also on the OS level, when memory usage reaches 100%, the system will now start using the Swap memory and things are still running fine. However, I have to restart my server when this happens just to make sure it wont consume 100% of the swap memory as well. But I don't like the idea of restarting my server on a daily basis, my game should be available 24/7 to avoid game interruptions.

Tasks: 189 total, 1 running, 187 sleeping, 0 stopped, 1 zombie
Cpu(s): 1.6%us, 0.4%sy, 0.0%ni, 39.4%id, 58.5%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 24733876k total, 17359696k used, 7374180k free, 302784k buffers
Swap: 2096060k total, 0k used, 2096060k free, 8069952k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1198 root 20 0 9359m 7.5g 11m S 16 31.6 145:11.74 java
22913 www-data 20 0 0 0 0 Z 2 0.0 0:00.13 apache2 <defunct>
1 root 20 0 23940 2016 1324 S 0 0.0 0:01.89 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0 0.0 0:02.29 ksoftirqd/0
4 root 20 0 0 0 0 S 0 0.0 0:38.21 kworker/0:0
6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
7 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
8 root 20 0 0 0 0 S 0 0.0 2:05.36 kworker/1:0
9 root 20 0 0 0 0 S 0 0.0 0:00.93 ksoftirqd/1
10 root 20 0 0 0 0 S 0 0.0 0:28.75 kworker/0:1
11 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/2
12 root 20 0 0 0 0 S 0 0.0 0:35.64 kworker/2:0
13 root 20 0 0 0 0 S 0 0.0 0:00.71 ksoftirqd/2
14 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/3
15 root 20 0 0 0 0 S 0 0.0 0:11.40 kworker/3:0
16 root 20 0 0 0 0 S 0 0.0 0:00.27 ksoftirqd/3
17 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/4
18 root 20 0 0 0 0 S 0 0.0 0:26.53 kworker/4:0
19 root 20 0 0 0 0 S 0 0.0 0:20.09 ksoftirqd/4
20 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/5
21 root 20 0 0 0 0 S 0 0.0 0:22.72 kworker/5:0
22 root 20 0 0 0 0 S 0 0.0 0:00.49 ksoftirqd/5

[Lapo]

Hi,
there could be a memory leak in your code, but I find it strange that the JVM doesn't fail with an OutOfMemory exception of some sort.
How did you configure the max heap size?

The easiest thing to do is running VisualVM on the server itself, and take a look at what it growing. I am 90% sure that it is heap memory, if it was PermGen it would have failed eventually instead of growing indefinitely.
With VisualVM you can run the memory profiler and take a look at which objects are cluttering the memory forcing continuos growth.
http://visualvm.java.net/

[foxboy]

Thanks for the prompt reply. I found threads about PermGen errors but that is not visible on my logs so I suspected something on the heap as well. That I am creating dynamic objects and JVM is not clearing them because it is still in use and my application seems to be working as the memory is big enough to handle it. Anyway, I will install visualvm and see how it goes.

[michael123]

hi, @Lapo -- Site Admin , i have the same problem, before restart sfs2x-service zabbix monitor java RSS-- RES memory status, it was stable and RSS was 15g, but after i restarted itdid nothing) , it become unstable and RSS is higher, almos 20g. how it can happened? i try to set JVM -- -XX:MaxDirectMemorySize=3g, but not working, it is still high. --- smartFox 2X version 2.13.0
and sfs2x admin dashboard memory used max value is 3.2g, average value is 200M. so i think it is not heap memory issue , right?

[Lapo]

Hi,
I would recommend starting a new thread trying to explain this in more details.
For starters, I have no idea what you're referring to when mentioning RSS / RES memory. Please clarify what tools you're using to monitor the JVM memory usage. Ideally it would be best to use a JVM native monitor, rather than a system one as system monitors cannot distinguish how the JVM memory is used internally (and how it is partitioned).

Also please specify the specs of your server and what custom JRE settings you're using. Additionally please list what additional libs/frameworks you're using server-side, if any.

As regards this:

-XX:MaxDirectMemorySize=3g

Why do you need 3GB of direct off-heap memory? SFS2X does not use off-heap memory.

Thanks.