Kotlin Coroutine and Smartfox Extension threads

[tobaonghiep]

Hi team,

Currently we are switching to Kotlin on Smartfox, as I understood that with Java is blocking and it block the extension thread during processing time. But with Kotlin my code when processing the request is like this:

Code: Select all

   
    override fun handleClientRequest(user: User, requestData: ISFSObject) {
        ServerScope.launch {
            try {
                handleClientRequestInternal(user, requestData)
            } catch (e: ServerErrorException) {
                log.error("Error handling admin, sending error to admin: {}", user.name, e)
                sendException(user, EMPTY_COMMAND_ID, e.errorCode.code, "Error handling admin, sending error to admin:${user.name}")
            } catch (e: Exception) {
                log.error("Error handling client request: {}", user.name, e)
                sendException(user,
                    EMPTY_COMMAND_ID,
                    ServerErrorCode.SERVER_ERROR_EXCEPTION.code,
                    "Error handling admin request, sending error to admin:${user.name}")
            }
        }
    }


With ServerScope.launch {...} We actually sending the whole process to Kotlin Coroutine and the extension thread will be release quickly.

Currently my server serving for about 12K users and with this Kotlin we are facing with I/O call request timeout fequently. From what I observe the request got 2 cases, some stuck and not be sent out until get timeout. Another case is some request have been already sent out and result have been responsed from the called server but some how that result is not be processed by SFS server and the request getting timeout.

When using Java we also getting that issue but very little. Mostly because of the blocking, server use up all the extension threads, the client request need to be wait until its turn so this request timeout rarely happen.

Please also have a look at my config thead pool on server side, my server using Linux and it have 32 cores and 64Gb Ram.
https://prnt.sc/1v4nznh

I wonder is there anything else I need to config more on Linux config or I need to increase the thr ead pool on SFS server.

[Lapo]

Hi,
I don't have enough experience with Kotlin to be able to help here, sorry
If I understand it correctly. coroutines in Kotlin are suspendable functions, so you have full control of when they are suspended and resumed, am I wrong? If so most of the performance/scalability issues might stem from how you manage those states.

If you're running lots of slow I/O operations in your Extension code, such as external HTTP or database calls you may be able to take advantage of coroutines, although these things should always be tested under heavy load to see if they behave correctly and if there are any improvements over a regular thread pool.

When using Java we also getting that issue but very little. Mostly because of the blocking, server use up all the extension threads, the client request need to be wait until its turn so this request timeout rarely happen.

In the screenshot you have sent the System Thread pool settings use lower values. I'd recommend setting the same values you used for the Extension Thread pool. This is because the System pool is also responsible for notifying events to Extensions. This means that if you run slow I/O in an Extension Event listener you are using a System thread.

I wonder is there anything else I need to config more on Linux config or I need to increase the thr ead pool on SFS server.

That's difficult for us to say as we don't know what your Extension code does, and all the relative variables at play. However by looking at the different settings between the two thread pools I'd recommend using the same settings as explained.

A 32 core machine should definitely be able to keep up with 12K CCU, unless you're running an insane amount of external I/O

Cheers

[tobaonghiep]

Hi Lapo,

Thanks for your advice I also think of the setting of thread pool but not so confident to adjust it . Let me update it 1st and see any improvement.

[Lapo]

If you're setting up your own thread pool you would make a duplicate of what we already provide, plus we offer a more sophisticated thread pool that monitors itself and is able to adjust itself at runtime.

So, in this case I would highly recommend to use what we already provide instead of reinventing the wheel.
Make sure to read how thread pools work in SFS2X here:
http://docs2x.smartfoxserver.com/Extens ... d-concepts

Cheers