{"id":973,"date":"2018-10-05T09:08:09","date_gmt":"2018-10-05T09:08:09","guid":{"rendered":"http:\/\/smartfoxserver.com\/blog\/?p=973"},"modified":"2018-10-05T09:12:23","modified_gmt":"2018-10-05T09:12:23","slug":"sfs2x-multi-threading-demystified","status":"publish","type":"post","link":"https:\/\/smartfoxserver.com\/blog\/sfs2x-multi-threading-demystified\/","title":{"rendered":"SFS2X multi-threading demystified"},"content":{"rendered":"<p>Often times articles in this blog are inspired by questions and issues raised by our users and this\u00a0new entry is no exception. One aspect of SmartFoxServer that seems to intimidate developers is the <strong>multi-threaded environment<\/strong> behind custom Extensions, and the relative implications in terms of concurrency, scalability and performance.<\/p>\n<p>In this\u00a0new entry we&#8217;re going to\u00a0demystify the subject and demonstrate how simple and painless is writing server side code, even when many other things are running concurrently.<!--more--><\/p>\n<p>Before continuing, this article assumes you already know the basics of Extension development. If not, we suggest to hold your horses and take a look at this <a href=\"http:\/\/docs2x.smartfoxserver.com\/ExtensionsJava\/quick-start\" target=\"_blank\" rel=\"noopener noreferrer\">introductory tutorial first<\/a>.<\/p>\n<h2>\u00bb Taming the multi-threaded beast<\/h2>\n<p><img loading=\"lazy\" src=\"http:\/\/smartfoxserver.com\/blog\/wp-content\/uploads\/2018\/10\/taming-the-beast-copia.jpg\" alt=\"\" width=\"624\" height=\"424\" \/><br \/>\nMany of the support questions we receive concern with how to run Extension code efficiently, how to avoid concurrency issues or how to manage threads in the system. The good news is that SmartFoxServer 2X deals with many of these aspects behind the scenes and there&#8217;s only a handful of cases where manual intervention is required.<\/p>\n<p>For example most of the concurrency issues, such as thread safety and management, are handled for you by the system. The server monitors itself and is capable of adjusting the internal resources to sustain higher loads or deal with slow I\/O calls that may drain the thread pools.<\/p>\n<p>If you&#8217;re entirely new to programming in a multi-threading environment this article will clarify which situations needs a bit of extra care, should you encounter them during development. On the other hand, Java veterans can learn what exactly the server does for them and where they can intervene manually to get even better results.<\/p>\n<h2>\u00bb A quick view from the top<\/h2>\n<p>If you have consulted our documentation before you should be familiar with\u00a0this diagram:<\/p>\n<p><img loading=\"lazy\" class=\"size-full wp-image-974 aligncenter\" src=\"http:\/\/smartfoxserver.com\/blog\/wp-content\/uploads\/2018\/09\/server-structure.png\" alt=\"\" width=\"676\" height=\"515\" srcset=\"https:\/\/smartfoxserver.com\/blog\/wp-content\/uploads\/2018\/09\/server-structure.png 676w, https:\/\/smartfoxserver.com\/blog\/wp-content\/uploads\/2018\/09\/server-structure-300x229.png 300w, https:\/\/smartfoxserver.com\/blog\/wp-content\/uploads\/2018\/09\/server-structure-624x475.png 624w\" sizes=\"(max-width: 676px) 100vw, 676px\" \/><\/p>\n<p>It shows the different layers of the server architecture and the flow of data from client to server side logic and viceversa. All these colored blocks\u00a0manage a specific activity of the server and work in parallel using multiple threads, to provide maximum efficiency.<\/p>\n<p>In essence SFS2X uses an <strong>event driven<\/strong> style of programming where your code responds to\u00a0different types of occurrences:<\/p>\n<ul>\n<li><strong>Client requests<\/strong>: any request sent by connected clients via the API&#8217;s ExtensionRequest object<\/li>\n<li><strong>Server events<\/strong>: any event that the Extension has subscribed to and that is triggered by the server when\u00a0a specific situation occurs. (e.g. a User has joined a Room in the current Zone)<\/li>\n<li><strong>Scheduled events<\/strong>: similar to the above, these events are scheduled by your own Extension code to run in the future (e.g. game timers, time based triggers, etc.)<\/li>\n<\/ul>\n<p>Each of these events is going to call our code from a different thread at any time and thus, potentially, create a concurrency issue.<\/p>\n<p>While this may sound\u00a0a bit scary, in actuality\u00a0there&#8217;s only a handful of cases that require our attention. The most popular one is <strong>accessing shared objects<\/strong>, i.e. objects referenced by multiple classes being accessed simultaneously by different threads.<\/p>\n<p>Below is an example of a simple scenario, where the <em>MyData<\/em> instance looks like the potential candidate for multiple thread access and, without proper synchronization, it could lead to an incoherent state.<\/p>\n<pre class=\"brush: java; title: ; notranslate\" title=\"\">\r\npublic class MyEventHandler extends BaseServerEventHandler\r\n{\r\n\tprivate MyData data;\r\n\t\r\n\t@Override\r\n\tpublic handleServerEvent(ISFSEvent event) throws SFSException\r\n\t{\r\n\t\t\/\/ event logic...\r\n\t\t\r\n\t\tdata.changeState(someValue);\r\n\t}\r\n}\r\n<\/pre>\n<p>Before we do anything to fix this potential issue we need to remember that by default SFS2X <strong>creates a new instance<\/strong> of our event handler on every call, so in this case there is no concurrency problem. Each call will act on a different instance of <em>MyData<\/em>.<\/p>\n<p>Let&#8217;s say however that we&#8217;re referencing one or more objects shared by the top Zone Extension. In this case we would be in a situation where multiple threads could call our objects concurrently, such as in this snippet:<\/p>\n<pre class=\"brush: java; title: ; notranslate\" title=\"\">\r\npublic class MyEventHandler extends BaseServerEventHandler\r\n{\t\r\n\t@Override\r\n\tpublic handleServerEvent(ISFSEvent event) throws SFSException\r\n\t{\r\n\t\tGlobalScores scores = ((ParentExt) getParentExtension()).getGlobalScores();\r\n\t\tscores.add(userName, newScore);\r\n\t\t\r\n\t\t\/\/ more game logic...\r\n\t}\r\n}\r\n<\/pre>\n<p>Here the <em>GlobalScore<\/em> instance needs to be treated with extra care as the <em>add(&#8230;)<\/em> method will be called concurrently. For example if the class manages data internally using collections, we should make sure to use <a href='https:\/\/www.codejava.net\/java-core\/collections\/understanding-collections-and-thread-safety-in-java' target='_blank'>concurrent collections<\/a>.<\/p>\n<pre class=\"brush: java; title: ; notranslate\" title=\"\">\r\npublic class GlobalScore \r\n{\t\r\n\tprivate final Map&lt;String, Integer&gt; scoreTable;\r\n\t\r\n\tpublic GlobalScore()\r\n\t{\r\n\t\tscoreTable = new HashMap&lt;&gt;();\r\n\t}\r\n\t\r\n\tpublic void add(String userName, int value)\r\n\t{\r\n\t\tscoreTable.put(userName, value);\r\n\t}\r\n\t\r\n\t\/\/...\r\n}\r\n<\/pre>\n<p>Instead of using a regular <em>HashMap<\/em>, which is not thread safe, we should employ a <a href=\"https:\/\/docs.oracle.com\/javase\/8\/docs\/api\/java\/util\/concurrent\/ConcurrentHashMap.html\" target=\"_blank\">ConcurrentHashMap<\/a> from the <em>java.util.concurrent<\/em> package.<\/p>\n<p>On the other hand if the class used a database to handle the scores we should probably not worry, as database calls don&#8217;t need any synchronization.<\/p>\n<p>Let&#8217;s consider another example: imagine we have added a <strong>SFSEventType.ROOM_REMOVED<\/strong> listener on the server side to count the number of games that are completed. When the event triggers we invoke the parent Zone Extension and call its <em>increment()<\/em> method.<\/p>\n<pre class=\"brush: java; title: ; notranslate\" title=\"\">\r\npublic class MyEventHandler extends BaseServerEventHandler\r\n{\t\r\n\t@Override\r\n\tpublic handleServerEvent(ISFSEvent event) throws SFSException\r\n\t{\r\n\t\tRoom room = (Room) event.getParameter(SFSEventParam.ROOM);\r\n\t\r\n\t\tif (room.isGame())\r\n\t\t\t((MyZoneExt) getParentExtension()).increment();\r\n\t\t\r\n\t\t\/\/ more game logic...\r\n\t}\r\n}\r\n<\/pre>\n<p>This is what happens in the the Extension <em>increment()<\/em> method:<\/p>\n<pre class=\"brush: java; title: ; notranslate\" title=\"\">\r\npublic MyZoneExt extends SFSExtension\r\n{\r\n\tprivate int counter = 0;\r\n\t\r\n\t\/\/ ...\r\n\r\n\tpublic void increment()\r\n\t{\r\n\t\tcounter++;\r\n\t}\r\n}\r\n<\/pre>\n<p>Even a simple operation such as <em>counter++<\/em> can&#8217;t be considered thread safe and implemented this way it could cause issues. One solution would be to introduce a lock object and a <em>synchronized<\/em> block:<\/p>\n<pre class=\"brush: java; title: ; notranslate\" title=\"\">\r\npublic MyZoneExt extends SFSExtension\r\n{\r\n\tprivate int counter = 0;\r\n\tprivate final Object counterLock = new Object();\r\n\t\r\n\t\/\/ ...\r\n\r\n\tpublic void increment()\r\n\t{\r\n\t\tsynchronized(counterLock)\r\n\t\t{\r\n\t\t\tcounter++;\r\n\t\t}\r\n\t}\r\n}\r\n<\/pre>\n<p>This is fine, but it&#8217;s also a bit of an antiquated style of synchronization and there&#8217;s a much better solution: using an atomic variable such as <a href=\"https:\/\/docs.oracle.com\/javase\/8\/docs\/api\/java\/util\/concurrent\/atomic\/AtomicInteger.html\" target=\"_blank\">AtomicInteger<\/a> from the Java SDK. <\/p>\n<pre class=\"brush: java; title: ; notranslate\" title=\"\">\r\npublic MyZoneExt extends SFSExtension\r\n{\r\n\tprivate final AtomicInteger counter = new AtomicInteger();\r\n\t\r\n\t\/\/ ...\r\n\r\n\tpublic void increment()\r\n\t{\r\n\t\tcounter.incrementAndGet();\r\n\t}\r\n}\r\n<\/pre>\n<p><strong>Bottom line<\/strong>: objects referenced by multiple event handlers are potentially subject to concurrent calls and can interfere with each other. To deal with this correctly we can employ different solutions: <\/p>\n<ul>\n<li><a href='https:\/\/www.codejava.net\/java-core\/collections\/understanding-collections-and-thread-safety-in-java' target='_blank'>concurrent collections<\/a><\/li>\n<li><a href=\"https:\/\/www.baeldung.com\/java-concurrent-locks\" target=\"_blank\">locking API<\/a><\/li>\n<li><a href=\"https:\/\/www.baeldung.com\/java-atomic-variables\" target=\"_blank\">atomic variables<\/a><\/li>\n<li><a href=\"http:\/\/tutorials.jenkov.com\/java-concurrency\/synchronized.html\" target=\"_blank\">synchronization primitives<\/a><\/li>\n<\/ul>\n<p>Please note that synchronization primitives are at the bottom of our list as we recommend to always prioritize any of the other options, if applicable. <\/p>\n<h2>\u00bb Concurrency and SmartFoxServer API<\/h2>\n<p>Since Extension code uses quite a lot of objects from the SmartFoxServer SDK such as Room, User, Buddy etc., we have already made sure that all server API calls are thread safe and so are most of the objects obtained from them. <\/p>\n<p>There are exceptions to this rule with data wrapper classes such as <strong>SFSObject<\/strong>\/<strong>SFSArray<\/strong> and those inheriting the <strong>Variable<\/strong> interface such as <strong>SFSRoomVariable<\/strong>, <strong>SFSBuddyVariable<\/strong> etc.<\/p>\n<p>These are collection-type classes used for sending and receiving data but not for storing it long term. In other words these objects are used locally in methods to wrap data that needs to be sent out, but they are not recommended as replacement for standard collections.<\/p>\n<p>Barring the exceptions just mentioned you can assume everything else is thread safe in the server API, and you can refer to the <a href='http:\/\/docs2x.smartfoxserver.com\/api-docs\/javadoc\/server\/' target='_blank'>server side documentation<\/a> for extra details on the specific classes\/methods you need to use.<\/p>\n<h2>\u00bb\u00a0Creating and managing threads<\/h2>\n<p>One misconception we have found is that sometimes developers assume they need to create and manage their own threads for slow I\/O operations, such as database queries, or scheduling tasks for timers and delayed triggers.<\/p>\n<p>In reality SmartFoxServer removes most of these responsibilities from the hand of the developer:<\/p>\n<ul>\n<li><strong>Slow I\/O<\/strong> is handled by SFS2X via the internal auto load-balancing thread pools that can increase and decrease the number of threads on demand, without manual intervention. To learn more about this feature we recommend this <a href='http:\/\/docs2x.smartfoxserver.com\/ExtensionsJava\/advanced-concepts' target='_blank'>article from our doc website<\/a>.<\/li>\n<p><\/p>\n<li><strong>Scheduling tasks<\/strong> is handled by a specific entity called the <strong>TaskScheduler<\/strong> which allows to run delayed and repetitive tasks. The server offers a central scheduler that can be used without any setup. If you plan to create hundreds of custom tasks it might be better to create a new instance of the <strong>TaskScheduler<\/strong> in your own Extension.<br \/> See <a href=\"https:\/\/smartfoxserver.com\/blog\/how-to-schedule-timed-tasks-in-an-extension\/\" target=\"_blank\">this tutorial<\/a> if you want to see a scheduler code example.<\/li>\n<\/ul>\n<p><strong>Bottom line<\/strong>: it&#8217;s very unlikely you will ever need to deal with creating or managing threads manually when developing server side code. SFS2X can handle different workloads autonomously and if your project makes heavy use of slow I\/O calls you will just need to fine tune the thread pools from the AdminTool.<\/p>\n<p>Similarly if your code requires thousands of scheduled tasks, all you need is to create a custom TaskScheduler specifically configured for your needs.<\/p>\n<h2>\u00bb\u00a0Excessive blocking I\/O<\/h2>\n<p>Before wrapping up we&#8217;d like to spend a few more words on the topic of blocking I\/O as we&#8217;ve helped many clients with projects relying too heavily on it. Calling databases and external web services is typically required in non trivial applications and it&#8217;s usually of no concern. As we have mentioned in the previous section, SmartFoxServer 2X is able to deal with slow I\/O by monitoring the thread pools and resizing them accordingly.<\/p>\n<p>There is however a broader consideration regarding the amount of blocking I\/O that a project can use before it becomes a bottleneck and it affects the system&#8217;s scalability. For instance calling remote web services via the internet is usually a bad practice, as each call incurs in significant lag. It would be much better to organize the servers that need to interact with each other in the same private network, to minimize latency.<\/p>\n<p>Other recurring issues are too frequent calls to a database resulting in poor Extension response times, or too heavy SQL queries that take a long time to execute causing in scalability issues.<\/p>\n<p><strong>How to evaluate the impact on scalability?<\/strong><br \/>\nThere is no simple answer to this question other than: by testing!<\/p>\n<p><strong>Testing is fundamental<\/strong>. Too often we&#8217;ve seen customers launch a multi-server project in production without having run a single stress test, only to find later that there are major performance issues.<\/p>\n<p>We always emphasize that it is <strong>well worth the time and money<\/strong> invested in a few weeks of proper testing, to at least verify that no major bottlenecks are present in the system. Plus, basic testing is relatively simple and cheap as we <a href=\"https:\/\/smartfoxserver.com\/blog\/building-a-simple-stress-test-tool\/\" target=\"_blank\">have outlined in this article<\/a>.<\/p>\n<p>Running a few stress tests while monitoring the server&#8217;s dashboard will already provide lot of useful information:<\/p>\n<ul>\n<li><strong>CPU usage<\/strong>: gives you an idea of how many CCUs you can run per machine and can highlight potential issues in your code, such as calls that are too demanding or not optimized enough.<\/li>\n<li><strong>RAM usage<\/strong>: gives you clues on the amount of RAM necessary for your application and can uncover potential memory leaks.<\/li>\n<li><strong>Network usage<\/strong>: provides a ballpark figure of the bandwidth needed per user and overall requirements for an expected peak time. It may also help finding potential areas of improvement to reduce the network usage.<\/li>\n<li><strong>Threads and queues<\/strong>: this is an indicator of good vs. bad scalability. If you find the server&#8217;s queues are often busy and threads are increasing rapidly, that&#8217;s an indication of a performance issue that should be further investigated.<\/li>\n<\/ul>\n<h2>\u00bb\u00a0Conclusion<\/h2>\n<p>We hope to have clarified the fundamental approach to building your game code with SmartFoxServer 2X. If you have any questions about what we have discussed so far, feel free to post your comments in <a href=\"https:\/\/www.smartfoxserver.com\/forums\/index.php\" target=\"_blank\">our support board<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Often times articles in this blog are inspired by questions and issues raised by our users and this\u00a0new entry is no exception. One aspect of SmartFoxServer that seems to intimidate developers is the multi-threaded environment behind custom Extensions, and the relative implications in terms of concurrency, scalability and performance. In this\u00a0new entry we&#8217;re going to\u00a0demystify [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[100,31,101,40,34,59],"_links":{"self":[{"href":"https:\/\/smartfoxserver.com\/blog\/wp-json\/wp\/v2\/posts\/973"}],"collection":[{"href":"https:\/\/smartfoxserver.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/smartfoxserver.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/smartfoxserver.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/smartfoxserver.com\/blog\/wp-json\/wp\/v2\/comments?post=973"}],"version-history":[{"count":20,"href":"https:\/\/smartfoxserver.com\/blog\/wp-json\/wp\/v2\/posts\/973\/revisions"}],"predecessor-version":[{"id":1045,"href":"https:\/\/smartfoxserver.com\/blog\/wp-json\/wp\/v2\/posts\/973\/revisions\/1045"}],"wp:attachment":[{"href":"https:\/\/smartfoxserver.com\/blog\/wp-json\/wp\/v2\/media?parent=973"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/smartfoxserver.com\/blog\/wp-json\/wp\/v2\/categories?post=973"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/smartfoxserver.com\/blog\/wp-json\/wp\/v2\/tags?post=973"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}