One of the questions that often pops up in our forums is “how do I run a stress test on my game”?<\/p>\n
There are several ways in which this can be done. A simple way to stress test your server side Extension is to build a client application that acts as a player, essentially a “bot”, which can be replicated several hundreds or thousands of times to simulate a large amount of clients.<\/p>\n
For this example we will build a simple Java client using the standard SFS2X Java API<\/strong> which can be downloaded from here<\/a>. The same could be done using C# or AS3 etc…<\/p>\n The simple client will connect to the server, login as guest, join a specific Room and start sending messages. This basic example can serve as a simple template to build more complex interactions for your tests.<\/p>\n Before we proceed with the creation of the client logic let’s see how the “Replicator” will work. With this name we mean the top-level application\u00a0that will take a generic client implementation and will generate many copies at a constant\u00a0interval, until all “test bots” are ready.<\/p>\n The class will startup by loading an external config.properties<\/em> file which looks like this:<\/p>\n The properties are:<\/p>\n Once these parameters are loaded the test will start by generating all the requested clients via a thread-pool based scheduled executor (ScheduledThreadPoolExecutor<\/a>)<\/p>\n In order for\u00a0the test class to be “neutral” to the Replicator we have created a base class called\u00a0BaseStressClient\u00a0<\/strong>which defines a couple of methods:<\/p>\n The startUp()<\/strong> method is where the client code gets initialized and it must be overridden in the child class. The onShutDown(…)\u00a0<\/strong>method is invoked by\u00a0the client implementation to signal the Replicator that the client has disconnected, so that they \u00a0can be disposed.<\/p>\n This\u00a0is the code for the client itself:<\/p>\n The class extends the BaseStressClient parent and instantiates the SmartFox API. We then proceed by setting up the event listeners and connection parameters. Finally we invoke the sfs.connect(…)<\/strong> method to get started.<\/p>\n Notice that we also declared a static ScheduledExecutorService<\/strong> at the top of the declarations. This is going to be used as the main scheduler for sending public messages at specific intervals, in this case one message every two second.<\/p>\n We chose to make it static so that we can share the same instance across all client objects, this way only one thread will take care of all our messages. If you plan to\u00a0run thousands of clients or use\u00a0faster message rates\u00a0you will probably need to increase the number of threads in the constructor.<\/p>\n When replicating many hundreds \/ thousands of clients we should keep in mind that every new instance of the SmartFox<\/strong> class (the main API class) will use a certain amount of resources, namely RAM and Java threads.<\/p>\n For this simple example each instance should take ~1MB of heap memory which means we\u00a0can expect 1000 clients to take approximately\u00a01GB of RAM. In this case you will probably need to adjust the heap settings of the JVM by adding the usual -Xmx<\/strong> switch to the startup script.<\/p>\n Similarly the number of threads in the JVM will increase by 2 units for each new client generated, so for 1000 clients we\u00a0will end up with 2000 threads, which is a pretty\u00a0high number.<\/p>\n Any relatively modern machine (e.g 2-4 cores, 4GB RAM) should be able to run at least 1000 clients, although the complexity of the client logic and the rate of network messages may reduce this value.<\/p>\n On more powerful\u00a0hardware, such as a dedicated server, you should be able to run several thousands of CCU without much effort.<\/p>\n Before we start running the test let’s make sure we have all the necessary monitoring tool to watch the basic performance parameters:<\/p>\n Here are some important suggestions to make sure that a stress test is executed successfully:<\/p>\n What happens when the dreaded 90% of the machine resources are all used up but we need more CCU for our performance test?<\/p>\n It’s probably time to add another dedicated machine to run more clients<\/strong>. If you don’t have access to more hardware you may consider running\u00a0the whole stress test in the cloud<\/strong>, so that you can choose the size and number of “stress clients” to employ.<\/p>\n The cloud is also convenient as it lets you clone one machine setup onto multiple servers, allowing a quick way for deploying more instances.<\/p>\n In order to choose the proper cloud provider for your tests make sure that they don’t charge you for internal bandwidth costs<\/strong> (i.e. data transfer between private IPs) and have a fast ping time between servers.<\/p>\n We have successfully run many performance\u00a0tests using Jelastic<\/a> and Rackspace\u00a0Cloud<\/a>. The former is economical and convenient\u00a0for medium-size tests, while the latter is great for very large scale tests and also provides physical dedicated servers on demand.<\/p>\n Amazon EC2<\/a> should also work fine for these purposes and there are probably many other valid options as well. You can do\u00a0a quick google\u00a0research, if you want more options.<\/p>\n 1) Login: i<\/strong>n our simple example we have used an anonymous login request and we don’t employ a server side Extension to check the user credentials. Chances are that your system will probably use a database for login and you wish to test how the DB performs with a high traffic.<\/p>\n A simple solution is to pre-populate the user’s database with index-based names such as User-1, User-2 … User-N. This way you can build a simple client side logic that will generate these names with an auto-increment counter and perform the login. Passwords can be handled similarly using the same formula, e.g. Password-1, Password-2… Password-N<\/p>\n TIP:<\/strong> When testing a system with an integrated database always monitor the Queue status under the AdminTool > Dashboard. Slowness with DB transactions will show up in those queues.<\/p>\n 2) Joining Rooms:\u00a0<\/strong>another problem is how to distribute clients to multiple Rooms. Suppose we have a game for 4 players\u00a0and we want to distribute a 1000 clients into Rooms for 4 users. A simple solution is to create this logic on the server side.<\/p>\n The Extension will take a generic “join” request\u00a0and perform a bit of custom logic:<\/p>\n\u00bb Replicating the load<\/h3>\n
\r\npublic class StressTestReplicator\r\n{\r\n\tprivate final List<BaseStressClient> clients;\r\n\tprivate final ScheduledThreadPoolExecutor generator;\r\n\r\n\tprivate String clientClassName;\t\t\/\/ name of the client class\r\n\tprivate int generationSpeed = 250; \t\/\/ interval between each client is connection\r\n\tprivate int totalCCU = 50;\t\t\t\/\/ #\u00a0of CCU\r\n\r\n\tprivate Class<?> clientClass;\r\n\tprivate ScheduledFuture<?> generationTask;\r\n\r\n\tpublic StressTestReplicator(Properties config)\r\n {\r\n\t\tclients = new LinkedList<>();\r\n\t\tgenerator = new ScheduledThreadPoolExecutor(1);\r\n\r\n\t\tclientClassName = config.getProperty("clientClassName");\r\n\r\n\t\ttry { generationSpeed = Integer.parseInt(config.getProperty("generationSpeed")); } catch (NumberFormatException e ) {};\r\n\t\ttry { totalCCU = Integer.parseInt(config.getProperty("totalCCU")); } catch (NumberFormatException e ) {};\r\n\r\n\t\tSystem.out.printf("%s, %s, %s\\n", clientClassName, generationSpeed, totalCCU);\r\n\r\n\t\ttry\r\n\t\t{\r\n\t\t\t\/\/ Load main client class\r\n\t\t\tclientClass = Class.forName(clientClassName);\r\n\r\n\t\t\t\/\/ Prepare generation\r\n\t\t\tgenerationTask = generator.scheduleAtFixedRate(new GeneratorRunner(), 0, generationSpeed, TimeUnit.MILLISECONDS);\r\n\t\t}\r\n\t\tcatch (ClassNotFoundException e)\r\n\t\t{\r\n\t\t\tSystem.out.println("Specified Client class: " + clientClassName + " not found! Quitting.");\r\n\t\t}\r\n }\r\n\r\n\tvoid handleClientDisconnect(BaseStressClient client)\r\n\t{\r\n\t\tsynchronized (clients)\r\n {\r\n\t clients.remove(client);\r\n }\r\n\r\n\t\tif (clients.size() == 0)\r\n\t\t{\r\n\t\t\tSystem.out.println("===== TEST COMPLETE =====");\r\n\t\t\tSystem.exit(0);\r\n\t\t}\r\n\t}\r\n\r\n\tpublic static void main(String[] args) throws Exception\r\n {\r\n\t\tString defaultCfg = args.length > 0 ? args[0] : "config.properties";\r\n\r\n\t\tProperties props = new Properties();\r\n\t\tprops.load(new FileInputStream(defaultCfg));\r\n\r\n\t new StressTestReplicator(props);\r\n }\r\n\r\n\t\/\/=====================================================================\r\n\r\n\tprivate class GeneratorRunner implements Runnable\r\n\t{\r\n\t\t@Override\r\n\t\tpublic void run()\r\n\t\t{\r\n\t\t\ttry\r\n {\r\n\t if (clients.size() < totalCCU)\r\n\t \tstartupNewClient();\r\n\t else\r\n\t \tgenerationTask.cancel(true);\r\n }\r\n catch (Exception e)\r\n {\r\n\t System.out.println("ERROR Generating client: " + e.getMessage());\r\n }\r\n\t\t}\r\n\r\n\t\tprivate void startupNewClient() throws Exception\r\n\t\t{\r\n\t\t\tBaseStressClient client = (BaseStressClient) clientClass.newInstance();\r\n\r\n\t\t\tsynchronized (clients)\r\n {\r\n\t\t\t\tclients.add(client);\r\n }\r\n\r\n\t\t\tclient.setShell(StressTestReplicator.this);\r\n\r\n\t\t\tclient.startUp();\r\n\t\t}\r\n\t}\r\n}\r\n<\/pre>\n\r\nclientClassName=sfs2x.example.stresstest.SimpleChatClient\r\n\r\ngenerationSpeed=500\r\n\r\ntotalCCU=20\r\n<\/pre>\n
\n
\r\npublic abstract class BaseStressClient\r\n{\r\n\tprivate StressTestReplicator shell;\r\n\r\n\tpublic abstract void startUp();\r\n\r\n\tpublic void setShell(StressTestReplicator shell)\r\n\t{\r\n\t\tthis.shell = shell;\r\n\t}\r\n\r\n\tprotected void onShutDown(BaseStressClient client)\r\n\t{\r\n\t\tshell.handleClientDisconnect(client);\r\n\t}\r\n}\r\n<\/pre>\n\u00bb\u00a0Building the client logic<\/h3>\n
\r\npublic class SimpleChatClient extends BaseStressClient\r\n{\r\n\t\/\/ A scheduler for sending messages shared among all client bots.\r\n\tprivate static ScheduledExecutorService sched = new ScheduledThreadPoolExecutor(1);\r\n\tprivate static final int TOT_PUB_MESSAGES = 50;\r\n\r\n\tprivate SmartFox sfs;\r\n\tprivate ConfigData cfg;\r\n\tprivate IEventListener evtListener;\r\n\tprivate ScheduledFuture<?> publicMessageTask;\r\n\tprivate int pubMessageCount = 0;\r\n\r\n\t@Override\r\n\tpublic void startUp()\r\n\t{\r\n\t sfs = new SmartFox();\r\n\t cfg = new ConfigData();\r\n\t evtListener = new SFSEventListener();\r\n\r\n\t cfg.setHost("localhost");\r\n\t cfg.setPort(9933);\r\n\t cfg.setZone("BasicExamples");\r\n\r\n\t sfs.addEventListener(SFSEvent.CONNECTION, evtListener);\r\n\t sfs.addEventListener(SFSEvent.CONNECTION_LOST, evtListener);\r\n\t sfs.addEventListener(SFSEvent.LOGIN, evtListener);\r\n\t sfs.addEventListener(SFSEvent.ROOM_JOIN, evtListener);\r\n\t sfs.addEventListener(SFSEvent.PUBLIC_MESSAGE, evtListener);\r\n\r\n\t sfs.connect(cfg);\r\n\t}\r\n\r\n\tpublic class SFSEventListener implements IEventListener\r\n\t{\r\n\t\t@Override\r\n\t\tpublic void dispatch(BaseEvent evt) throws SFSException\r\n\t\t{\r\n\t\t String type = evt.getType();\r\n\t\t Map<String, Object> params = evt.getArguments();\r\n\r\n\t\t if (type.equals(SFSEvent.CONNECTION))\r\n\t\t {\r\n\t\t \tboolean success = (Boolean) params.get("success");\r\n\r\n\t\t \tif (success)\r\n\t\t \t\tsfs.send(new LoginRequest(""));\r\n\t\t \telse\r\n\t\t \t{\r\n\t\t \t\tSystem.err.println("Connection failed");\r\n\t\t \t\tcleanUp();\r\n\t\t \t}\r\n\t\t }\r\n\r\n\t\t else if (type.equals(SFSEvent.CONNECTION_LOST))\r\n\t\t {\r\n\t\t \tSystem.out.println("Client disconnected. ");\r\n\t\t \tcleanUp();\r\n\t\t }\r\n\r\n\t\t else if (type.equals(SFSEvent.LOGIN))\r\n\t\t {\r\n\t\t \t\/\/ Join room\r\n\t\t \tsfs.send(new JoinRoomRequest("The Lobby"));\r\n\t\t }\r\n\r\n\t\t else if (type.equals(SFSEvent.ROOM_JOIN))\r\n\t\t {\r\n\t\t \tpublicMessageTask = sched.scheduleAtFixedRate(new Runnable()\r\n\t\t\t\t{\r\n\t\t\t\t\t@Override\r\n\t\t\t\t\tpublic void run()\r\n\t\t\t\t\t{\r\n\t\t\t\t\t\tif (pubMessageCount < TOT_PUB_MESSAGES)\r\n\t\t\t\t\t\t{\r\n\t\t\t\t\t\t\tsfs.send(new PublicMessageRequest("Hello, this is a test public message."));\r\n\t\t\t\t\t\t\tpubMessageCount++;\r\n\r\n\t\t\t\t\t\t\tSystem.out.println(sfs.getMySelf().getName() + " --> Message: " + pubMessageCount);\r\n\t\t\t\t\t\t}\r\n\t\t\t\t\t\telse\r\n\t\t\t\t\t\t{\r\n\t\t\t\t\t\t\t\/\/ End of test\r\n\t\t\t\t\t\t\tsfs.disconnect();\r\n\t\t\t\t\t\t}\r\n\r\n\t\t\t\t\t}\r\n\t\t\t\t}, 0, 2, TimeUnit.SECONDS);\r\n\t\t }\r\n\r\n\t\t}\r\n\t}\r\n\r\n\tprivate void cleanUp()\r\n\t{\r\n\t\t\/\/ Remove listeners\r\n \tsfs.removeAllEventListeners();\r\n\r\n \t\/\/ Stop task\r\n \tif (publicMessageTask != null)\r\n\t\t\tpublicMessageTask.cancel(true);\r\n\r\n \t\/\/ Signal end of session to Shell\r\n \tonShutDown(this);\r\n\t}\r\n}\r\n<\/pre>\n\u00bb\u00a0Performance notes<\/h3>\n
\n
\n
\u00bb\u00a0Adding more client machines<\/h3>\n
\u00bb\u00a0Advanced testing<\/h3>\n
\n
\n