Horizontal Scalability with Community Edition

Post here your questions about SFS2X. Here we discuss all server-side matters. For client API questions see the dedicated forums.

Moderators: Lapo, Bax

BonyYousuf
Posts: 120
Joined: 20 Feb 2011, 12:54

Horizontal Scalability with Community Edition

Postby BonyYousuf » 09 Oct 2017, 00:37

Hi,
I want to use SFS2X in my game but apparently it can only scale vertically and not horizontally. Other services like photon can scale to a degree that you cannot even imagine with SFS2X. Right now it has introduced some elastic license which requires manual interaction(not really suitable for instant scaling like photon) and is very costly, whereas I would prefer a perpetual license if it offered horizontal scalability.

SFS2X's approach really just disables all the scalability functionality cloud computing could offer. I wanted a solution where I would spin up as many server instances(EC2) with smartfox running as required as my user grows and turn off whenever there was no need. This is not possible with perpetual license without buying who knows how many licenses beforehand. The only way I see it working is with community license which allows 100 concurrent user. Of course I would give up some functionality such as buddy list, room list etc. But I am fine with that since I would be able to manage that with other services which are scalable (Firebase). With this I could run a server with 100 players in and if more player arrives I would quickly instantiate a new server with another community license for additional 100 player to play my game(My game only has 1vs1 mode, so this shouldn't be a problem) and so on.

So my question is, for my commercial game, am I allowed to run as many community license as I want to each with a different computer(EC2)?
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Horizontal Scalability with Community Edition

Postby Lapo » 09 Oct 2017, 07:52

Hi,
I think you're misunderstanding different aspects of scalability and licensing with SFS2X.

Let's clarify a couple of aspects first:
1) The Community Edition does not necessarily require a commercial license even for a commercial game.
So technically you could spawn as many CE as you want as long as you comply with our EULA, which asks you to add a "Powered by SmartFoxServer" to your client opening screen.

2) Elastic licenses allow you pay for how many CCU you're handling so they are very good fit for scaling up or down your expenditure.
As regards the mentioned "manual interaction", I am not sure why this should represent a problem.
You're only required to adjust the the number of CCU for your license. It's not that you have to do it on an a daily basis.
If you have peaks of 1000 users, you can configure the license to, say, 1200 to give it a bit of headroom and that's it. If your traffic increases you can bump it up accordingly.

From our experience, and that of our customers, the manual interaction required is minimal and the Elastic prices are already configured so that you don't have to micromanage the license to avoid paying big fees.

Have you taken a look at the prices calculator? Prices range between 0,02-0,05 per CCU per month!
http://smartfoxserver.com/buy/sfs2x#p=elastic

I wanted a solution where I would spin up as many server instances(EC2) with smartfox running as required as my user grows and turn off whenever there was no need.

I am not sure what kind of scalability we're talking about here. Can you give me an idea?
SFS2X can run in excess of 100 000 (100K) clients on a single machine. These are AAA game levels of scalability.

You don't necessarily need to run everything on a single machine but at the same time the idea of spawning hundreds of small 100 CCU instances seems like a bad idea. Additionally load balancing multiplayer games is not as straight forward as web sites. Different kind of games require entirely different logic.

Finally you've mentioned Photon as an example of what you're looking for, but that's not even correct. They don't provide that, at least they don't on their website. I have no idea if they have some other secret product that does that.
They offer pre-made packages at set prices. E.g. you pay $185 for 1000 CCU. It's not that you go up and down and just pay by the minute.

If there's any other question or clarifications let us know.
Also you could clarify what is your use case, how your project should work and we'll be glad to help you get started in the right direction. Even if it's not with SmartFoxServer.

Cheers
Lapo
--
gotoAndPlay()
...addicted to flash games
BonyYousuf
Posts: 120
Joined: 20 Feb 2011, 12:54

Re: Horizontal Scalability with Community Edition

Postby BonyYousuf » 09 Oct 2017, 20:45

Hi,
Thank you so much for your response. I am planning to make a board game like chess or say something like ClashRoyale. Most of the game nowadays runs on multiple server which supports horizontal scaling including ClashRoyale, Golf clash and what not, just read the success story of any game on AWS case-study page. The need for horizontal scaling cannot be overstated.

Instant scaling: With horizontal scaling what you theoretically get is instant planet size scaling. With elastic license you would have to be prepare beforehand sometimes that is just not possible. What if a popular youtube streamer plays my game or my game gets reviewed by a popular site or gets featured by apple and it gets 10 times more hit than I have bought elastic license? This is just not feasible. But if your server supports horizontal scaling this is not an issue. More server would be instantiated as user count increases and later it will auto scale down when the user leaves. All of this happens without me worrying about it, and breaking the bank.

Powerful server at all time: With elastic license your server would have to be very powerful if you want to increase your elastic license later on. Otherwise when you plan to increase your limit you manually have to instantiate a new powerful server so it can take the work load. And again when the peak drops you would have to revert back to a less powerful server to decrease the cost. Stoping a game on production to move to a higher/lower end server to save money from server cost is not feasible at all. Again horizontal scaling does not have this issue.

Multi-region With horizontal scaling you can also support your game on multiple region while your main server is in one region which handles matchmaking and user data and regional server handles actual game. This way the ping will be less since player closer to the server would play on that server. For example your match making server could be in US which also handles leaderboard, login etc. If two player from India comes online to play, it will choose a server in Mumbai, india and let them play together with the lowest latency possible for them. If I'm not mistaken, this is how LOL, PUBG, CSGo all works.

Crash If for some reason one of your server crashes it will only affect a limited number of player not everyone. And another server can quickly instantiate to take the load. With smarfox if your server crashes for any reason everyone will be affected and most probably would require a manual restart.

Photon does indeed support true horizontal scaling. You are right when you mentioned 1000 CCU for $180. But this CCU is not in one server. It can be however many server you want(almost). You can preconfiger how many games run on a single server. If the user count increases it will just spawn new server and create a match with new players on that server. and when enough player leaves that server will auto shut down. Exactly how a proper horizontal scaling system works. And all of the current user in all of those server combined needs to be 1000 to exceed that package. This is why Photon, Gamesparks are gaining a lot of popularity with Unity3d and other game engines nowadays.


Now that you understand my need, let me tell you how I plan to solve my issue. I have previously bought a perpetual license with Smartfox. I will user that server as my matchmaking, login purpose. This server will also work for clan-chat support. When player requests for a match, I would select from pre-instantiated small EC2 server which runs SFS2X CE version and send the user on that server. These servers will only run the game nothing else. Based on the user load I will create as many of these EC2 instant required and delete them when they have no need. I know I would have to do a lot of manual work for creating/deleting these servers and keep them talking to the main matchmaking server. But it's far better than not being able to scale properly. With this approach I think I could support 1000CCU for my game just under $50 which is 3 times cheaper than Photon. So with time my hard work will pay off.


Let me know what you think. And one thing. I would have happily paid for more perpetual license than to go this approach. But your license only works on one server, which is a bummer. You guys should change your policy and if someone buys a limitless perpetual license you should just let them run it on as many server as he needs for one particular application/game. That way a true horizontal scaling could have been supported with SFS2X.

Thanks
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Horizontal Scalability with Community Edition

Postby Lapo » 10 Oct 2017, 13:31

Thanks for providing more details about your project.
I have a two part reply for your, one part (this) addresses your main question about licenses while the second one address more in general some of the topics you've raised.

I have previously bought a perpetual license with Smartfox. I will user that server as my matchmaking, login purpose. This server will also work for clan-chat support. When player requests for a match, I would select from pre-instantiated small EC2 server which runs SFS2X CE version and send the user on that server. These servers will only run the game nothing else. Based on the user load I will create as many of these EC2 instant required and delete them when they have no need. I know I would have to do a lot of manual work for creating/deleting these servers and keep them talking to the main matchmaking server. But it's far better than not being able to scale properly. With this approach I think I could support 1000CCU for my game just under $50 which is 3 times cheaper than Photon. So with time my hard work will pay off.

Seems like a valid solution, plus you get the added bonus of using SFS2X features compared to Photon: server side code, a whole lot of security, persistence, fine grained config, buddy lists, etc. Maybe you don't need all of the extras right off the bat, but they're there for any future expansion, imho.

You guys should change your policy and if someone buys a limitless perpetual license you should just let them run it on as many server as he needs for one particular application/game. That way a true horizontal scaling could have been supported with SFS2X.

Yes, I understand what you're saying.
With the per-server license model there is no way to satisfy your request. Instead we'd need to charge by usage regardless of the server count of CCU parameters. As you may imagine this is a completely different model that requires to restructure the whole licensing system and management (for us and our customers).

We're have explored several ways of doing this and the good news is that there will be a similar solution (to what you suggest) available in the future with several key advantages over the competition. Can't say more than that for now, but you'll see it when it's ready :)

Thanks.
p.s. = my second part of the answer will come later as I need to address several things you've said.
Lapo

--

gotoAndPlay()

...addicted to flash games
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Horizontal Scalability with Community Edition

Postby Lapo » 11 Oct 2017, 08:47

Here a couple more comments on what I think are several misconceptions.

Instant scaling: With horizontal scaling what you theoretically get is instant planet size scaling.

This is easier said than done.
You can scale up and down by adding discrete chunks of resources (i.e. cloud servers) but the story doesn't end there.
The scenario you have just described includes a central lobby and a bunch of servers for running the game Rooms, so far so good. This is essentially the use case covered by Photon, which is pretty barebone.

You're not going to tell me that with such a large scale system you won't have requirements for persistence and a database service.
So now you have a new component in the server architecture that "magically" needs to be scaled and doen't become a bottleneck.

What about cross-server communication? Are your users totally isolated from one another? How about friends lists and cross-room chat? That's another not-so-obvious problem to solve that does not magically scale on its own. Servers must be aware of other machines in the region and keep each other updated, which is another potential bottleneck to watch out for.

What about load balancing users?
Games are not as easy to load balance as websites. In fact it can get much more complicated. Every game has its specific requirements in terms of how to distribute players over a list of available servers and how to move them around. Main lobby servers need to query game servers to find available Rooms or perform match making according to the game's own rules. All that kind of cross communication and synchronization doesn't come for free when you run servers in the cloud.

What about server side code?
How do you hot-redeploy server side code on dozens of server across the globe? How do you synchronize it? Maybe along with updates to your distributed and replicated database? Does the cloud automagically solves it for you...?

What I am saying is that you make it sound very simple, as if you could build any server architecture in the cloud and it will manage itself, handling all of the issues above (and more) while you sip a cup of coffe or take a nap :)

The reality is that every game requires it's own server architecture and quite a lot of work to orchestrate and manage such a distributed system spread all over the world. Thanks goodness for Amazon Web Services and similar tools! But it's far from simple than a "set it and forget it".

Powerful server at all time: With elastic license your server would have to be very powerful if you want to increase your elastic license later on.

I hardly think so. This seems another broad generalization.
You can setup a decent medium EC2 instance in AWS and pay it on-demand (i.e. pay for actual used resources)
When the traffic grows the machine will simply put to good use the resource that lie dormant and will be able to handle 1000s of CCU without breaking a sweat.

Also while "horizontal scaling" is the new buzzword, scalability is not exclusively accomplished by throwing more servers at the problem but also by increasing the resources of existing servers. (i.e. vertical scaling).
In fact throwing more servers at the problem can even make things worse by causing bottlenecks in cross-server communication and fragmenting the players excessively.

Photon does indeed support true horizontal scaling.

Yes, for a single, very specific use case though :)
The case where you have a lobby/matchmaking entry point and a bunch of isolated rooms. Period.
No datastore services, no persistence, no server side coding, no cross room communication...

If you need any of the above you will have to roll out your own solutions or add other services, make sure they scale adequately, integrate them manually etc...

We have many customers running medium to large scale projects in the cloud and we have never seen major issues related to licensing and scaling while working with them. In fact many of them run several medium/large instances that are paid on the demand and can scale up to the hundreds of thousands, when needed. So there is no particular drama with the "what if the world goes crazy for my game". (which is always a good problem to have, btw)

Cheers
Lapo

--

gotoAndPlay()

...addicted to flash games
BonyYousuf
Posts: 120
Joined: 20 Feb 2011, 12:54

Re: Horizontal Scalability with Community Edition

Postby BonyYousuf » 11 Oct 2017, 09:30

I am well aware of the hard work that will be put into making such a system. All I am saying that it is worth it if you truly require horizontal scaling. Here are a bit more information on how I think will solve some of the issues.

Whenever a new game server instantiates it will send an update to the match making server reporting it's user count, in fact it will continue to do so every 5 minutes. If a server is not heard from within 5-10 min matchmaking server will consider it dead and will fire a shut down command for that server after 30 minutes. (None of my game/match will lasts for that long).
And whenever a new user will request for a match the matchmaking server will check to see which server has least amount of people and it will send the user to that server with command to create new game room for him. If most of the server has say 70% users out of it's max user limit. I will fire a command to instantiate a new server. If the user count is less than 30% then I will stop sending people to one particular server and when the user in that server reaches zero, that server will shut down.
I think this will be enough for load balancing.

What about cross-server communication? Are your users totally isolated from one another? How about friends lists and cross-room chat? That's another not-so-obvious problem to solve that does not magically scale on its own. Servers must be aware of other machines in the region and keep each other updated, which is another potential bottleneck to watch

This is where Google's Firestore database comes in, or I can just use DynamoDB both of which solves my problem for a scaling database. I don't need cross room chat and clan chat is only available from the matchmaking server. So that won't be an issue. Infact I can use Google's Firebase database for chatting as well.


What about server side code?
How do you hot-redeploy server side code on dozens of server across the globe? How do you synchronize it? Maybe along with updates to your distributed and replicated database? Does the cloud automagically solves it for you...?


I won't be able to do hot-redeploy and thats fine. I will have to set a time when server will be shut down for maintenance. This is how almost every other game works.

The reality is that every game requires it's own server architecture and quite a lot of work to orchestrate and manage such a distributed system spread all over the world. Thanks goodness for Amazon Web Services and similar services! But it's far from simple than a "set it and forget it".


True.


Photon does indeed support true horizontal scaling.

Yes, for a single, very specific use case though :)

Yes, and this is why photon is often paired with PlayFab which solves all the problem you have mentioned regarding database and everything. And Gamesparks is like a combination of photon and playfab. But photon and playfab is a bit costly for me. So I am trying to build a system with SFS2X which I am familiar with.


Powerful server at all time: With elastic license your server would have to be very powerful if you want to increase your elastic license later on.

I hardly think so. This seems another broad generalization.
You can setup a decent medium EC2 instance in AWS and pay it on-demand (i.e. pay for actual used resources)
When the traffic grows the machine will simply put to good use the resource that lie dormant and will be able to handle 1000s of CCU without breaking a sweat.


I'm not sure if I follow. Say I setup a decent medium sized EC2 instance prepared for max load. If I don't get any spike in my traffic, I am still paying the full cost of that EC2 instance. So if I have to go to a less powerful machine to save cost since I dont have much user, I would manually have to shut down that server and instantiate a cheaper one. And once you have cheaper server, you are not prepared for spike anymore.

I understand that this won't be a big issue with big company where they could run a moderate server even if there is not much user. Or if your game runs for quite sometimes you can already have some idea about your user base and run a server based on that info. But this is not possible for new games and specially for an indie.

Let me know if i'm mistaken somewhere.

Thanks
User avatar
Lapo
Site Admin
Posts: 23008
Joined: 21 Mar 2005, 09:50
Location: Italy

Re: Horizontal Scalability with Community Edition

Postby Lapo » 12 Oct 2017, 07:49

Just to be clear, my previous response was not a critique of the solution you're looking for your project. Far from it.
I was just pointing out the inherent issues in building those kind of systems and how different each architecture can be based on the game requirements.

Other than that I think the improvements on the SFS side you have asked are very reasonable and, as I mentioned, we're working on a solution that will answer to those kind of requisites.

Cheers
Lapo

--

gotoAndPlay()

...addicted to flash games

Return to “SFS2X Questions”

Who is online

Users browsing this forum: No registered users and 52 guests