Hi there, I'm having some issues with server communication happening in the wrong order.
Basically, I send an event when I land on a special platform in my game and another event when I jump off that platform. When I jump as soon as I land, a bug happens where I get the events back from the server in the wrong order. Here's what happens on the client.
// Landed on platform at frame 21209
SendSetOnKingPlatform true 21209
// Jump off platform one frame later at frame 21210
SendSetOnKingPlatform false 21210
The server receives this as I print it in the logs with the system time (millisecond) it was received
{KOTMTest}: OnSetOnKingPlatform false 1430146784443
{KOTMTest}: OnSetOnKingPlatform true 1430146784443
So seems like both events arrive at the same time but in the wrong order.
And so the server sends back the events to the players in the wrong order as well and the clients receives the 2 events at the same frame 21211
HandleSetOnKingPlatform false 21211
HandleSetOnKingPlatform true 21211
Is that normal? Is that a common problem, what's the best way to deal with it? I think that's the cause of many other glitch I'm having
Thank you
Server reponse received in the wrong order
Re: Server reponse received in the wrong order
Yes, it is normal.
That is the essence of asynchronous programming.
It sounds like the two events, landing on the platform and jumping off, are happening very close in time on the client side. What happens is that the TCP stack on the client side aggregates the two requests into a single packet and that's why the server receives them at the same exact time.
If you need to make absolutely sure that the two requests are processed in order you will need to synchronize the threads on the server side, using a common lock.
Is this something you're familiar with?
cheers
That is the essence of asynchronous programming.
It sounds like the two events, landing on the platform and jumping off, are happening very close in time on the client side. What happens is that the TCP stack on the client side aggregates the two requests into a single packet and that's why the server receives them at the same exact time.
If you need to make absolutely sure that the two requests are processed in order you will need to synchronize the threads on the server side, using a common lock.
Is this something you're familiar with?
cheers
Re: Server reponse received in the wrong order
Sadly I'm not familiar with that Can you elaborate or point to resources about the technique?
Re: Server reponse received in the wrong order
Ok I remember I've learned that at the University and it's now in the far and obscure recesses of my brain but how do I apply that to fix the problem I'm having? I *think* I understand the concepts but not how that applies to SFS handling my requests in the wrong order. Can you elaborate with where you're leading me with this? Is this the same issue and solution?
viewtopic.php?f=18&t=17721&p=76957&hilit=lock#p76957
Thanks a bunch
viewtopic.php?f=18&t=17721&p=76957&hilit=lock#p76957
Thanks a bunch
Re: Server reponse received in the wrong order
If you want to take a different, and possibly simpler, perspective on the problem you could try to set the Extension ThreadPool to 1.
This means that all requests are processed by a single thread avoiding parallel processing enirely. Since all messages are processed exactly in the order in which they arrive you should never see the problem happening.
This of course has side effects:
1- it doesn't allow Extensions to scale taking advantage of multiple cores
2- it can cause performance issues if your Extension calls take a long time to execute (e.g. database calls)
You could solve all of the above by creating your own Extension ThreadPool and delegating to it only the slow invocations such as database/file related calls, if any. Maybe also very computational-intense calls, such as pathfinding?
This would entirely avoid the problem of getting into the trouble of manually synchronizing threads for your use case. Come to think of it, it seems a better idea.
This means that all requests are processed by a single thread avoiding parallel processing enirely. Since all messages are processed exactly in the order in which they arrive you should never see the problem happening.
This of course has side effects:
1- it doesn't allow Extensions to scale taking advantage of multiple cores
2- it can cause performance issues if your Extension calls take a long time to execute (e.g. database calls)
You could solve all of the above by creating your own Extension ThreadPool and delegating to it only the slow invocations such as database/file related calls, if any. Maybe also very computational-intense calls, such as pathfinding?
This would entirely avoid the problem of getting into the trouble of manually synchronizing threads for your use case. Come to think of it, it seems a better idea.
Re: Server reponse received in the wrong order
This is a bit out of my comfort zone but looking around to figure this out, I see a lot of people saying that their threadpool is overloaded, and causing lag, so setting the extension's threadpool size to 1 seems like a bad idea doesn't it?
Seems like any message in the wrong order can break your game state, how do huge game handle that? If sending the current weapon or setting an object state can be in the wrong order, that complicate things a lot no?
Seems like any message in the wrong order can break your game state, how do huge game handle that? If sending the current weapon or setting an object state can be in the wrong order, that complicate things a lot no?
Re: Server reponse received in the wrong order
Let me just clarify one point before I answer your questions.
The client requests always arrive at the server in right order if your use TCP (the default protocol). With UDP you have no guarantee for order and delivery, but I suppose we're talking about TCP here.
What happens is that they get handled by multiple threads. These threads start their work almost simultaneously but they will likely finish in different orders, because they run different code. Also if they access the same data, they will run into race conditions or make the server state logically wrong, because their parallel activity cannot be predicted deterministically.
Overloading is typically due to slow tasks, as I have mentioned. 99% of the cases it's the DB's fault.
What I suggest is to discriminate in your Extension code between code expected to be slow (e.g. database calls) vs anything else and delegate the former to a dedicated ThreadPool (which is no big deal, one line of code, just to be clear)
There can be many ways:
a) their game code doesn't break if messages come in wrong order, by design
b) they deal with the wrong order (it can be very game specific)
c) they design the game to avoid the problem entirely... (variation on a)
The bottom line is that you will need to experiment either with thread synchronization or with a mono-thread + thread pool, and see which works best for you.
The client requests always arrive at the server in right order if your use TCP (the default protocol). With UDP you have no guarantee for order and delivery, but I suppose we're talking about TCP here.
What happens is that they get handled by multiple threads. These threads start their work almost simultaneously but they will likely finish in different orders, because they run different code. Also if they access the same data, they will run into race conditions or make the server state logically wrong, because their parallel activity cannot be predicted deterministically.
Evil-Dog wrote:This is a bit out of my comfort zone but looking around to figure this out, I see a lot of people saying that their threadpool is overloaded, and causing lag, so setting the extension's threadpool size to 1 seems like a bad idea doesn't it?
Overloading is typically due to slow tasks, as I have mentioned. 99% of the cases it's the DB's fault.
What I suggest is to discriminate in your Extension code between code expected to be slow (e.g. database calls) vs anything else and delegate the former to a dedicated ThreadPool (which is no big deal, one line of code, just to be clear)
Seems like any message in the wrong order can break your game state, how do huge game handle that?
There can be many ways:
a) their game code doesn't break if messages come in wrong order, by design
b) they deal with the wrong order (it can be very game specific)
c) they design the game to avoid the problem entirely... (variation on a)
The bottom line is that you will need to experiment either with thread synchronization or with a mono-thread + thread pool, and see which works best for you.
Re: Server reponse received in the wrong order
Thanks for the clarification, it makes sense, threads finishing in different orders.
That solution seems like the most logical. So I know it's not your job to teach me java or how to work with threads but are we talking about like using Executors.newSingleThreadExecutor() for the extension and Executors.newFixedThreadPool(SomeAmountOfThreads) to handle longer tasks? Or am I in the wrong direction?
I read in another SFS forum thread about creating a new runnable task to execute the longer tasks (DB access, HTTP request) in the manually created threadpool, but how does the single thread apply to the extension? That's something I haven't came across in the docs or in the forums. Maybe I'm missing some key notions about threads.
Thanks for the help
What I suggest is to discriminate in your Extension code between code expected to be slow (e.g. database calls) vs anything else and delegate the former to a dedicated ThreadPool (which is no big deal, one line of code, just to be clear)
That solution seems like the most logical. So I know it's not your job to teach me java or how to work with threads but are we talking about like using Executors.newSingleThreadExecutor() for the extension and Executors.newFixedThreadPool(SomeAmountOfThreads) to handle longer tasks? Or am I in the wrong direction?
I read in another SFS forum thread about creating a new runnable task to execute the longer tasks (DB access, HTTP request) in the manually created threadpool, but how does the single thread apply to the extension? That's something I haven't came across in the docs or in the forums. Maybe I'm missing some key notions about threads.
Thanks for the help
Re: Server reponse received in the wrong order
Evil-Dog wrote:Thanks for the clarification, it makes sense, threads finishing in different orders.
That solution seems like the most logical. So I know it's not your job to teach me java or how to work with threads but are we talking about like using Executors.newSingleThreadExecutor() for the extension
No the Extension thread is out of your control. You just need to configure the Extension Thread count to 1, from the AdminTool. This way all requests are executed serially.
and Executors.newFixedThreadPool(SomeAmountOfThreads) to handle longer tasks? Or am I in the wrong direction?
I would probably use and unbounded thread pool so that you can set it and forget it --> Executors.newCachedThreadPool()
Here's how it should work in simple steps:
1- you receive client requests in your code
2- you can determine whether the request should be executed by your main thread (the one you're already in) or the separate thread pool (for slow/other processes)
3- if the request should be delegated you simply hand the operation to the pool (e.g. t_pool.execute(...));
4- otherwise just execute your code.
Makes sense?
Re: Server reponse received in the wrong order
So I was trying to find an extension setting but are you talking about the global server configurator where it says Extension Thread Pool? If so, would I set Core threads to 1 and Backup threads and Maximum backups to 0? or I don't touch the backup values?
Can you only do it globally for all extensions/zones/games on the server?
Regarding newCachedThreadPool(), I thought it would be newFixedThreadPool since the doc says newCachedThreadPool() is for frequent quick tasks, but I guess it's not like DB requests will stack like crazy. Again, maybe I'm missing some important notions.
So in your workflow t_pool is the ExecutorService returned by newCachedThreadPool and I'd call like
t_pool.submit(new Runnable() {
public void run()
{
// My heavier code here.
}});
Is that right?
Can you only do it globally for all extensions/zones/games on the server?
Regarding newCachedThreadPool(), I thought it would be newFixedThreadPool since the doc says newCachedThreadPool() is for frequent quick tasks, but I guess it's not like DB requests will stack like crazy. Again, maybe I'm missing some important notions.
So in your workflow t_pool is the ExecutorService returned by newCachedThreadPool and I'd call like
t_pool.submit(new Runnable() {
public void run()
{
// My heavier code here.
}});
Is that right?
Re: Server reponse received in the wrong order
Evil-Dog wrote:So I was trying to find an extension setting but are you talking about the global server configurator where it says Extension Thread Pool? If so, would I set Core threads to 1 and Backup threads and Maximum backups to 0? or I don't touch the backup values?
Can you only do it globally for all extensions/zones/games on the server?
Yes it is a global setting:
coreThreads = 1
backupThreads = 0
maxBackups = 0
Regarding newCachedThreadPool(), I thought it would be newFixedThreadPool since the doc says newCachedThreadPool() is for frequent quick tasks, but I guess it's not like DB requests will stack like crazy. Again, maybe I'm missing some important notions.
The cached thread will remove all headaches of re-sizing and fine tuning. I suggest this option because it's "set and forget". It will get the job done. You may replace it with something else when you're more familiar with the whole concurrency business in Java, if necessary.
So in your workflow t_pool is the ExecutorService returned by newCachedThreadPool and I'd call like
t_pool.submit(new Runnable() {
public void run()
{
// My heavier code here.
}});
Is that right?
Correct.
Cheers
Re: Server reponse received in the wrong order
Excellent, I'll try that and see how it goes
Thanks for the help!
Thanks for the help!
Re: Server reponse received in the wrong order
I'm unable to set the backup thread values to 0, it just resets to 1. Is that normal? It seems to fix my issue as far as my testing goes but yeah, can't set these values to 0 like you told me to.
Re: Server reponse received in the wrong order
You mean in the AdminTool?
If so, you can edit the value directly in config/server.xml (bottom of the file)
I will add a note to our bug system to check the setting in the Admin.
Thanks
If so, you can edit the value directly in config/server.xml (bottom of the file)
I will add a note to our bug system to check the setting in the Admin.
Thanks
Return to “SFS2X ActionScript 3 API”
Who is online
Users browsing this forum: No registered users and 35 guests