Hello Everyone,
I was playing at the no-explosives metro server last night (as always), during my session, the server lag got really bad that other players started commenting about it. Here are my thoughts on why this lag happens, and why it happens at a specific time.
Note that i am assuming that we are running the game server from one of the cloud provider services (like rackspace).
I am pretty certain that the BF3 game server isnt running on a dedicated server in Rackspace and is sharing the server with multiple virtual machines. Virtual machines are copies of multiple Windows Server instances running on a single physical server (see http://en.wikipedia.org/wiki/Virtual_machine for more info).
Since BF3 doesnt run exclusively on the server, the performance of the virtual machine it sits on is relatively affected by processing occuring on other virtual machines. This means that if one virtual machine instance is running a processor/memory intensive application with a high priority, this in turn triggers other virtual machines to throttle down their use of the processor/memory.
Which in turn causes the rage you see in chat sometimes.
Whenever the virtual machine is given the throttle down command, it pages out whatever it had in memory to the hard drive, and works on a smaller memory footprint (for example, during non-peak times, the virtual machine might have access to 4gb of ram and 50% cpu; but at 8am, the virtual machine host runs a backup routine, or is running an adminstrative app; suddenly our BF3 virtual machine throttle's down to 2gb available physical ram and 2gb paged ram and 25% cpu). Do note that paged ram is significantly slower than having the app use physical ram.
So as you can see, the performance of the server (bf3 in this case) is indirectly dependent on the environment it sits on.
To backup my claim, we have a huge client at work (a web 2.0 site with more than 1000 unique page impressions per day) and its servers are situated in the cloud (a cloud service provider similar to Rackspace). During certain parts of the day, performance degraded to the point where we are no longer able to service more than half of our normal visitors.
After much hair-pulling, distress, finger-pointing and corporate b.s., the infrastructure team was able to determine that the cause of all of this "lag" in performance was due to the fact that another company (not our client) was generating a very complex report on their virtual machine server every morning which negatively affected performance on ours and everyone else on the server.
We were able to resolve this issue by purchasing collocated servers and having the cloud provider manage it. Since we had full access to the server, the lag disappered and everyone was happy.
Just my 0.02 cents, please feel free to disprove my claim or maybe add to it too :)