Stress testing internet stuff

[Phil Steinmeyer]

My game has a global high score system that accesses the 'net.

I'd like to do a bit of a stress test against it - simulating N requests per minute against the system. I need to determine what value N is, but it'll probably be in the dozens/hundreds - likely more than I can simulate by sitting at my PC and pressing the refresh button on my browser really fast.

Is there a cheap 'n easy way for a lone-wolf developer to simulate dozens of PCs hitting a given web page in a short period of time? I could issue lots of requests programmatically, but I'm not sure if that's a valid simulation, as they might queue up in some way that is not the same as requests coming from multiple PCs/IPs.

[Phil Steinmeyer]

Hmm, by my calculations, if my game goes wide (to all the portals) and does reasonably well, and most (but not all) users use the high score system some, then my PEAK #s (first month or two, and assuming that the peak hour of the day accounts for about 15% of daily volume) is:

(Transactions per minute)

High score inserts: 78
High score selects (i.e. requests to View the high score list): 127

That seems kinda scary high, both from an overall internet traffic number (bandwidth), and from a mySQL angle (too many requests).

Does anybody have any real data on this kind of thing for a game that had global high scores and was on all the portals?

I made conservative (i.e. high) estiamates at most junctures (number of downloads, how long each downloader played the game, how much they used the high score stuff), and it's conceivably I'm off by a significant multiple...

[Ryan Clark]

If you think you're going to get max 120ish requests per minute, then you just need to make sure your server can handle 2 requests per second.

It should be able to... webservers can handle more than 2 requests per second, afterall.

[Phil Steinmeyer]

Yeah, but I'm on a shared server - don't want to annoy the hosting site too much.

[Ryan Clark]

Then I would suggest you do what we did: Set up your high scores on a different domain name. That way, if it proves to be too much load for the server, you can move it to a separate server.

[soniCron]

Phil, I know you're using Lunarpages, and they support virtual domains. You can buy a cheap domain (NewCrayonCommunity.com) and set it up in your control panel under "Addon Domains." It'll register the name with the server and give you the DNS addresses to use with your registrar. If you ever need to upgrade, just change the DNS settings with your registrar and you're set. :)

[Phil Steinmeyer]

Yeah, I've already got it setup that way (separate domain).

[Matthew]

You can use ApacheBench ("ab") to do stress testing:

http://httpd.apache.org/docs/1.3/programs/ab.html

There are probably some step-by-step tutorials on stress testing servers out there, too. 2 queries/second shouldn't be a problem, even on shared hosting (most shared hosting providers are set up with seperate MySQL boxes). I've worked sysadmin on websites that run 100+ queries/second on very normal hardware.

If you're doing anything dynamic with the high scores (today/month/alltime), just make sure you put indexes on any columns you're referencing. A MySQL EXPLAIN statement (http://dev.mysql.com/doc/refman/5.0/en/explain.html) will show you which keys the DB is using for any particular query.

[Phil Steinmeyer]

Yeah, my back of the envelope calculations show a rough peak of 3 calls to my high score code per second, each of which generate 2 queries (so 6 calls all total).

Currently, I'm returning 100 rows for my high score list, but I can cut that back on the server side if necessary...

[Backov]

Phil, don't worry - that is incredibly low load for a good mysql setup. Some of my MySQL setups do hundreds of queries a second.

If it's PHP/HTML, also no issue with proper apache config.

[soniCron]

It goes without saying, but I'll say it just to be safe:

With a shared server, the other websites will count against the "hundreds of queries per second." That's really likely not to be a problem, but if you wanna see for yourself, check the domain names that lead to your server (http://www.seologs.com/ip-domains.html?domainname=newcrayongames.com&primaryAction=submit).

[Drake]

What are the advantages of using a separate domain for the DB server, as opposed to a subdomain of your primary one?

[soniCron]

All DNS requests go to the main domain's DNS server and most won't forward the subdomain to another server, or at least outside the host's array. By using a separate domain, you're guaranteeing flexibility over where you handle the data. If you need it to utilize more bandwidth or more CPU, you can. It's also forward thinking: These games will traverse the Internet for all time. By offloading the requests, you can prevent unneccessary strain in the future if you decide to stop supporting previous games. Why hit your main site with a bunch of dead requests?

[Drake]

All DNS requests go to the main domain's DNS server and most won't forward the subdomain to another server, or at least outside the host's array.Huh. Can't a CNAME record point to anything, though? I thought it was just a matter of an extra DNS lookup for the CNAME versus an A record.

[soniCron]

My appologies if I'm incorrect - I'm just regurgitating what I've been told in the past. That said, I'm not speaking of technical limitations of the DNS paradigm - I'm talking about the limitations of control registrars and hosts extend to their users. Of course, I've only used shared hosting for the last 10+ years, so I don't know what kind of setup dedicated servers provide. (Do they handle their own DNS requests? etc.)

[Backov]

A subdomain can point to any box you like at any IP.

So yes, you can have the DB server seperate from the main domain on a sub like db.example.com.

As for the mysql load thing again - don't use shared hosting. Colocation is cheap and avoids lots and lots of problems.

[Phil Steinmeyer]

So, after playing with the mySQL stuff for a while, I discovered the following:

At my projected peak of ~3 calls (with 2 queries each) per second, there's no real problem, because they only take about 10-30 ms each, and my host allows me to have as many as 50 simultaneous mySQL connections.

But I did discover another problem. I had been planning to keep all submitted scores indefinitely, so that a user could see where there submitted score placed, all-time (i.e. 5713 all time). However, I discovered that when I expanded my table to 500,000 rows, that performance fell off steeply - to ~600 ms for the same call that took 10-30 ms with a nearly empty table (1000 rows, originally).

So I had to change my plans - you still see where your score is for the day, and you see if your score makes the top 100 for the last month or all time, but I now run a cron job that deletes all rows not in the top 100 and older than a day. This will keep the table fairly small and fast.

Anyways, thought I'd pass the info on...

[jankoM]

Just a observation... you say you would like to store all score but 500.000 rows makes DB slow so you will be limiting high score to 100?? I think you are going to the other extreme now with only 100 rows..Holding last 1000, 2000, or 5000 scores is still 100x smaller and probably a lot faster than 500.000 for example..

[Indiepath]

Sounds like you need some serious query and table optimisation - 500K records should not be a problem at all. Have you indexed the tables? are you using commands like ON DUPLICATE KEY to prevent duplication?

[Phil Steinmeyer]

Yes, I played with indexes and optimized things some. The larger the table, the more data that must be scanned. Indexes help, but do not fully resolve. There are some ninja tricks I could use to go further in this regard, but, IMO, they weren't worth the development/testing/maintenance trouble.

Basically, I'm keeping ALL scores from the last 24 hours, and for older scores, only the top 100 scores in each game type for each relevant period (monthly, all-time).

Since the game only shows the top 100 per category anyways, that's all that's really needed. The extra scores were to show your particular scores place in the charts, and you can still see that for the dailies (no matter how bad your score), or for the monthly/all-time if you place in the top 100.

[Huge]

Is there some way to cache the results? Presumably, the results only change when you add an entry, which happens less frequently than requests to view? Does apache offer some help here - one day I might have a game that has a high score table that people are actually interested in, so I would like to know too.