I noticed further down that the comments system seems to be somewhat of a problem, with its eating of resources (appologies if i got the wrong end of the stick) have you considered using vBulletin forum software and the Wordpress->vBulletin bridge for comments, this basically uses vBulletin to handle the comments instead of Wordpress. vBulletin is far more capable of handling huge amounts of data and what not, only problem is it costs quite a bit. Just thought i’d make the suggestion as I like this site and wouldnt want anything to happen to it .

Good luck with this, Nick. I wish I had some useful advice to give you on this. Your site is a daily destination for me, and it is rare to see a writer so mindful of the quality of his site.

Well, I havent looked into vBulletin. That would basically be like sending comments over to the forums. Which, if necessary, could be done although it would certainly be less convient.

But you are right. While working on the site the last few days I realized that the comments are a huge part of this site. In terms of megs of data, they comprise more than half of everything on the site.

Which shows just how important comments are. They are a huge part of the site and one that really helps the sense of community.

Anyway, Im up for looking at all options. Thanks for the headsup.

I’m not too familiar with either Wordpress nor Drupal, but I’ll look into the matter and see if I can find solutions to alleviate the load on the site.

downtime sucks :p

I’m not sure about the specifics of your WordPress setup, but moving to static pages instead pulling your content from the database will help tremendously.

Again I’m not sure how WordPress works, but I know Blogger publishes static pages (not that I’m suggesting that you move to Blogger). Movable Type gives you the option too. Wordpress may have the option as well.

Basically, the hard work is done when you publish a post. The app writes a static HTML file to your web server rather than directing the page to look into a database. Some portions will still have to come out of a database (comments, etc.), but there will be much less of a load on the database, and if it goes down the static content will still be there. Of course, all the content still exists in the database, and you will have to republish the file whenever you make changes to a story.

Anyway, that’s a dumbed down simplification, but it might get you in the right direction. Or you may already be doing static pages, and it could be no help at all.

Yeah, Im already running static pages. The site runs static unless a new entry is posted or a new comment is posted.

Are the forums and the blog on the same server? I’d think the forums are the ones doing it.

Otherwise, if it’s actually the main page and comments system (which would really surprise me) then probably the move to a more static format like xml would be efficient, since it doesn’t have to update unless something changes, and there are no more database requests (if it’s done the right way…)

I’m not really familiar with cafepress or other blogs to know however… maybe find out what engadget/gizmodo are running off of?

Can you be more specific about the database errors? Too many calls to the database at the same time?

I’m not very experienced with PHP (the langauge behind Wordpress), but it should natively handle connection management without issue (unless you are serving IGN-type bandwidth). I’ve run Wordpress before without problems, so I can’t help to think you or your host has PHP or Wordpress misconfigured. I’ll search the Wordpress forums for something…

I have to learn how to read ALL the comment entries before I reply…

Well, the forums are on the same hosting plan, but not on the same database. If youve ever checked, the forums never go down (gonna jinx myself). Probably because they arent getting as many users, and possibly because phpBB handles the load better.

This most recent error was that there were too many calls to the database. Honestly, now that I dig a little deeper I think I may have screwed something up the last time the site went down. I may not have re-enabled the caching mechanism correctly. It seemed to be working as usual though. But, Ive had many more users on the site than I did today and it hasnt gone down since I started using the caching plugin.

So, maybe this time it was a fluke…but still, Id rather be safe and plan for the future. I mean, I in no way get IGN or even Joystiq/Kotaku type numbers. The amount of readers does increase every month though.

Thats why Im thinking that either Im doing something wrong or that wordpress just isnt built to handle this. Although wordpress.org and other big Wordpress sites seem to be able to handle large amounts of traffic.

I’m sorry to say this, because I know it holds a sentimental value- but I know why there is downtime, and I know what must be done.

At the top of 4CR there has been an epic battle between the forces of The Iguana and Link. Surely, because of the size and sheer length of this campaign (how many months, now?), our window into the unholy war has stolen much of the bandwidth as we all sit and stare at the brutal nature of combat, waiting for the next blow.

You must end this aggression at once.

…or it could be wordpress.

Lol, nope, its not a bandwidth issue. That lovely image of Yoshi battling link causes so concern for the site.

have a look at some of these
http://cgi.resourceindex.com/Programs_and_Scripts/Perl/News_ Posting/

Is there a reporting tool that will show you how many of your database requests come from what scripts? If everything is also done with static pages there must be a page or two slipping through to push you to your maxium number of queries.

At the bottom of this screen I notice it says 20 “pikmin”/queries were used to create this page. Are you sure it’s static? Even so, 20 is quite a few queries for a page like this.

You could grab an older version of Movable Type - the old ones that make everything static HTML files, so database queries are reduced to basically nothing…

Shit, man- I could have sworn that was a video stream; I’ve been sitting here with popcorn watching intently for days now.

If you go without sleep long enough, Link TALKS to you.

Congratulations on all the traffic, though; that’s a good thing. I’ll bet most of the gamers clicking on the Yahoo link aren’t a finiky bunch; once they read a few of your articles they’ll get hooked like the rest of us.

You folks write very much well enough for IGN, The Escapist and the like; so have faith, man. We’ll still read ya!

Newer versions of Moveable type are still static by default Johan, but you now have the option of making dynamically generated pages.

What specifically is the error message Nick and what db are you using.. MySQL presumably?

Yeah, using MySQL. The error that I got this time was a “max_questions” error that said I had exceeded my 5000 queries an hour limit.

I’m not familiar with Wordpress unfortunately (just MT really), but I did notice at the bottom of your page this comment:

Are you expecting your index to be served dynamically?

bugger sorry (couldn’t post with the brackets): Dynamic Page Served (once) in 0.468 seconds

Hmm, Ive never noticed that before…

If you are exceeding 5000 queries an hour, I suspect you are serving a lot of dynamic pages. If everything is static there shouldn’t be any calls to the database, other than when you post a new item, and in that case it’s an INSERT, rather than a SELECT if that makes sense… i.e. a user hitting your site should never initiate a database connection if it’s all static html.

Another thing I am noticing is, every other refresh of your site appears to be cached. Refresh your site, and have a look at your source (at the bottom). It looks like wp-cache is only serving cached files occasionally (when it does, there’s a comment indicating that the page has been served from the cache).

Try this thread on the wordpress site:
http://wordpress.org/support/topic/39167

Basically, your ISP needs to up the max on their end. 5000 is low, IMO.

Fammy 5000 isn’t low if you are expecting your pages to be served statically (which any high traffic website must).. that’s a huge amount of database traffic.

5000 connections in an hour? Let’s assume that one page view requires one database connection. That’s only 83 and change a minute. If 4cr averages more than 83 a minute for that hour, screwed.

I’m sure the ISP would be reasonable and raise the limit, even 10,000 would help. If I had a data driven website (I’m not arguing against caching techniques), I’d expect it to be able to handle more than 83 pages a minute. At work, we test for a lot more than that.

Yeah, like KIT says, I belive that the 5000 is adequate.

The fact that wp-cache might only be serving static pages occasionaly is troubling. In fact, WP-Cache isnt supposed to allow any dynamic content to run. Thats why Ive had to use a google search for this site…because I couldnt get the traditional/dynamic search to work.

The only time the site should update that cache is if a new post is made, or a new comment is made. Or if Im fiddling around behind the scenes.

I might have to try disabling comments for a bit and checking it out.

Fammy - page views shouldn’t require ANY database connections if the site is being served statically.

Well FAMMY, thats the thing. A single page view shouldnt require a database connection at all, if they are static pages. Which they should be. Now, if there is a lot of commenting going on, then everytime a new comment is placed then a call to the database must be made to account for the changes and thus start the caching cycle over again.

Well (besides wasting 1 of your 5000 DB calls per hour), if 4cr was doing more dynamic stuff (or you had a website that required dynamic content), I’d be very unhappy with 5000 DB connections per hour. That’s not much in the real world. With news, you can make it static, so 4cr lucks out… sorta.

It appears that Wordpress doesn’t use the PHP/MySQL connection pooling (leave the conneciton open for a while in case someone else needs it). Maybe the newest version does or allows the option. If it did, you may be able to squeeze hundreds of DB calls through the same connection. There are ways around it, but your ISP isn’t making it any easier.

Also, not having a Captcha or other bot comment defeating scheme would make it easy for someone to bring commenting offline for that hour.

Sorry, not too good with this kind of stuff. Infact probably won’t be able to help you with any computer problems.
Anyways I hope you figure it out with the help of the other readers. Happy Hunting for the answer!!

Yes, Im looking at WP-Hashcash right now as a spam deterent. It works with WP-Cache.

http://elliottback.com/wp/archives/2005/05/11/wordpress-hash cash-20/

Although the builit in Wordpress blacklist might be sufficient. Ill see…

If you’re unable to resolve your troubles with WP-cache Nick, I can recommend MT. It isn’t free however, and won’t leave you with that refreshing ‘I’m supporting Open Source’ tingle.

If by chance you would want a mirror, I would be more then happy to provide you with a free mirror. I work over at Godaddy.com and since i’m such a frequent visitor I would love to “hook you up”. Send me an email….then we’ll talk

Instead of coments it could be forum comments

I visit another popular wordpress blog http://google.blognewschannel.com

They have the exact same problem. Actually, when I saw the error message come up on 4cr, I recognized it from this other site. Perhaps it’s just a problem with WordPress? I’d try moving to another platform, especially something like MT, not that I’ve ever used it.

I use Serendipity for my blog and the hosting is the best I’ve been with in a LONG time. HostGator = your friend and if you want to try them out for a month I have a code for a free month of hosting!

Gack, i should have followed up my initial post sooner, gotta be honest havent read all the other comments (just a few, its almost 1am here, im tired) but wanted to follow up, so sorry if what im about to say has been covered.

Yeah, the vBulletin thing, it would indeed mean making comments a forum thing, HOWEVER you can be rather smart ass with it if you please due to vBulletins “conditionals” (think thats what they are called) which are basically “” statements, and with some creative html and a little bit of php you could effectively mimic the current comments system but use vBulletin as the backbone (its the way I plan to do it with my site when it FINALLY gets out of design stage (8months late ._.).

I’ll be honest and say i’m not the sharpest tool in the shed when it comes to html and php and co, but having played around with vBulletin for almost 5 years now, from the humble 2.0 days, upto 3.5. Im more than willing to dispense some advice, as I’m sure the nice folk over at vBulletin.com and (probably better suited due to integration needs) vBulletin.org

Hope you get it all sorted out.

The IGN thing thing better ave been a joke

Any idea why some comments aren’t being accepted at all? hm?

Please do not put comments on the forum! The only time I read comments are when they are inline, and i’m sure i’m not the only one!

You should be able to see from your DB log the transactions with your DB for any given period.

I work with MSSQL daily on many high-volume sites and have never had a problem with an excessive number of database calls - probably because we our own servers tho.

I agree with FAMMY that 5000 an hour is too small. Might not be too small for you as I have really no idea how many visits you get.

You could try to see how many calls to your DB are being made by each page by placing a small debug code on your page.

Or, depending on how this WordPress thing works (haven’t looked), make your own comments DB on another cheap host somewhere and throw it in on each page.

I’m a little late to the party here, but check out www.opensourcecms.com - They’ve got a load of CMSs that you can try about before you install them, and reviews and comments regarding each one too…