Forum temporarily disabled, bandwidth exceeded, website dying, er, “help?”

July 3rd, 2007 by Ben Goldacre in bad science | 72 Comments »

Sorry about the very long outage, my host suddenly and without any announcement simply pulled the plug on all CGI (geek stuff) because I was apparently putting too great a strain on their servers, and I’ve had to switch the forum off (at the very least) until I get it sorted. It turns out that I was the hog making everything run slowly: who’d have thought? This blog is back online but only on borrowed time now I’m afraid, to give me a chance to upgrade to WordPress 2.2.1 and see if that will help reduce the CGI load for a bit. I’ve barely got time to do even that right now.

If anyone could offer help that would be amazing, I need to upgrade WordPress and phpBB. I really can’t pay for web skills at the moment – must start vitamin pill company instead of giving talks for free in schools and losing train ticket receipts – but I can give love, kudos/publicity, and I know some pretty girls. Might be better to email instead of posting below as the site will probably go down again soon, maybe for quite a while, so ben@badscience.net is the best.

Also unfortunately – with the best will in the world – I guess I maybe now ought to move to a bigger host to speed everything up, and most importantly remove any chance of this kind of thing happening again (the hours wasted already…) so it would be good to get opinions on what hosting can handle a min of 12,000 visitors a day to a mySQL intensive WordPress and phpBB installation, and run really quickly? Ideally somewhere a step up from my current ten dollar arrangement but a bit short of the proper big money for a dedicated server?

Anyway thanks and sorry for the outage, hopefully going to sort it all out for good this time. Thank god nobody ever goes here or here eh. Look ma, no adverts. No evil drug company funding. Just a broken website. Looks like the web2.0 cashless volunteer ideas economy may not work if you lack time and 3L33t webskillz.

Sticks in my throat. “Help.” Can’t someone please just become my best mate and hold my hand for the web stuff? I really love this website being here, but I just can’t spend hours sorting it out at the moment.

update: they say if it goes over the mysql limit at all again then it will get cut off, i closed the forums, upgrading wordpress might buy some time but i cant do it myself right now.

to be clear, copied from a scattily written email, if anyone can help right now my needs (as well as getting through punishing outpatients clinic) are:

1. upgrade wordpress and phpbb to latest versions to buy some time with current host, and get myself good back ups

2. identify good medium grade shared host – would rather pay and have certainty etc. i need somewhere better than $10 but ideally not £100 a month dedicated server if can avoid. i want to pay and have reliability and not be beholden. site gets 12,000 visitors a day, 2gb/day, but that will go up quite a lot with the podcasts, interviews, and lectures that are coming v soon. it really hammers the mysql servers, because of being wordpress and phpbb, this is much more important than bandwidth: fast good mysql servers.

3. migrate to new server

4. there is also some really interesting medium term work to be done on interesting modest community building tweaky aspirations, eg get the forum better integrated, but most importantly, take the miniblog, and instead of it just being a tiny feed, making it so that when you click, you go to a discussion page, like the b3ta links board i guess, with the link at the top, so people can follow it from there, or discuss the content, because at the mo there’s great stuff in the links board and could form the focus of some really interesting new geek community building, especially if i can start dishing out logins to other people to post interesting links there too. the stuff in the miniblog gets discussed a bit in the forums, but would be good to have more opps for chatter, since the standard of banter on the board is so high.

5. ideally someone to help me with everyday maintenance of when things cock up etc, or adding amusing tweaks.

ideally there’d be a couple of people to help fix this current problem, and then a dedicated young whippersnapper who fancies the community building challenge of the new miniblog links discussion thing.

cant really offer cash, although some paypal things are coming in today. the site costs me money as a rule, which i’m fine with, although cant carry frightening 40 quid an hour fixing things bills and very tough to find the full day it would take me to sort this with my legendarily poor webskillz as fulltime junior doc and column/blog only hobby, and all these unpaid talks i spend my time doing which i love but where i lose the receipts even for expenses are frighteningly closely booked right now.

so, dont like to be dependent or demanding, but i guess i do spend half my time talking to people about their kids medical problems, notably without charging them 40 pounds an hour for the privilege. can offer adulation and free glory, and very good karma. and at some stage i’ll leave my low pay jobs, qualify fully as a proper grown up major doctor, accept an offer from another newspaper or something, and be rich enough to pay through nose. that day will indeed come. ideally i’d become best mates with a really motivated geek with lots of spare time looking to build a portfolio and cv from a well visited site, and if cash ever comes in from some related big web project (working on it) then you’d be first port of call. trreading fine line between grovelling, whining and carroting here.


++++++++++++++++++++++++++++++++++++++++++
If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

72 Responses



  1. JQH said,

    July 4, 2007 at 1:26 pm

    Donated a tenner of my beer fund.

    As a long term measure, a decent forum host is going to cost something to run is i worth making the forum subscription only either when you have reached a post limit (say 30?) or after a certain period of membership (say three months?). That way we might avoid this sort of problem in the future.

    Just a thought. I’m missing my fix and getting a bit twitchy…

  2. tom1 said,

    July 4, 2007 at 1:47 pm

    JQH,

    I think there is a philosophical question here about whether this is to be a lets-invent-radar-with-some-damp-string-and-whatever-I-can-find-in-the-shed type operation, or start inching in the direction of the JREF where they recently spent $5000 on a new server (though there is considerable room for an intermediate position).

    Also, it’s not clear yet whether the issue is Ben’s global popularity having exceeded the capacity of his host, or just some silly configuration issue.

    If we do need investment, I suppose Ben could always look at some kind of PFI model for funding. I am assured that they are extremely prudent.

    Tom

  3. briantist@work said,

    July 4, 2007 at 2:29 pm

    Ben,

    I’m always happy to help in my spare time – I can install/program/configure/maintain as necessary.

    Anything to keep up the excellent work.

  4. Alex McLean said,

    July 4, 2007 at 10:20 pm

    Perhaps you could create a facebook group, so we have somewhere to look for news / provide help if the site goes down?

  5. Despard said,

    July 4, 2007 at 11:03 pm

    I’ve already found two using a global search for “badscience”. :-)

  6. SteveNaive said,

    July 4, 2007 at 11:59 pm

    Important stuff – keep it up – donation made!

  7. Ben Goldacre said,

    July 5, 2007 at 12:13 am

    woah

    ok

    can barely get into the site myself

    and busting up with dayjob work

    briefly: got a server sorted i think, positive internet are the new gods, with a great track record of supporting geek political causes (eg they host richard stallman too)

    lengthy grateful post and a serious lifetime of proper noisy public adulation to follow

    i really dont need money, always happy to sub the site within reason, but anything that comes in will go to petty cash for podcast gear, expenses, sundries, and crack cocaine as ever.

    this has been a v motivating experience and the new website will grow to be much bigger and more community. like the colquhoun thing i think more good will come of it than anything.

    the best thing to come out of it has been some very good offers of help from some v sharp web people, and i really hope they stick around, theres some great stuff we could sort out, will post properly soon (but can barely get in to admin pages and dont want to clog blog with work-in-progress stuff).

    i’ve also got a lot of great new content stuff coming, more to follow. also spending lots of time atm giving talks in various places and spreading the word on putting content free online. i think the royal statistical society for example are hopefully going to put some mp3 stuff up, which is great, the stuff ive been to there has been great. specialists talking to high end generalists is the future.

    thanks again and no more money, to be fair, unless you just want to. i need your time, energy, and banter more than anything. viva etc. a new fast blog, a new bigger site, hurrrah!

  8. seenoevil said,

    July 5, 2007 at 9:21 am

    Brilliant. Hope it starts soon.

  9. tom1 said,

    July 5, 2007 at 9:41 am

    Feel free to put my donation towards beer, or a stethoscope made of gold.

    Glad you’ve got it covered. You know where we are if you need us.

    :-)

  10. Frank Chalk said,

    July 5, 2007 at 9:59 am

    Dear Ben,

    As I have made a small fortune selling my patented WIFI Shields to schools who wish to protect their teachers and pupils against homeopathic radiation, I feel duty bound to contribute in order to help you out.

    ps If you want to know more about the role of Crystal Healing, perpetual motion machines and Ley Lines in the new Science curriculum then pop over to frankchalk.blogspot.com

    Regards
    Frank Chalk

  11. stever said,

    July 5, 2007 at 10:27 am

    thanks to everyone for helping out.
    And for giving me a nice little holiday from moderating duties.

  12. tom1 said,

    July 5, 2007 at 2:09 pm

    For the t-shirt:

    A picture of Ben saying:

    “I ran a double blind, peer reviewed study showing your quack medicine is junk, and all I got is this lousy t-shirt!”

    Tom

  13. coracle said,

    July 5, 2007 at 6:22 pm

    I’ve got a design in the making featuring a very stylish double helix. I’ll have to get my arse into gear and finish it off.

  14. Gimpy said,

    July 5, 2007 at 8:51 pm

    For the t-shirt:

    I’ve got a picture of Patrick Holford performing an act of onanism on a courgette cock.

  15. tom1 said,

    July 5, 2007 at 10:12 pm

    Ummm… What internet sites have you been visiting Gimpy?

  16. bootboy said,

    July 5, 2007 at 11:36 pm

    Although it seems that Ben is well served already (yay) I can help out on the technical side if needed. I have designed and developed a couple of content management systems over the years, managed pretty large-scale internet infrastructures and know the insides of php, apache and mysql pretty well.

    With your set up the rate limiting factor is definitely going to be the database – fancy caching mechanisms can go so far, but when you reach a certain number of visitors, and as your archive grows (meaning people arrive at diverse pages more frequently – pages which will not be cached), you soon reach a stage where you start saturating connections to shared database hosts.

    The problem with this situation is that the upgrade path is not always straightforward. The reason that you are getting cut off is because you are sucking a disproportionate amount of stuff from the ISP’s big fast database server. You may find that, given the level of traffic, an upgrade to a virtual server makes the site impossibly slow. The only guaranteed solution that leaves you headroom for more growth is a multi-server set up with a dedicated database server. Problem being that that’s expensive.

  17. Andrew Clegg said,

    July 6, 2007 at 9:49 am

    I remember the courgette one from the forums — nearly fell off my chair. Would definitely buy a t-shirt with that on.

    Andrew.

  18. tom1 said,

    July 6, 2007 at 10:14 am

    Bootboy,

    quite a few of us have been trying to assist Ben, but you’re the first poster who seems to be a specialist on the main issue.

    As I understand it, Ben is off sorting out some alternative hosting with his new best mates at Positive Internet. Presumably with a few more horse power the site should at least be bought some time.

    If Positive Internet are providing enough support to address the longer term issues then cool. If not, then perhaps some community assistance (hopefully intellectual rather than tin) may be necessary if Ben isn’t going to have to spend all his crack money on his website.

    Tom

  19. pauls said,

    July 6, 2007 at 11:41 pm

    Speaking as a member of the Positive Internet support team, we’ve just got the new server set up, and the details off to Ben this evening. I’m sure he’ll be contacting people for help over the weekend, but we’ll be here to assist too, in the short, medium and (hopefully) long term :-).

  20. manigen said,

    July 7, 2007 at 10:26 am

    Now that’s a good sign. The support team actually visiting the site, I mean.

  21. tom1 said,

    July 7, 2007 at 12:22 pm

    Wow! “Positive Internet” – does what it says on the tin.

    I’m around and available if anything needs helping with. It sounds pretty well covered though.

  22. nmailer said,

    July 11, 2007 at 3:27 pm

    I’m one of the Directors of Positive. Things seem to be progressing nicely, and the good Doctor has transferred to his new dedicated server. One of our head techies is giving him a call just to make sure everything’s running ok, that the backups are working and so on.

    Some people above have expressed concerns about future capacity. This shouldn’t be a problem. We’re well used to running popular things on GNU/Linux platforms. For example, we did Ricky Gervais’s podcast, and that got us into the Guinness Book of Records for the number of hits it got. Perhaps this site’ll be next 😉

    Anyway, in the unlikely event that the site does begin to overburden the dedicated machine we’ve given Ben (he was previously in a shared environment), then we’ll give him an even more powerful one, or a cluster. This site is an important candle against the return to medieval darkness, so we won’t be the ones to let it go out.