Offline or long load times

[Foozinator]

I've noticed that GitP seems to take a long time to load, sometimes. I don't know if this is a traffic problem or what, but it often times out on my browser.

Pingdom Tools seems to confirm that the server takes an awful long time to respond.

http://tools.pingdom.com/fpt/?url=http://www.giantitp.com/comics/oots.html/&id=647150

[Assassin89]

Images increase the amount of data on a page. Since this is a webcomic site, images are necessary. There is also the matter of large text based inputs, which also increases the size of the files.

[Foozinator]

It's not just loading images over a slow line. This is a "server took too long to respond" issue. I'm getting it on the forums, here, and the table of contents as well.

The page contents are irrelevant if the server takes more than 20 seconds to start sending the HTML.

[The Neoclassic]

Yeah, I have that problem too; not less than 10% of the time it'll tell me I can't load that page. Refreshing usually works, but it's really annoying when it takes so long so consistently (even when it doesn't give me that "Can't load" message). I am guessing this is related to the amount of site traffic, but I know nothing about computers so I won't start throwing around terms that I don't understand. :smallbiggrin:

[Vonriel]

I've experienced this as well, and it's only this site that it happens with. Typically, it happens when I first try to connect to the site, and again whenever I try to post a reply. Whenever it does lag when I try to post a reply, it often gets eaten by the angry lag monster, meaning I've somewhat given up on trying to post recently out of frustration with so many lost posts.

Time for fun! Let's count the number of times this post gets eaten by lag: 1 (Yes, ironically enough, I lost a post wherein I complained about losing posts to lag. Thankfully, it only happened the once.)

[Mauve Shirt]

I'm pretty sure it's related to site traffic. It gets a lot of it.

[Deathslayer7]

I've had that problem on mutliple occasions. Usually when the new web comic is up, or shortly afterwards.

Best thing to do is just wait it out. That is what I usually do.

[Destro_Yersul]

I'm pretty sure it's because of traffic. This is a popular site, with a lot of visitors. High traffic increases the amount of time the server takes to respond, simply because of the huge number of data requests it has to deal with. Imagine, if you will, that you click on the reply button.

Given that this is a forum, there are probably a few thousand other people who also just clicked the reply button. So the server has to deal with a few thousand requests for replies in different threads on different subforums. That tends to clog things up a bit.

It gets really bad when a new comic has just been posted, especially considering the RSS feed, because then the umpteen thousand fans of the comic will flood the site with data requests. This is also the reason the forums are sometimes taken offline just after a new comic comes up. With all the people discussing the new comic, in a thread that can move three pages per minute... yeah. It gets busy.

[B-Man]

On the topic of the RSS feeds, why do they list all the comics? Why can't it just be the most recent 10 comics?

[Nerd-o-rama]

This is a problem. It's been a problem for years. There's nothing new here - the site always slows to a crawl when a new comic comes out (such as Erfworld #136 today), and that's if we're lucky and it doesn't collapse entirely.

It's because the site is image-heavy and has two popular webcomics and an ridiculously highly-populated forum all on one server (the huge freaking RSS feeds for the comics and email-based subscriptions to threads don't help either). The processing power of the host computer just can't handle it properly.

What I'd suggest offhand is springing for a separate server to host the forums on, but that's fiendishly expensive and apparently not within Giant In The Playground's budget, so we get stopgap measures when it gets particularly bad. Today...isn't even particularly bad.

[Fostire]

This is a problem. It's been a problem for years. There's nothing new here - the site always slows to a crawl when a new comic comes out (such as Erfworld #136 today), and that's if we're lucky and it doesn't collapse entirely.

Actually I've been having this problem for several days now (including the holidays when the traffic was low), so I don't think the new comic is the issue (though when a new comic is up it becomes a LOT worse).

[Shhalahr Windrider]

Yeah. GiantITP is a victim of its own popularity. :smallsigh:

[Rawhide]

If the server is taking a long time to respond, please cancel loading the page and come back another time. Constant refreshing only makes the problem worse.

[Adumbration]

The problem is that I had this problem even before the update. Even though I've managed to save all my post so far by backing out and trying again, it's still a bit frustration. It's also strange how I can get to the forums faster by going through the main page, and then clicking the forum button, instead of going straight to the forums.

The good news is that when it does respond, it loads fast, but that's about it.

[Zherog]

On the topic of the RSS feeds, why do they list all the comics? Why can't it just be the most recent 10 comics?

You're not the first person to bring up this question. I don't have answers -- I don't use RSS feeds (hell, I'm not even entirely sure what an RSS feed is :smallredface: )

[silas the monk]

I am finding it increasingly necessary to write a post in a text editor and paste it in. That way if the lag monster gobbles your post, you can heal up, come back and thwack the brute on the head, with your very polished post.
If nothing else at least copy the post to the copy/paste buffer.

[pearl jam]

I will say that this seems to be happening much more regularly of late...

[The Glyphstone]

Definitely, the server's been all over the place for me in the past week or so. Is this happening for other people?

[valadil]

Usually I don't bother with the forums here on a comic day. The rest of the time it's fine though. This week has been an exception. One minute it loads, the next it won't. This is especially annoying when the timeouts come while I'm trying to send a post.

[Nerd-o-rama]

You know, I have to admit, the problem is persisting more than it usually does. It was still far worse just before and after comic uploads, but there's still echoes of it even today.

[BizzaroStormy]

For the past few weeks my internet as a whole was slowed to a crawl. Thrusday, I found out that it must've wanted my 360 as a sacrifice because the day my system red-ringed, my internet speed doubled.

[Tormsskull]

Yeah, its been a problem for a while. Tends to flare up at different times. There have been several times when people have thrown out suggestions, but all have been unacceptable to the admins. I don't imagine that changing anytime soon.

[DreadSpoon]

What I'd suggest offhand is springing for a separate server to host the forums on, but that's fiendishly expensive and apparently not within Giant In The Playground's budget, so we get stopgap measures when it gets particularly bad. Today...isn't even particularly bad.

It is not fiendishly expensive at all. We host sites that get over 10x the traffic (including lots of large image downloads) for far less than ThePlanet charges. For the kind of server ThePlanet charges $500/mo for, our colo facility would charge $1000 up front followed by $50/mo. Within three months you'd already start saving almost half a grand per month in charges.

The loading problem really does not really big, expensive servers at all. It usually comes down to poorly optimized site code. I mean, heck, somebody actually paid for vbulletin for giantitp, which is a relatively inefficient forum system. It takes far more resource than is necessary for what it does.

The images are being served off the same Apache instance as well instead of serving them from a dedicated light-weight httpd server (which can run on the same physical machine -- just needs a separate IP to bind to). That alone can have HUGE impact on performance for image-heavy sites, as the setup giantitp.com is currently using has to share the total number of httpd connection between slow, long-running PHP scripts and serving a ton of huge images. I've seen sites brought to their knees because Apache just wasn't able to keep up with requests due to too many connections being taken up by long-running (but non-CPU-intensive) scripts combined with serving large files that just took a while for the server to send out over the pipe. Fixing the poorly-written scripts combined with moving the big files to be served by Lighttpd is a solid fix that requires not a single extra dollar to be spent.

It could also help to use a better image format. I have no clue why the Giant keeps using GIFs when 8-bit PNGs are smaller and supported by all browsers. (IE6 and IE7 can't handle translucency in PNGs, although they can handle GIF-style transparency in 8-bit PNGs just fine.) A few quick tests show that 8-bit PNGs are on average 85% the size of the GIFs the Giant uses now. A lot of people don't realize that PNGs can be saved in 8-bit format, so perhaps the Giant just didn't know that that was an option.

I agree with previous posters that the RSS feed has no need to be more than 10-20 entries at most, especially given how infrequently both comics update. If there is some external service that the admins want to integrate with that requires a full RSS feed then there is no reason that the full RSS feed could not be moved to a private, hidden URL just for those services while the public one for people and their comic RSS aggregators is trimmed down.

The server itself could also possibly use some proper tuning. Too many people just buy the most "powerful" server they can afford, slap in a stock install of Red Hat, and call it a day. The default Apache install for instance has a ton of tuning options that are all set at very modest values by today's standards. Apache does not auto-tune itself to the server's capabilities, so it requires hand-tuning and keen understanding of what the server and its load requires.

[InaVegt]

If the server is taking a long time to respond, please cancel loading the page and come back another time. Constant refreshing only makes the problem worse.

While I wouldn't mind doing this, there is a big problem that needs solving.

How do I stop my session from timing out in this case?

Because a times out session means losing all details on read status, and I can't do what I used to do on YABB (Look at the page list on forum overview, and choose the last visited one.)

If you can provide me with a solution for this, I'd be very much obliged.

[Levyathyn]

I make a WordPad backup-copy of every post I make, because when I don't, it crashes 50% of the time without fail. I wish it wouldn't, but it's even more infuriating when it locks me out for hours. =(

[Nerd-o-rama]

*snipped*Well, you have to remember that the forum owners and admins here are not very technical (newsposts from years ago indicate that Mr. Burlew, for all his artistic prowess, is often bested by his own laptop), and there's no one on staff to write the site code. In fact, I don't think this site even has paid staff; our forum admins and webmaster work pro bonum as far as I know, and code optimization for a whole website is work you want to be paid (hourly) to do.

Your points about tuning and CPU/application management, however, I will keep in mind if I should ever happen to host a site myself for any reason - I can . In fact, I'll pass them on to the next webmaster contract that hires me.

[Shhalahr Windrider]

Well, you have to remember that the forum owners and admins here are not very technical (newsposts from years ago indicate that Mr. Burlew, for all his artistic prowess, is often bested by his own laptop), and there's no one on staff to write the site code.
The admins other than the Giant (Rawhide, Raw Bear, and WampaX) are quite techinically inclined. That's why they're listed as Technical Support in the Forum Rules. And they've shown a lot of technical know-how in their customization and management of these forums as it is.

[metallica48423]

The problem with vBulletin, while being a fairly solid forum platform, is that it tends to be very VERY hard on servers. It is probably the heaviest of the big 5 forum softwares out there. Correct me if i'm wrong, but you guys were using YaBB before? vBulletin isn't solely to blame though. Any software will have problems as resources wear thin. Different softwares react differently, however.

DreadSpoon pretty much hits the head right on the nail. I do not intend to profess on anyone's choice of hosting provider, as I do not know details, but depending on your skill level with server administration and optimization as well, you might consider an unmanaged server solution. However, what you save in cost you lose in support, so its only really an option if you are able to manage and administrate your own server.

You might consider using an httpd such as lighthttpd to serve static content such as images and HTML pages -- and, coincidentally, the comics -- of course i'm assuming that they aren't dynamically generated and instead using some sort of system which creates HTML pages?

While the main bottleneck is generally USUALLY either in SQL or in php memory usage, reducing the load of serving static content can help towards having more resources available for the more intensive things, and can improve responsiveness in the server.

In any case, i'm not trying to profess or assume I know how these things work here, nor tell anyone what to do. I saw this and thought i'd offer some ideas :) Thanks for a great comic

[Rawhide]

Yeah, we were using YaBB before, it was... less than efficient and struggling under the size of the board. An upgrade was required and a balance between efficiency/speed, security, reliability, features, usability, user friendlieness, aesthetics, customisability, administration overhead, cost and support needed to be found.

As everyone have their own lives, work and responsibilities, a hosting solution that provides good support, monitoring and security was required on top of the usual speed, cost and reliability. I also cannot stress enough just how important good support is. You can't just buy the cheapest hosting provider around if they aren't going to provide you with the service that you require.

Running static and dynamic pages on the same webserver is less than efficient and far from ideal on a busy server as each needs different optimisation settings, however, this is a limitation we cannot currently overcome and the settings need to be balanced between them. Using a separate http daemon on the same server is not an option either as it will unfortunately break certain back-end software.

[DarkCloud]

Perhaps I just didn't see it, but it seems that the place with the most burden on the site is the Silly Message Board Games... and no plans to prune that section appear to be posted there?

Lots of posts there are borderline spam...(Though obviously many there enjoy the posts) But there's really little reason to preserve any post there more than 9 months old, or even 6...

[Rawhide]

We constantly prune threads older than 1 month in the SMBG and have been for over a year now. It's even in the forum description.

[metallica48423]

As everyone have their own lives, work and responsibilities, a hosting solution that provides good support, monitoring and security was required on top of the usual speed, cost and reliability. I also cannot stress enough just how important good support is. You can't just buy the cheapest hosting provider around if they aren't going to provide you with the service that you require.

Completely and totally understood! Sometimes going with unmanaged solutions just isn't an option as you need available that knowledge and ability to call on someone when things break -- and you know computers, they can and WILL do so! Especially under such load. I simply thought to mention it as a cost cutting measure but like i noted in my post -- in some circumstances it simply is not an option.


Running static and dynamic pages on the same webserver is less than efficient and far from ideal on a busy server as each needs different optimisation settings, however, this is a limitation we cannot currently overcome and the settings need to be balanced between them. Using a separate http daemon on the same server is not an option either as it will unfortunately break certain back-end software.

Yeah, that is one of the problematic things, especially if you don't have the time or resources available to... well... basically break and then concurrently fix everything, to put it in crude terms. That's about what it'd amount to. I've found the only real time to make such a fundamental change is when making a changeover to a new server where you would have to do that for the most part ANYWAYS. In addition, with a site like this, taking the downtime to do such isn't much of an option either.

Regardless, you guys do well to try to keep the server under control and I do commend you guys for it. I know how it goes all too well :P Hats off to you guys.