en-us de es fr it pl pt-pt ro ru

Short interruption

8 replies [Last post]
Joined: 10/27/2008
IGN: Muwing
Server: Macedon
User offline. Last seen 25 weeks 5 days ago.

During a few hours, the AOP host has technical problem, and the site was unavailable.

This short interruption was totally independant of the site itself, and unfortunately it sometimes occurs - it was the longest break (about 6 hours) since AOP has been released.

My apologies for the inconvenience.

__________________

To aim at Perfection feeds our souls.

Joined: 10/27/2008
IGN: Muwing
Server: Macedon
User offline. Last seen 25 weeks 5 days ago.
Motherboard replaced

AOP was available during about 1h20min today: the server motherboard died, and has been replaced.

Joined: 10/27/2008
IGN: Muwing
Server: Macedon
User offline. Last seen 25 weeks 5 days ago.
Kernel updates

AOP was unavailable during a several hours today, the server kernel needed some work and updates (too hard and useless to explain).

Sorry for the problems, some additionnal configurations may be necessary in the next days (what would imply other interruptions too).

Joined: 10/27/2008
IGN: Muwing
Server: Macedon
User offline. Last seen 25 weeks 5 days ago.
Migration

No luck, the rack hosting the server had several servers down these last hours, and the AOP's one was among them.

By the way, I'm going to move to a new server - a bit more performant - by next days. The migration should be pretty transparent, even if there will be some short interruptions during the process.

Joined: 10/27/2008
IGN: Muwing
Server: Macedon
User offline. Last seen 25 weeks 5 days ago.
Database moved

The database has been successfully moved to the new server.

The migration should be fully finished by 48 hours (files, domain name...).

Joined: 10/27/2008
IGN: Muwing
Server: Macedon
User offline. Last seen 25 weeks 5 days ago.
Migration completed

The migration is completed, and AOP runs on the new server from now, just in time for the new Atlantica patch!

If you see any problem, please tell me :)

Joined: 10/27/2008
IGN: Muwing
Server: Macedon
User offline. Last seen 25 weeks 5 days ago.
AOP server down

The server was down during several hours.

Unfortunately, it wasn't a random & inevitable incident, but an association of 2 factors:

  1. The (type of) server hosting AOP will disappear soon, and some limitations have been added. To put it plainly, the server is less performant than before, and I'll have to change it by next month.
  2. There was a peak of traffic. Indeed more and more players visit AOP each day, especially this summer (students on holidays). While the game is in maintenance, the peak is still higer. In short: there never were so many simultaneous visitors on AOP.

As I don't have the money to pay a better server, I'll have to work harder on the new website so as to optimize it (mainly the SQL requests).
On AOP, I might disable some greedy and (almost) unused features too - as the guild forums.

Joined: 10/27/2008
IGN: Muwing
Server: Macedon
User offline. Last seen 25 weeks 5 days ago.
Guild forums disabled

The server was down during near 3 hours for the same reasons, plus the fact I was doing heavy work on it (as updating the site with the new patch info).

That's why I disabled the Guild Forums. Only 5 guilds used them (and the last message was posted 6 weeks ago...), and they still are available for them. New guids, and old ones with 0 message, won't benefit this feature anymore.

The Craft Calculator is temporarly under maintenance too, and will re-open soon: when AOP is updated with the last patch, including the craft database.

That should be enough to provide a good stability until the next website/server, else I'll do the necessary.

Joined: 10/27/2008
IGN: Muwing
Server: Macedon
User offline. Last seen 25 weeks 5 days ago.
Web crawler

After studying the logs, I found that a nasty web crawler from Poland (chello.pl ISP) was bombarding the site with numerous simultaneous requests. Too many for the server, which finished to freeze/crash.

I tried various solutions to counter this problem (that's why the site was very chaotic these last 2-3 hours), but nothing was very good.
So I banned the IP, simple but efficient, until I find a better solution. I saw the result live (with ~80 visitors + ~10 members on the site):

Before the ban:
CPU (the 2 cores): 85% to 100%
Memory: Very high, when it comes to the swap the server "freezes"

After the ban:
CPU (the 2 cores): 0% to 35%
Memory: Normal

This kind of crawler (see the Wikipedia definition)

Quote:
Crawlers can retrieve data much quicker and in greater depth than human searchers, so they can have a crippling impact on the performance of a site. Needless to say, if a single crawler is performing multiple requests per second and/or downloading large files, a server would have a hard time keeping up with requests from multiple crawlers.

As noted by Koster [25], the use of Web crawlers is useful for a number of tasks, but comes with a price for the general community. The costs of using Web crawlers include:

  • network resources, as crawlers require considerable bandwidth and operate with a high degree of parallelism during a long period of time;
  • server overload, especially if the frequency of accesses to a given server is too high;
  • poorly-written crawlers, which can crash servers or routers, or which download pages they cannot handle; and
  • personal crawlers that, if deployed by too many users, can disrupt networks and Web servers
.
which doesn't respect any standard (as Google / Bing robots) has a very bad impact on the AOP's server.