Trouble has just undergone its biggest change in a few years: It is now CURRENT because I upgraded it to Ubuntu 8.04, Hardy Heron yesterday. Yay!

For those who use it, the biggest change is that we’re now running Apache 2.2, which thoroughly screwed my authentication setup for a few hours yesterday. There was also a glitch during the upgrade where Trouble forgot who its users were for about an hour in the middle of the day. This is important because anyone with a high frequency mailing-list subscription on Trouble may have been kicked due to ‘no such user’ bounce messages generated during this window. If so, my sincere apologies: I am usually much better at avoiding this kind of thing.

As always, O user-base, let me know if anything isn’t to your liking.

1. If I don’t mind being in the USA, it still doesn’t get cheaper than ServerPronto, for equivalent resources, not even with VPS.
2. If I insist on local (Australian) hosting for latency reasons, I would have to go with a VPS, and probably pay more, especially for traffic allowance.
3. The pain of moving to a new OS instance, essentially rebuilding from scratch, is too horrible to contemplate right now.

So Trouble is staying put for the time being.

In case anyone is interested, here’s a list of Australian entry-level VPS places, compiled with the help of the SAGE-AU technical mailing list:

• netlogistics.com.au
• labyrinthdata.net.au
• gplhost.com
• xenion.com.au
• web24.com.au
• rimuhosting.com

I am, however, probably going to upgrade, once Hardy Heron is out, since it’s the next LTS release, and as such it has an in-place automated upgrade path from 6.06, which is getting a bit long in the tooth. This will be preceded by plenty of careful backing up and provision of lots of nice rollback mechanisms, so there will probably be more news here before any such upgrade goes ahead.

Later-the-same-day-edit: E points out that crucial.com.au are currently having a special which brings their rates down roughly into line with the competition!

It may not be tomorrow, nor even the next day, but Trouble will move.

You may have noticed that Trouble vanished on Wednesday and reappeared late last night.  This is because ServerPronto decided to cancel my account with them without warning or any reason that they could explain. This has now been remedied, but not explained.

So, we’re moving, probably to entic.net.

Dear Customer:

In order to ensure continunity of operations, we must perform emergency maintenance on the electrical system. This maintenance will be performed on September 9, 2007 beween 10:00PM and 12:00AM, USA Eastern Daylight Time (New York City). During this period approximately 600 of our over 6000 servers will experience a brief power interuption that should not last longer then 30 minutes.

If your server will not reboot without issues in the event of a power interupttion, we would recommend that you:

1. Open a ticket with us to determine if your server will in-fact be affected.

2. If we confirm it is affected, and your server will not reboot gracefully, shut the server down gracefully immediately prior to this time, and open a reboot ticket requesting that we re-start the server for you.

3. When the maintenance is completed, our staff will reboot your server per the ticket you created.

We will have extra staff onsite during this period to address this and any other issues that may arise.

We apologize for any inconvenience this may cause you, but assure you it is necesary to maintain the reliability of our backup power systems.

Thank you for your patience,

The ServerPronto Team

Fortunately, Trouble is quite polite about coming up intact after a reboot, so there should be no extra unexpected downtime or data loss. Just make sure you’re not half-way though editing an unsaved blog-post at the time.

Both Trouble and Zikzak went down at about 10:00am AEST and came up again briefly from 10:50 to 11:00, then went down down again until about 11:05.

Uptimes on Trouble and Zikzak at 10:50 were 3 mins and 20 mins respectively, suggesting that there had been some kind of widespread power outage at the site, but there is nothing to suggest that power was lost during the second outage.

ServerPronto‘s billing website was down throughout and is still down at time of writing.

Will follow up here as any further information becomes available.

If you require ultra-urgent fixes, like because you’re being spam-bombed by something, call me.

My sincere apologies to all those who missed it over the weekend and Monday. No mail should have been lost, and if you find that a file you want is missing, give me a hoy: there are probably still some holes I’ve missed.

In reaction to this, I have installed the smartmontools package and set it up to frantically mail me if anything goes even slightly wrong with the disk. Will keep you posted, if the posting forum remains available.

In unrelated but reassuring news, the OSSEC author Daniel Cid has posted a straightforward exaplanation of the spurious rootkit detection issue: netstat won’t list a socket if it is allocated but never used, however attempts to re-allocate the socket will still get ‘in use’ messages. This will very likely happen a lot if you have something moderately hefty stomping on your available ports like, say, the LDAP which underlies trouble. Hmmm. There may be a fix, we hope. I have asked for one, anyway.

At some stage in the not-too-distant future, I will get around to publishing the thoroughly over-engineered ‘hedge’ script that’s handling the backups, since it seems to fulfill a useful purpose. I’ll post a pointer here when I do.

Also, wanting to welcome two new admins, and acknowledge one long-standing one. Wanting to, but I’m not going to use anyone’s real names or even usernames here. You know who you are.

Like Descartes, I must resign myself to the knowledge that if evil demons are in fact deceiving me about the state of things, they at least seem to be doing an adequate job of it.

At least now we have the rudiments of a backup system, and a lot more security than before. It is, I suppose, astonishing enough that Trouble is running at all, given how little time I have to work on it these days.