Technological Wanderings Travels in code, databases and networks

Go to: http://api.rubyonrails.org/classes/ActiveRecord/Migration.html

It opens without problems.

Go to: http://code.djangoproject.com/wiki/SchemaEvolution

This page works also. There is a link to the Rails page about half way down - click it. The rubyonrails.org page will now fail with "Precondition failed" (try a refresh if it does open, as your browser should have already cached it).

Try both methods in separate tabs: you can refresh both rubyonrails.org URLs all day long, but the tab with the djangoproject.com Refer header will always fail in the same way.

Now that, my friends, is messed up.
I will let you draw your own conclusions.

A week or two ago I was presented with this on two of my older Gentoo servers:


[blocks B ] sys-libs/ss (is blocking sys-libs/e2fsprogs-libs-1.41.2)
[blocks B ] <sys-fs/e2fsprogs-1.41 (is blocking sys-libs/e2fsprogs-libs-1.41.2)
[blocks B ] sys-libs/e2fsprogs-libs (is blocking sys-libs/ss-1.40.9, sys-libs/com_err-1.40.9)
[blocks B ] sys-libs/com_err (is blocking sys-libs/e2fsprogs-libs-1.41.2)


This should strike fear into any Gentoo user out there. Usually when this happens, there will be a good result in Google which states exactly what to do. Not this time.
Importantly, you should wait for someone else to do this first and post in the Gentoo forums about it. Because Gentoo is not a professional distribution - far from it - as much as I love working with sources and portage (as a long time FreeBSD admin).
In this case, doing the obvious of unmerging the old blocking packages will render emerge useless as you will break wget. Yes, this is ridiculous.

Well, lucky you - I've done this now on both machines successfully and present to you the winning formula:


emerge -fv sys-libs/e2fsprogs-libs sys-fs/e2fsprogs
emerge --unmerge sys-fs/e2fsprogs sys-libs/ss sys-libs/com_err ; emerge sys-libs/e2fsprogs-libs sys-fs/e2fsprogs


We fetch the new packages before breaking the system. Then we break the system, then we fix the system with the parts of it that are still working. e2fsprogs is very much needed, unlike what I previously said here - it is what provides such gems as fsck...

Now I can continue emerging the weeks of updates that have built up. Including an OpenSSH 4->5 update that needs careful merging of configurations.

I really like Gentoo, but it can drive me crazy. It also has a problem with the people behind it - the culture is much like that behind Rails, in that I have a real problem trusting it. See the next post for an example from Rails...

Last year I spent an inordinate amount of time bidding on jobs on guru.com and elance.com. I did win a few jobs and managed to make a small pittance (which was completely wiped out by subscription costs, escrow fees, commissions and taxes - oh my god, taxes: double my calculations... why do I continue to put up with life in the UK? It's the best I've got I suppose).

The win rate was tiny. 3% or so.

Towards the end I started placing bids at an equivalent of 3 or 4 pounds per hour and was still being undercut. Incredible. The rates in the UK for software development are 40 to 150 pounds per hour, even more for specialist work. The absolute minimum to live on and provide the infrastructure for doing the work, given a 50% utilisation rate, is £30 per hour (equivalent to perhaps £6/hour at a normal salaried job excluding any holidays) - but the utilisation rate with online boards ends up as just a few percent as incredible amounts of time is spent writing quotes to disappear into black holes.

Normally it'd be time for a sales person (which would be me, again) to make a follow up call, sell the quote and get the job - but no, not possible. You can continue to drop messages into private message boards all you like but you'll not get a response for most bids - 99 in a hundred.

I've ended up back on the boards. They've changed recently. Elance.com I no longer understand. I can't find where the jobs are listed. Guru.com has started revealing win rates in the job listings for those leaving bids. Posting on these boards really is soul destroying and I wish I wouldn't do it.

What I need really is an agent to find jobs for me - I've seen many examples of people working full time like this, one guy brokering the sales and acting as the support/contact point, the other the programmer/implementor. I don't have the time to both find and do jobs, and unfortunately work simply isn't going to fall into my lap.

It isn't just to keep me in work really, there is some altruism in my words. The quality of work I've seen from these hundred or thousand times underpriced projects can be quite shocking. Some of the jobs I've won are to add features to existing sites, often originally built from job board postings. I've found most to be dangerously badly written. SQL injection, even code injection, unencrypted critical data, etc.

I did once post a job myself. If you can't beat them, join them - I'll get a chap in Pune to write my software. Well, the quality of quotes that came back were very variable. I suspect many people, including those receiving the bids, have never seen a professional services quote. There were perhaps one or two from thirty that appeared to know what they were doing. The prices quoted also varied widely - I'd estimated the project at a day's work perhaps, say £500 worth of time from a developer or small team. I had quotes from £10,000 to £100. Disturbingly modal average was around £1000 - double what I'd have quoted. The cheaper quotes were much cheaper, putting the mean average well over £1000. From this I concluded that most employers are going with the lowest bid, no matter what. Eventually I didn't go ahead with the project, and stopped bidding from that point on. Until now.

I ambled onto the Coding Style Guidelines for Webkit recently. http://webkit.org/coding/coding-style.html
Reading this ignited in me a desire to voice something that's been gnawing away at me for years now - styleguides, and what it means when they are wrong.

Now of course, styleguides are really just personal preferences enforced to keep code easy to pick up and read for larger teams. Really, if the code works, is readable, is expandable, what difference does it make?

There's also the issue of language. C-inspired languages are easy to compare, but doing so naively can result in misunderstandings. Then of course there are languages such as Python, which I've recently started learning and using for a huge commercial project. The very layout of the language is significantly different to C, which means a different mindset is needed for visual layout.

Anyway, since Webkit got me into this line of thought, I'll pick at its rules to make my points. I'm going to describe how I think C-like languages should be presented. My mind has changed on this over time, and it will in the future. This is not the word of god - except in the literary sense that the word of god has been changed and rewritten over time to suit whatever truth people want to push on others.

1. Indentation.

Don't use spaces. Really. The amount of time spent cursoring over spaces to get to code wastes any benefits from needing to set up your editor to tabs of four space widths.
The indentation required varies with the readability of a language. I've found that classic C is best with three space tabs. Rexx favours two spaces. Java and PHP read easiest in four spaces.
There are no disadvantages of using tabs if you know how to change the width in your editor (and your editor doesn't enforce tab to space conversion - I'm looking at you, PyDev). There are disadvantages to spaces.

2. Switch statements.

Webkits says:

switch (condition) {
case fooCondition:
case barCondition:
    i++;
    break;
default:
    i--;
}

I say, this is unreadable. Use this:

switch( condition )
{
    case fooCondition:
    case barCondition:
        i++;
        break;
    default:
        i--;
}

3. Spacing.

Make it readable. I am starting to think many modern coders have never read a magazine, or have read only American newspapers - which seem to be stuck in the 19th century in terms of presentation. There is no need to fill every space on the screen with code - in fact, it's counter productive. Use whitespace. Whitespace is good (when it's not significant whitespace).
Use existing features of the language, such as curly braces, to promote spacing.

if( mycondition )
{
    doThis( a );
}
else
{
    a = getThat();
}

For the spaces around items in expressions, use common sense. ONLY common sense. Every situation is different.

a = t*4
b = myCall( a*( b*4 ) )
c = longname * something else

Just make it readable. Put spaces where they made sense, take them away when they don't.

For functions and statements, don't make them different on religious grounds. Yes, if() and myfunc() are very different things, but that is obvious and ingrained in a programmers mind and vision.

if( me )
{
    call( that );
}

"if(" is always recognisable as an if. Always.
And 'me' is always me.
"if (" offers nothing of advantage, except to say "I am clever and know 'if' is not a function".
But "(me)" confuses 'me' with parenthesises - they are part of 'if', not 'me'.

4. Line breaks.

Keep things spaced out. As far as possible, keep separate logical steps on separate lines, using your common sense (don't go breaking up 'if' statements..)
This includes braces as far as I'm concerned.

if( this )
{
    doThat();
    andThat();
}

if( this ) { doThat(); andThat(); }

5. On braces.

Always use them, even for one line clauses. This comes down to uniformity, readability and not needing to switch gears in your brain to work with single and multi-line clauses - they are the same things, just with different numbers of lines.
Put them on their own lines - try paging through a thousand lines looking for something with all the code bunched up - the braces enforce whitespace as well as making it possible to visually detect blocks of code. No, indent don't do it - there are too many exceptions where we break indenting rules.

I agree with Webkit that empty code blocks should have empty braces, it's far more readable with less room for mistakes or miss-readings of a fairly uncommon construct:

for( ; i++; );
for( ; i++; ) {};

6. Naming.

I believe Java has it right - use camel case.
Don't usually place a variable's type in its name. But use your common sense - if you have a variable for which it is not obvious, or similarly named variables tracking similar aspects using different types, then mark them out by all means.

The VB style coding of prefixing every variable is ugly.

http://news.bbc.co.uk/1/hi/england/shropshire/7639792.stm

Oh my, think of the children!

When they switch the mobile phone mast on, 99% of mothers who saw one episode of Horizon dubbed into American when they were on holiday in Florida say that it will form miniature black holes that will grow to destroy the earth!

Ignorance: it's fun when it's a thousand miles away. But put it in my back yard, and I'll be complaining.

I don't have a mobile signal while I'm sitting at work, and I've seen 3G on my iPhone only once - when I deliberately stopped in a city centre to try it out. And it's partly because of these cavemen (and partly because O2 are pathetic, but I have yet to have a proper rant about that).

gpsd seems to be broken when talking to a BU303 GPS device. If the BU303 is put into NMEA mode, gpsd will think it sees SIRF data from it. When this happens, it will change its mode to SIRF which the triggers then BU303 to really go into SIRF. The BU303 is unreliable enough as it is.

This is bad for me, as my Navigator software needs raw access to the device in NMEA mode and doesn't know how to switch it back over.

The answer is compile gpsd from source without SIRF support:

./configure --disable-sirf

(As an aside, I would really like to write a 'fake' serial device which feeds data from gpsd as raw NMEA. That way I wouldn't need to have scripting which kills gpsd (and thus my tracking system) when I want to load Navigator 4).

Microsoft: Get the Facts on the London Stock Exchange:
http://www.microsoft.com/uk/getthefacts/lse.mspx

Well, a few months ago the LSE was sending out incorrect data. Trading data was wrong. The graphs on the BBC were showing random static instead of value data.

Now it's completely fell apart, on possibly one of the most important trading days of the year (bye bye another big bank):
http://news.bbc.co.uk/1/hi/business/7603981.stm

Ah, lovely, somebody has a grab of one of the MS adverts on the LSE:
http://tipotheday.com/2008/09/08/microsofts-foot-in-mouth-london-stock-e...

Microsoft pushing f**king reliability... Jesus... I changed my job recently and a big thing is not being woken at 3am whenever Microsoft deliberately cause a failure on an Server 2003 machine (aka enforced Windows updates reboot).
Or the mornings after a MS update is pushed out to client machines and two dozen of our customers lose their email or file store or whatever MS have f**ked up that time.
Still, it ensures the client is more willing to pay an extra couple of grand for the support contract to keep their machines running (or at least, running most of the time). MS at least cause movement of money.

Now I work with Linux, BSD and Mac. So much happier.

I find it very difficult to watch a big Steve Jobs presentation. It's not Apple stuff, or indeed Steve. It's the audience. At the end of each sentence people start making howling noises, screaming, all sorts of weirdness. It's the same thing you hear whenever NASA gets a rocket to lift off - people who, in theory are quite intelligent, devolve into lower order primates or worse.

Why aren't they ejected for making a nuisance?

My six year old Nokia 7650 has become too difficult to use. A single long call and it's dead, or a day without charging and it'll show full power right up until the moment I need it, when it'll just die. It needs only a battery but my cheap replacement from eBay is no better than the original (and also I imagine it will probably explode; it looks a legitimate Nokia item but at a price of £2.99 inc postage it's either fake or stolen).

I got a freebie Motorola Razr, which has unfortunately completely given up the ghost too (crashes after booting). Plus, it is awful. A truly terrible phone, from the GUI to the hardware and everything in-between. Who put those stupid unlockable buttons on the side? I walk around beeping and taking photos of the keys in my pocket!

Nokia phones remain great - a simple 6300 is light years ahead of the 7650 while offering the same features and more in a faster, tiny and more expandable unit. But if I'm going to the Carphonewarehouse there's only one phone I'm going to walk out with: the iPhone 3G. I've put it off due to the huge costs (£30 a month doubles my bill and reduces my free talk time by around 7 times). I'm also put off by it being with O2 - which I have been with since late 2000 and have learned to despise. I might write about my O2 experiences later, but suffice it to say that if you have a choice you should go elsewhere. No matter what they promise.

Anyway, this is the end of my first week of being an iPhone user. The phone really is as good as the hype. Email and web are great. It is intuitive - you don't need to be told how to use it, it just works as nature would have it - if nature produced hand-held microwave communication devices.

As an iPod, it's as good as the hype in this aspect too. Even the built-in speakers are 'good enough' to be truly usable in the real world. Video? Also good. YouTube works well but the interface lacks a lot of the youtube.com site - but you can use the real site in Safari, only jumping to the YouTube player when clicking on a video. It's also possible to import your own videos into iTunes to sync down to the iPhone - you just need to get the format right, e.g. using ffmpeg on a unix box:

ffmpeg -i SourceVideo.avi -f mp4 -vcodec mpeg4 -maxrate 1000 -b 700 -qmin 3 -qmax 5 -bufsize 4096 -g 300 -acodec aac -ab 192 DestVideo.m4v


Is the iPhone a general purpose computer? Almost. There are lots of apps to download and at very cheap prices. You're not going to compile your own software or write your own scripts though.
Most apps are not free, but with the majority of those at 59p and almost all under £10 it is unreasonable to complain. If you do think 59p is too much to pay for weeks of someone's time to develop the software then I suggest you consider living in the Sahara as a nomad, which would perhaps fit better with your life views than western civilisation.

SSH and VNC apps make the iPhone a reasonable tool for a sysadmin.

There are even car performance apps that use the iPhone's accelerometer to replace hundreds of pounds of specialised in-car equipment to give 0-60 times and such like.

GPS is partly useful: I used a GPS monitoring app to report my speed at all points along my commute. Very cool, except for when I lost GPS in a tunnel and it told me I was doing 130MPH. I was actually doing 40 though roadworks. The navigation works as it does on Google maps, but it misses the one thing to make it usable: voice prompts. I suppose it isn't to be used as an in car sat-nav, as if you lose Internet (GPRS) then you lose your directions.

There are problems with the iPhone:

Safari stability:
When playing tunes in the iPod and browsing, Safari will bomb out regularly.
After a week or so heavy browsing, it starts to bomb out all the time.
This applies to all apps that use Safari - e.g. YouTube and Maps.
A reboot (power off, power on) fixes this. I rebooted this morning and it has remained stable for today.

The ringer button:
1. You only have two options: no sounds, or sounds. The Nokia flexibility for car mode, pager, loud, etc. is lost.
2. The button falls off. It's a metal button glued to a plastic switch. Mine did so after exactly a week. A quick look at Google reveals that it's very, very common, so I decided not to bother getting a replacement phone (which Apple will provide in 15 minutes, but who know what Carphonewarehouse would do...).
I decided to superglue it back into place. This is not without risks on a 7 day old iPhone: you might get glue on the screen, you might glue the switch into place so it is immovable. Etc., etc. Unlikely to covered by warranty.

Reception:
I have yet to see a 3G signal. A problem partly due to having changed jobs and now working in a very rural location. Hopefully I'll get into a bigger town sometime soon so that I can at least test it. My new employer doesn't even have wifi at the moment (and I should mention that phone reception is available only in the car park - for a major software company, I'm very disappointed by the infrastructure - regrets for another post..) so the phone is used almost exclusively at home right now.

Call notification:
Later Nokias have lights that flash every minute to alert you of calls or SMS that have been received while you have been away from the phone. For the iPhone, you need to switch the screen on. There is no other notification.

Anyway, beyond all that, I have to say it's the best toy I've bought in ages. In fact, thinking back over the last two years the full list of interesting new toys have been in the majority from Apple:
MacBook, iMac, PowerMac 12", iPod Nano, iPhone. Only perhaps my Linksys IP phone could be added to that list. This excludes non-tangibles such as Python and Django, two things I will praise in a future post.

I am fascinated by storage on computers. The fact is, I'm fascinated by most things computing and storage is one of those things. Hard disks are noisy, slow and unreliable - yet still today the best mass storage devices we have.

On my home network, incoming data from the Internet first hits my firewall machine. This has been upgraded many times using the spares from other machines, but the storage is a legacy dating back maybe 6 years. It's powered by a single 30GB disk. It's actually been replaced, and even the OS replaced, but as a spare parts machine replacements have never been at the same time: no co-ordinated upgrades. So this is an unRAIDed disk. To compensate, it's fully backed up and can be restored in entirety from tape.

The firewall also runs CCTV and other odd security tasks - not all IP related. I have a pair of spare 120GB disks for it (30GB minus OS doesn't leave much room for video). Strangely these disks were once in the firewall as part of a RAID5 when I needed some extra PCI and IDE slots for a storage array, which was later moved to a more appropriate machine.

The next machine is my primary server. This is by far the most interesting storage machine:
8 disks
3x RAID1 mirrors - /boot & /, /home, music
1x RAID5 (4 disks) - everything else

The first 1GB of the RAID5 disks are actually two RAID1s. That is, using Linux software raid I have the ability to use different levels of RAID on the same physical disks. The upper 300GB portions of the RAID5 disks are the RAID5 section.

On top of the RAID5 is LVM. LVM is truly great. I have 400GB allocated to partitions - /var and /usr etc. 400GB is currently completely unallocated - if I need it for anything, I can bring it online at any moment, either in addition to existing partitions or as new partitions. If I ever run out of space, I can add more physical disks and extend the LVM onto them.

Some of the RAID1 disks backup by replication onto a partition on the LVM. For this I use rsync run nightly in a small script, which also handles backing up and archival of databases.

The root and boot partitions are not on LVM, as I followed general advice. If you lose the LVM for whatever reason, you want to be able to boot the machine to fix it. Especially if you don't have a CDROM drive in the machine to boot a rescue disk.

Each disk in on it's own IDE channel. No slave disks - on IDE, if one disk fails, the other on the same cable will be lost too. I used to split RAID1 over channels - then a failure on one array kills another array. Not good for diagnosing it later.

Moving on: my 1U server.
This has a pair of 20GB disks, in a Linux RAID1. The machine has front access disks so you can remove them without disassembling the machine (and removing it from a rack - it's not on rails). However they are not hot-swap. It's IDE again. IDE is a very poor interface, but for decades the cheapest and most available. The machine has a BIOS based hard/software RAID. It's awful. It is supported only by Windows, FreeBSD and Linux 2.4, and to rebuild the array you have to drop down to the BIOS.

On to my main desktop, one which isn't a Mac:
One 160GB SATA2 disk. Almost empty as all storage is held on the server. There's a lot of temporary files on it and the OS though - the disk was chosen for speed and the size is a reflection of the best price point for a disk used for only temporary files.

Further: Amiga 4000
Two disks, a 40GB and a 4GB, both IDE. It was a 4 and 20GB but the 20GB developed 'noise' - it worked fine since 1999 until mid 2008, and still does, but it's loud. The Amiga runs 24h and needs to be quiet. So I replaced it with a silent 40GB and decided to get rid of the 4GB (dating from 1997). Bah, lots of work - the machine doesn't like the large disk. I used to run a 80GB on it but this is different somehow. So the 4GB boots the basic OS then reboots into a 40GB supporting version of the OS.

Onwards: iMac
250GB SATA2, 3.5". Not interesting. Boots MacOS, works until it fails (no RAID in an iMac). Feels very fast.

Onwards: MacBook
80GB SATA2, 2.5". Same, but is vastly slower than the iMac. the iMac has almost the same spec - same CPU, same memory - but the overall performance is completely different and pretty much entirely down to the disk.

Onwards: Amiga 4000T
36GB and 9GB SCSI2-Ultrawide. Old, tried, tested, and as fast as some new IDE disks I've bought. IDE sucks. SCSI is good (except for the plethora of connectors which all do much the same thing - beyond physical connectors SCSI is almost completely forward and backward compatible).

Boredom is hitting right now. Disks are disks and I have lots, so to continue on with something else...:

Backups: DLT-7000 x2 drives, DLT-IV tapes, all in a robot tape changer unit with 32 tape cells.
This is the coolest piece of hardware I have. A huge, power hungry Dell box (rebranded StorageTek, of course - do Dell make anything?). Lots of tapes. Terabytes of potential storage. Most importantly, a big robot arm that moves tapes around, loads and unloads the tape drives. Very cool to watch.

It's connected to the primary server using high voltage differential SCSI (I didn't know this existed back when I bought the changer, it was called just 'differential SCSI' - cue lots of effort trying to get it working with LVD SCSI...).

Amanda is the software I used to manage it. It does everything for me, now I have it set up. It did take a great deal of configuration though... I have it dump backups to a holding disk overnight on Friday and send it to tape on Saturday (switch on the unit, run the command on the server, wait for the notification that it has finished).

I was hoping to write something interesting in this post. It seems that I haven't. I blame MIT and UC Berkeley, as I'm watching some of their physics lectures on YouTube while I (try to) write this.