Technological Wanderings Travels in code, databases and networks

Gentoo has switched over to Python 2.6 from 2.5. This is good - I develop on 2.6, but have been deploying to 2.5. I migrated a huge commercial app from 2.4 to 2.6 earlier this year. I like 2.6.

But I didn't noticed that Gentoo had pushed out the update to become the default before merrily answering 'y' to 'emerge -avu world'. Oops.

It took down my Django based website for some time, maybe an hour in total; many hundreds of error 500 emails were received. The site usually receives anything from 5000-8000 hits on a Sunday. Oops.

It doesn't seem possible to avoid downtime with Django when doing a Gentoo portage-based Python upgrade. Perhaps now is the time to cluster the site over a couple of servers. That is, after all, part of what I do for my day job - clustering Django based apps and webapps.

Why is downtime unavoidable? Or at least, from my immediate knowledge?

First you do 'emerge world' - this brings in the new python, and maybe some new python modules.

If you use those modules (it's a live server, everything on there is used) then they will be removed from 2.5 and installed for 2.6 - your 2.5 apps loose access to them. Boom. Solution - pay attention, don't emerge world and emerge only python, so the packages are installed in a controlled way.

Afterwards, you need to run python-update. So even if you manage to emerge python 2.6 okay, you'll still get the remove/reinstall cycles here as python-update reinstalls each package for python 2.6. Boom.

This is far from being a disaster, but got a little exciting as I waited for pygtk to compile on a 256MB Slicehost - it needs a fair bit of memory at one point of the compilation, so I shut down Apache and everything else to reduce swap usage to a mere ~256MB. My word did that get slow.

Snow Leopard was received in the post on the 29th and quickly installed. I say quickly; it took a little less than an hour to upgrade my machine though I am assured a fresh install takes less than 10 minutes.

No unexpected problems after the install: iStat menus don't work, neither does PlugSuit. iStat I was surprised about but PlugSuit is a hack which, had I thought about it, would have been sure it would fail.

First impressions? Exactly the same as for Leopard 10.5: "is that it?"

The only really obvious improvement is Mail, which is now blindingly fast. Seriously, it's nearly as quick as Thor on the Amiga. The problem here is that gpgMail has failed again. When will Apple give us a stable API to add these features in? PGP / GPG is essential!

Expose is better. Spotlight seems quicker, but I've not tested in detail as I've had little use of it yet. In general, the whole machine does seem quicker overall. I really don't think I've had any pauses or waits for something to react while using 10.6 today, in situations where 10.5 would leave me waiting for Finder to open a directory or crawl through for preview thumbnails.

I expect Snow Leopard to mature over the coming months as software is released and upgraded to take advantage of the revamped internals. I really do feel the same about 10.6 as I did for 10.5 (in that I couldn't see the need to upgrade from 10.4) but as 10.5 matured and realised its potential I found Tiger to be less and less attractive.

A while ago I had a CRB check at work. There were servers onsite used to host websites for schools and so, to be safe, everyone was checked with the Criminal Records Bureau. It was more to give customers confidence than for any practical reason.

What sticks in my mind was this:

I received a copy of the CRB results, and naturally the employer received their copy. Mine letter and results were of course marked 'confidential'. The odd part was that the employer wanted to have a photocopy of my results in addition to what they received - they are not supposed to do that. I remarked to the HR person at the time that it was a bit odd, but I wasn't opposed to letting them have a look at my "private and confidential" documents.

The next day I receive an email from the boss. I'm not a good communicator perhaps, as by the time my quip had been relayed to his ears it'd changed its meaning and he'd taken some offence.

The email stated why the CRB was done, why it was important, why they wanted it, etc. etc. - missing the point of my light-hearted concern that they wanted copies of my documents, labelled explicitly as for my eyes only, when they had received their own versions.

The best part was that in the email I was assured that the CRB check was completely confidential. To highlight this, he extracted details from my results and placed them in the email (fair enough that the 'details' were that there was nothing on CRB).

http://news.bbc.co.uk/1/hi/england/tyne/8115216.stm

A group of passengers refuse to board a plane when told that a plane should be balanced when flying.

What the story fails to state is that they were all avionic engineer.. oh, no they weren't? They were all physics profess... no? They were all pilots? No?

Ah ain't getting on no dahm pe-lane.

Silly pilot. Planes go faster when all the weight is in the nose. Ask any boy racer - you have to put the back seats down in your Saxo for best performance.

iPhone tethering uses standard Bluetooth protocols, so you can tether from Linux. I'm not aware of how you'd do this over USB - for my situation (using the iPhone as connectivity for my CarPC) Bluetooth is ideal.

First install Bluetooth utilities on Linux and ensure you are running a kernel with all Bluetooth support compiled in or otherwise available as modules.
Make sure you have something on Linux that'll prompt you for the PIN - kbluetooth or similar for graphical desktops. Pair the phone and the computer - I did this from the iPhone end - and enter the same PIN on each end. Ensure tethering is enabled and switched on on the iPhone.

You can get the iPhone's address on Bluetooth using 'hcitool scan' while the iPhone is in discovery mode.

On Linux:

modprobe bnep
pand --connect [IPHONE_ADDRESS] --service NAP --autozap


You should now see a new interface connected:

ifconfig bnep0


Depending on your Linux distribution, it may automatically run DHCP (this is how the iPhone gives your computer a NAT'd IP). Otherwise:

dhclient bnep0


(or whatever you use normally)

All done, have fun with Internet everywhere :-)

There is a famous list among developers called The Joel Test: http://www.joelonsoftware.com/articles/fog0000000043.html

It gives a guide to how best to develop software. I don't agree with everything - e.g. writing code during an interview, but it's a good guide to best practise in keeping your software quality up and your developers happy.

This list is all well and good if you're building a development team, or looking to work with one. Do they score 10? I'd like a job. Do they score 3? Let's keep looking...

But what about appraising somewhere you already work? I found a question on stackoverflow.com (my new favourite website) which appealed to the streak of good old British negativity in me - The Anti-Joel Test. What aspects of what you see around you every day contribute to worst-practise, making development ever more difficult?

Take a look here: http://stackoverflow.com/questions/901320/anti-joel-test
There are some jaded, real-world-affected people posting in there..

I've compiled a list of my favourites, based on past and present experiences.

Do you do your development work on the live production server?

Many, many small web development companies do just this. A client has brought me onto a project where the original developers are doing exactly this, and they are a London based company with a very impressive website and portfolio! My first suggestion was a separate development environment per developer which I'm glad to say that they've taken up.

Is the chief technology officer an accountant?

This could work, if you're developing accounting software and they listened to advice.

Is there a dress code which includes a suit and tie?

I sat at a desk for years like this (though without a tie), as a developer and rarely seeing clients. Now I feel I have to wear a shirt and smart trousers to work - even when all around me are in jeans and t-shirt or the equivalent. What really drove my negativity up was when the on-site engineers at the company were told to start wearing ties. These are the people crawling around factory floors to change the cheap RAM in the machine we sold them last year.

Is StackOverflow blocked at work?

In general, and I've mentioned this before, don't block websites. If you have people looking at facebook all day, sort them out. Don't start down the slippery slope.

Would management view time off to speak at a conference or other technical gathering relevant to your work as suspicious and/or unconditionally refuse it?

Do developers feel frustrated or stymied by managerial/organizational problems more often than technical ones?

Does your source control consist of physically backing up your source tree into a source.backup.n directory every time you want to make too many changes?

It can work; the developer of Postfix reportedly works much like this - but then he also mentioned printing off submitted diffs and scouring them character by character to ensure quality. I started using RCS back in the mid-90s when I was still in school.

Is the development team driven and controlled by sales people?

There must always be an aspect of this, since you're producing something for a customer and the sales person is your interface to them. But there are times when this is taken to extremes - at a previous job, the sales people started to get a little too powerful and decided that they would no longer be sales people, but something like 'customer requirements providers' or some such rubbish. The effect of this was that they became project managers as well as sales, even with no project management experience and certainly no time to do it. And add to that a complete lack of understanding of development. And being in a different room with little interaction with developers. And the conflict of interest. There were a number of high profile failures due to this.

Do you usualy promise to create the documentation after the software is finished?

Or, is documentation not provided as part of the quote. Does the customer sign off the quote, then later demand documentation? Do they then demand that documentation provided by email is not enough and that they must have someone onsite to hand over a printed copy?

Do you hire cheap people because they cost less?

Cheap people are the driving force behind many companies. "The work was poor, but never mind, at least he is cheap."

Are strategic alliances (instead of technical merrits) the main argument for resp. against the use of a technology?

So much pain... F-Secure have brilliant UK sales people. They really got their hooks in with one sales team I worked with - even though we had constant problems with the product. One of the problems is incredibly high system requirements, but our sales people were over the moon with this since they sold memory (cheap stuff which failed every six months) and whole network and deployment upgrades off the back of it.

Is the work environment noisy making it difficult to concentrate?

Do they expect tier 1 support in addition to your programming duties?

Do they fail to provide adequate development hard/software?

Do they fail to grant developers admin rights to their own box?

Is refactoring discouraged?

There is an aspect of not changing a customer's codebase without payment.

Do developers have to account for their time in small increments?

Every 15 minutes is usual. Made even more difficult when between writing lines of code you need to take support calls. Do you account for 15 one minute calls?

Has marketing a higher budget than development?

Is the Testing Plan, for a given piece of software, developed after the software itself is developed?

Tests? What are they?

Are testing plans discouraged?

Do I need to work weekends just before a big release?

How about re-implementing a product which has had three years solid development using a different framework, and setting a two week deadline?

I've been working on some software for a while. The idea was a search engine which answered questions directly, rather than sending you to a page with the answer.

I'm giving up now and hope to make some cashmoney using the algorithms elsewhere, since I've just had a play with Wolfram|Alpha. The primary test which I have been basing my idea on was asking what the weather is today and getting a straight answer. Wolfram isn't perfect but it passes the test.

As an aside, I asked it to compare the population and GDP of the UK and France. Try it yourself and look at how France has much less money per head than us. Then ask yourself why France is capable of building planes, spacecraft, rockets, trains, nukes, subs, helicopters and just about everything non-trivial that the UK now is only capable of buying from abroad. A little bit of politics there but since the MP's allowances have been made public I've been wondering where the supposed vast riches of the nation are going - not to imply that it all goes on moats and biscuits, of course; the total allowances for every MP might only pay for a feasibility study to put some traffic lights on a B-road in rural Wales.

mod_wsgi in daemon mode is a massive improvement over both mod_python and mod_wsgi in embedded mode.

mod_python was causing Apache2 to use the full memory and swap of my 256MB Slicehost server; that's around 700MB total commit. The same happened with mod_wsgi initially. The reason being that for each Apache process launched, the whole of my application's footprint was fetched into memory.

Partial solutions are to reduce the number of Apache processes, which bring its own problems when you don't have enough to serve your users.

mod_wsgi in daemon mode solves this by launching separate python-specific processes to look after your code, leaving Apache to look after incoming requests and static files. A single python process can look after a hundred Apaches.

It should have only taken a few minutes to configure daemon mode over embedded, but it was not obvious to me how to start the daemon. I spent a lot of time researching how to do it until I finally read the correct part of the documentation: mod_wsgi actually looks after this for you; there's no separate daemon to start.

Here's my configuration:

WSGIDaemonProcess unique_process_id processes=2 threads=4 maximum-requests=100 display-name=%{GROUP}
WSGIScriptAlias / /var/www/website/wsgi.py
WSGIProcessGroup unique_process_id

That's all you need over the daemon mode. There are a huge number of configuration options to tweak the way it runs which I won't go into here, but you can make it do pretty much whatever you need.

In terms of performance, I suspect that a daemon won't be as fast as a python-per-request, but since embedded in my experience eventually leads to swapping, daemon modes wins out overall.

What is it that you want your developers to do? I suspect that you want them to write code. The smaller the company, the more additional tasks there may be. But fundamentally, it's about writing code.

So how do you as an employer or manager ensure that they can do this most effectively?

There is much to be balanced. Lets try to dissect the options, paths and consequences, ranging from make-or-break to petty annoyances.

General office environment noise

The office must be reasonably quiet, but not silent.

Bad: sales people yelling down phones. The words enter the consciousness of the programmer, causing them to name variables after whatever product the salesbod is selling. The lies, ignorance and half truths also affect their state of mind: my thoughts are still swimming with a statement made by a salesbod about four years ago, regarding how Linux was so much easier to hack than Windows since you can see the source code.

Total quietness is bad too, at least in shared offices - you don't want to hear every creak of the office chairs, every slurp of coffee. Create some ambient noise with some desktop computers or aircon.

The computer

Programmers really do tend to need very good computers. Yes, even when hacking PHP/HTML. It's also a good idea to allow the programmer to use their platform of choice. If you're writing cross platform code, as everyone outside of Microsoft now does, then platform should not matter.

This, however, is much more than allowing your coder to have a Mac or Linux box. It must permeate the very fabric of your IT infrastructure.
I once tried to run Linux within a wholly Microsoft office. It did not go well. There is no reason for it not to have worked, but every aspect of the network fought against the alien invader. Exchange would not talk to open clients, network shares defaulted to settings which resulted in poor performance or even lack of connection with Samba, and more.
It's not just the infrastructure but also choice of software, as I'll come on to in a moment. Case in point: Microsoft Dynamics CRM. This truly awful, bottom of the barrel CRM software is a web application - however, it's a web application which runs only in Internet Explorer. Not only must you only connect from Windows, but you're locked into using IE no matter what your browser preference.

The computer must work for the developer. It must not be a continual uphill struggle to get anything done.

Software - development

You can code using only a text editor, but while that's fun for a bit you'll need more to keep developers happy and to improve productivity. I spent many years with only a text editor, even one which doesn't colourise code. You can learn much from coding in that way, but a business is less about learning and more about doing - get tools that help with the doing.

Thankfully much of the best software development tools are free for most development environments, but don't be afraid to spend. Example: Trac is quite a wonderful tool for ticketing and viewing source and changes in a Subversion repository. But if you have a few developers and the budget, then Atlassian products are a much better proposition.
Don't be afraid to spend if your company is profitable. Especially, don't dither over essentials - if your development environments are contained on VMWare images, ensure that your new Mac developers have a copy of VMWare Fusion. Don't leave them to spend months using every email address they have to continually download expiring trial versions - it will infect their thinking for years to come.

Software - office

You hope that your developers should never need to touch anything but their IDE, but in small companies it will be the case that developers need to raise sales orders, consult the CRM, query the ERP, connect to their email, use the phones, ...

I've touched on this above. Basically if you base your infrastructure on an MSDN subscription then you'll only keep .NET programmers happy. Good luck with that.

Of course, there's the other side to this - many non-programmers will be conditioned to use Windows, Office, and Outlook. Be careful that those not ready to use a non-Exchange mail server are retrained and don't poison the project - it is possible for an incorrectly used Outlook to make it appear that Zimbra is broken and certain types of people will use that to destroy the working atmosphere.

The computer - my home

So, you standardise your company on Windows. You have a Windows Server or five to run the network. You decide to ensure uniformity across your field of view in the office and implement Group Policy from Windows Server. You force a bland image with the company logo as a desktop image onto every Windows client desktop.

Good: no more sci/fi fantasy images. No more car images. No more nakidity.

Bad: your programmer probably had only a black backdrop, visible only for the few seconds that it takes to swap applications. Like one of a million paper cuts, this adds to a general sense of depression.

You go further. You announce that all webpages visited are tracked and monitored, and action will be taken if anything bad is accessed. 'Bad' is not defined, but you announce that you're already tracking people going to job sites. MSN is banned, then allowed, then conversations tracked. I understand this and recognise the business requirements, but you should also realise that it continues to poison the thoughts of those who never took advantage, never used MSN, but did like to check the news pages at lunch time. Especially if implemented before announced (if you announce) that it is happening.

You go further. You start blocking websites. Perhaps you have an employee who spends all day on Facebook. Fine, discipline them. Don't take it out on everyone else. Very especially, don't blanket ban websites for minor causes. Don't ban BBC News because some sales bod keeps watching the boxing results. This can have consequences beyond getting an extra minute or two of sales call time.

My example: every Monday morning I'd listen to Goon Show on BBC7 while trudging through the mindless tasks of the day. The same tasks, each Monday, performed with my eyes closed and my mind switched off. I'd often return and listen to other BBC shows throughout the week. Yes, this used bandwidth - but I took advantage of my position of being able to see MRTG graphs of utilisation to ensure that this 30kbit/s never impacted the business. As noted, my 5 minute lunch break would consist of browsing the news sites, and BBC News is the one site to rule them all. Imagine the impact on my day if the BBC was blanket banned. The BBC of all things.

Push your developers

Lack of challenge will turn them away. Real programmers want new challenges. New languages. New opportunities to push the boundaries. If the opportunity is there, push them into doing something new. Some will grumble early on, but the rewards from success blot out any and all negative aspects.

Don't push your developers

In non-coding aspects, don't push so hard. Programmers want to be stretched in technology, science, maths and logic. They don't like being needled about things irrelevant to the task in hand until they submit. Time sheets are a good example of this. How closely are you watching?

Get buy-in

Not just from developers, but in general: ask opinions of those involved before implementing policies.
"We're putting a new wall across your office."
"We're changing the CPU in your computer."
It gets difficult when you start scaling the numbers, but for a team of less than 10 its not difficult.

The developer abstraction layer

Read: http://www.joelonsoftware.com/articles/DevelopmentAbstraction.html

Some will want to be involved in raising invoices, the sales process, etc. etc. - these people may well become managers. Most find this a distraction.

Don't insult or threaten

Here's another general point. No matter who you employ, unless they do something incredibly stupid then you really shouldn't tell them their work is awful (at least when done to spec), or indeed threaten to kill them for asking where the printer paper is. Just good manners.

http://www.dailypost.co.uk/news/north-wales-news/2009/04/11/anger-at-nor...

A parent commenting on a mobile phone mast on a school:
“And whose bright idea was it to place a mast on a school? It’s disgraceful as there is still a lot of uncertainty around the safety of these antennas.”

Exactly how much uncertainty is there regarding the electromagnetic spectrum?

Should we just give up, go back to stone tools? Oh crap, but stones emit radiation... Wood tools? No, that would emit carbon. No tools? But we're surrounded by rock and soil and plant and animals, all emitting poisons and radiation and evil! And the sun! My word, how much radiation comes off the sun! More than earth has ever produced. Quickly, we must jettison ourselves into deep space, away from sources of energy. Oh no, we would need oxygen - oxygen destroys cells... Bother.