Cycling and Whatnot

Argh! Why is there a cyclist on the road when there’s a perfectly fine cycle track right next to them?

I used to get annoyed at the cyclists: perhaps too lazy or ignorant to switch across to the track. Now I realise I was wrong. It’s the tracks themselves which ought to attract the blame.

There are a few roundabouts near our home. The council has invested goodness-knows many thousands of pounds installing traffic lights at almost every junction. They are bike-activated. The cyclist must negotiate each set of lights in turn, cross to the wrong side of the road, use a prepared track and cross back over several hundred yards later.

It’s quicker to cycle on the road. Despite all the efforts of the council, cyclists do just that: they keep to the road. Mission failed.

As a cyclist I couldn’t bear to break the rules at first. I’d stop, wait and follow the lines all the way through. It took a long time but I felt good for being good. Over time though, my patience slipped and I’d follow the quicker route on the road. I felt a little dirty, irresponsible and guilty, but by goodness it was nice to be able to get somewhere without faffing about.

Nowadays the British highway engineers are increasingly looking across the North Sea to our Dutch neighbours for inspiration. Cycling is a national passion over there, right up with pancakes and windmills.

Having now had the pleasure of nearly a decade in the Netherlands, both as a regular cyclist and a car driver, I can attest to the sheer brilliance of the Dutch cycle network.

It creates an equal footing between the driver and the cyclist: drivers yield to cyclists to offer the latter some continuity; cyclists face penalties for breaking the rules. Both are expected to offer and receive respect in equal doses.

Above all else, the Dutch cycle network offers continuity. The cycle paths, be they dedicated, shared or by road, are uninterrupted. Bar few exceptions, traffic lights exist only where they also exist for motor users – you can pretty well cycle to most places without having to get off.

Similarly, the small matters count a great deal: cycle paths have priority over side road traffic. If a car turns into a side road they must yield to cycle traffic before completing their turn. Compare that to the UK where the convention is squarely in favour of the driver: cyclists on tracks must yield when crossing side-roads.

In my mind, these priorities are all wrong.
In my mind, these priorities are all wrong.

In fact, in a nearby estate, I noticed the shared footpath/cycle track had regular give way signs. The cyclists were being instructed to give way to cars from driveways. This is a completely counter-intuitive priority system and reinforces the driver’s perceived rule of the road.

It comes back to my first point, why would a cyclist willingly use a path where they must yield to every turning car when, by cycling on the road, they are afforded greater priority? The whole system is daft from the outset.

Of course, one could argue the Great British driver is unaccustomed to this. They’ll turn into a side road or driveway assuming priority. Bang goes the cyclist; accidents shoot through the roof. Fear often inhibits progress, and I’d suggest this is one such occasion. We need to be more confident about re-thinking our strategies around cycling.

While I accept the UK roads are often irregular, too narrow or too congested to offer widespread benefits to cyclists, I do hope that more can be made to make an integrated, continuous cycle path system. This autumn, I’m hoping to get my UK bike back in order. There are cycle paths all the way to the nearest two major towns but the priority system is a mess. It’s a genuine dissuasion to ride.

I hope that one day, riding a bike in the UK is as convenient to all road users as they’ve already shown to be in the Netherlands and that our many miles of investment in cycle infrastructure can be fully utilised, rather than ignored and bypassed in the ways that they seem to be currently.

Automating Press Clippings

One of my projects is running a journal devoted to Business Improvement Districts (BIDs) in the UK. These are organisations formed around town & city centres to improve the local environment and trading conditions.

We’ve just started recording mentions of BIDs in the press, which is a useful way to track industry updates. At first, I was running this manually, but it because apparent pretty quickly that this would be unsustainable, so I looked at how to automate as much as possible.

I think I have a solution, dipping into various methods I’ve been meaning to use for a while.

The system starts by receiving an update from Google Alerts on a particular topic. This email normally contains 20-30 links to various worldwide news articles.

Email to script

This was something I’ve used before but the basic gist is that an email address is piped directly to a script. I created a new account on a domain I manage and set it to pipe to a script. In Postfix the virtuals file tells the server that username@domain.tld is mapped to a local alias, let’s say mailcatcher. In aliases, this alias is mapped to a script. The line looks like this:

mailcatcher "|/var/mailscripts/script.php"

Run all the mappers, reload Postfix and the email address is set up. Now, provided that script is executable, the entirety of the email will be sent via STDIN to that script.

Message Queueing

I’ve dabbled but never got too far into message queueing. The high-level idea is that pieces of information (‘messages’) are pushed around and progressively worked on by a number of scripts. These scripts are small – they usually do only one job, but they manipulate that message in some way or do something in response to it. Once a script’s work is done, the result is usually passed on to another. Depending on the complexity of the system, multiple scripts might work on a single queue of items.

The outcome is a chain of incremental processing, much like a factory line. For this task it works extremely well as I can break a fairly complex chain of actions into digestable pieces, so once an email is received the following chain is followed:

Receive email via mail script (above). This adds the whole email contents as a message to a ‘mail’ queue.

A script watches the ‘mail’ queue and takes any new messages. Its job is to find all the links in the email and create one new message for each link. These go into a ‘link’ queue.

Another script on the ‘link’ queue checks the database for known links to see if the link is a duplicate. Any message that isn’t a duplicate goes into the ‘retrieve’ queue.

The ‘retrieve’ script follows the URL and grabs a copy of the page in full. This then goes into the ‘parse’ queue.

The ‘parse’ script parses the HTML and looks for common elements (headline, publish date, source, etc.). Once it has enough information, these elements end up in a message in the ‘publish’ queue.

The ‘publish’ script creates a new database entry for the article and publishes it to WordPress. The article clipping is now live.

There are several good reasons for breaking up a procedural process into separate scripts and queues. For one, it’s very easy to scale up. Parsing might be a great deal harder (require more processing power) than anything else, so could need more machines working on it. Grabbing the contents of a URL is notoriously unreliable, so if a script fails here the queue entry is merely postponed until later when it’ll try again.

Debugging is also trivial. I can simply watch a particularly (/troublesome) message work its way through the system and see exactly where any errors are coming up.

This is also good for development. The parser only knows how to read a pre-defined selection of news pages, so if a new page arrives which can’t be parsed I need to add new rules to the parser. By holding the message in a queue I simply postpone the message until the parser is upgraded and the message can continue through the system.

There are a fair few messaging systems out there which handle this stuff well, but I needed something quick & dirty, so wrote something up to handle this myself. Down the line, I’m planning to move to a more robust, distributed system.

Web Interface

Many of the news articles are rejected because they cover regions we don’t care about (US, New Zealand, Canada, etc.) so I use domain blacklisting to disregard foreign news sources. This isn’t fool-proof. Some news aggregators are international, and might carry articles we care about.

Furthermore, once a UK-based article is picked up, I need to figure out which BID it relates to. Doing a simple search for town names is not enough (‘Rugby’ was pretty popular because of the sport-related mentions on pages), so I look for ‘{town name} business’ and phrases like that. It gets about 75% of the results satisfactorily.

For the rest, I need to manually intervene and provide a BID association myself.

This is the bit I need to work on. Currently the parsers and labelling systems are all hardwired. No good. It’s manageable for now, but in the near future I’m going to need a web interface to handle all the messages that don’t get through.

Overview

The whole system is running automatically and probably saving me at least an hour a day. It took around 8 hours to build so I’m happy with the returns. Having now put together a basic framework I expect to be able to use this for other tools as well.

Automation was one of the key lessons I learnt a decade ago with Blogwise, a blog directory that was easily attracting 100+ new submissions per day. The labour involved in handling submissions was phenomenal and automation scaled up in ways labourers could not.

It’s now something I advise others to do as well, particularly with repetitive tasks. A relatively small development investment can replace many man-hours of activity, and the economies of scale are often much more impressive.

Adapting your Design

Producing a web design that adapts to screen size isn’t just about saying “ooh – you’re on a desktop, I’m going to give you a full-width page.”

On the desktop, windows can be resized. They can be full-screen, half-screen or any other shape. I can easily hit Windows key and the right-arrow and *bam* the browser is now taking up half the screen. Great for multi-tasking, but a pain in the backside if the web designer decided their creation had to be seen full-screen.

Assuming the user will always oblige by running your website full-screen is dangerous. Thankfully truly responsive websites are taking over.

EU VAT Rules after 1 January 2015

I’ve been following Rachel Andrew’s post on EU VAT rules with interest. New laws come into play on the 1st January 2015 which shake up the charging of VAT in various EU Member States.

VAT is charged at the Place of Supply, and it’s this definition which appears to be changing. For electronic services at least, instead of the supplier charging VAT at their domestic rate, they will effectively charge it at the customer’s national rate.

If I (a UK VAT-registered supplier) provide you (a consumer) with a piece of software and you live in the Netherlands, I would apply a VAT charge of 20% – the UK rate.

From 1 January, I would charge 21%, the Dutch rate.

On the face of it, this doesn’t seem so unreasonable. Otherwise, companies will (and do) flock to low-VAT companies to sell their electronic wares. However, there is also an apparent requirement to register in the customer’s country as well, so I (the supplier) would need to be registered for VAT in the UK, in NL, and any other applicable country I happen to supply to.

That’s a pain in the backside for any small business.

However, there does seem to be an alternative solution, called MOSS. This appears to act as a gateway for cross-EU VAT handling, and saves the supplier having to register in each EU country.

The way I’m reading this (usual disclaimer: I am not an accountant) suggests that the government are pushing MOSS quite firmly as the de facto way of trading in Europe for small businesses. Instead of filing with each member state, UK suppliers will use the MOSS system to indicate their supplies to non-UK EU countries. Much simpler, except:

  • Place of Supply is still with the customer, so the supplier needs to apply the local VAT rate accordingly.
  • MOSS requires 10 years’ of records to be kept. VAT only required six.
  • MOSS returns have a shorter deadline than standard UK returns (20 days).

Whichever route is chosen, suppliers will need to make sure they have plans in place to support the new VAT rules come 1 January. Websites need to be updated, procedures need to be reviewed and reminders need to be set.

It’s an interesting development and one which affects a great number of small businesses. Rachel has set up a GitHub page to collect and distribute further information. If you are a EU-trading electronic business I’d strongly suggest a look.

What is Legal Tender?

After a recent trip to Northern Ireland I ended up with a Bank of Ulster five pound note. In N.I. a visitor needs to be careful which cash machinulster five pointes they use. Some dish out Bank of England notes; others issue local banknotes. They have the same value, but try using an N.I. note in England and you’re out of luck: They’re rarely accepted.

At uni, a Scottish friend would regularly get himself in a bit of a trouble by walking into shops (in Southampton) and trying to spend Scottish banknotes. On refusal, he would argue that they’re legal tender – so should be acceptable in England. He never got very far.

Now, with a banknote from Northern Ireland in hand, I wondered whether these notes are indeed valid for use in England.

It turns out that no, they’re not. As far as I can tell, legal tender is a very specific description which refers to the ability to use certain assets in repayment of debt. If you end up with a debt of some kind, you can use legal tender to settle the debt and the recipient (the debtor) cannot refuse.

With this in mind, my friend at the shop has two issues. First, there is no debt to settle. The shop can simply refuse to exchange the goods for the money so any legal tender argument is irrelevant, since there’s no debt. They can easily turn him away.

Restaurants are a slightly different matter. If you eat your meal then pay afterwards, a debt has been created (you certainly owe them for the meal) so my understanding is they would be obliged to accept legal tender.

On to the second point. Scottish banknotes won’t work anyway since they’re not legal tender in England. In fact, it turns out they’re not even legal tender in Scotland. There are specific combinations of coins and notes which are valid in each country:

In Scotland, only coins are valid legal tender. Scottish banknotes have no legal tender status anywhere. As far as I can tell, this is also true for N.I.-issued notes.

In England and Wales, banknotes issued by the Bank of England are legal tender (provided they’re current – old notes are no longer valid). You can settle any debt with banknotes.

Coins are legal tender throughout the whole United Kingdom, although there are restrictions. You can only settle a debt up to £5 if you only have 10p coins, for instance.

Legal tender is a very specific definition, and seems to be the subject of misunderstanding. My Bank of Ulster five pound note means very little in England, but the novelty of having it hasn’t worn off yet, so I’ll be keeping it for now.

As always, I am not a lawyer, currency expert or accountant so check with a pro before you attempt to settle your debts…

Email Forwarding & SPF

mailA friend emailed me the other day to ask why their hosting company have announced they will stop forwarding.

Roughly speaking, they have an email address joe@customdomain.com which is hosted by said company. This forwards to their Gmail address joespersonalaccount@gmail.com [these are, if you hadn’t guessed, not their real emails].

The company has stated that they will no longer allow forwarding of emails from the custom domain to an external server. Why? my friend asked. It is, after all, a quite normal thing to do with domains.

As far as I can tell – it looks like the reason is down to the way popular mail services filter spam. Google’s GMail, for example, uses a thing called SPF. This is a mechanism that checks whether emails came from a legitimate source.

Let’s say bob@bigcorp.com is sending my friend Joe an email, but they use his Gmail account directly – this is the simplest path:

1. Bob writes the email and sends it. The mail server at bigcorp.com looks up the mail server handling Gmail.com accounts and connects to it.

2. The mail server at Gmail.com receives the email and puts it in Joe’s inbox.

Now, the mail server at Gmail.com is applying SPF. This looks up the public record for bigcorp.com to see which mail servers have authority to send from that address. The mail server that sent the mail to Gmail is indeed legitimate, so it’s allowed through.

Now, Bob sends another email, this time to joe@customdomain.com

1. Bob writes the email. The bigcorp mail server looks up customdomain.com, finds its mail server and connects to it.

2. The customdomain.com mail server realises it’s supposed to forward mail on, so once it has the email it finds the gmail.com mail server and resends the email there.

3. The gmail.com mail server receives the email and puts it in Joe’s inbox.

So, same as last time, the gmail.com server applies SPF. However, this email didn’t come directly from bigcorp.com, it came via customdomain.com. This isn’t on bigcorp’s list of authorised servers, so as far as Gmail is concerned this email could have been forged. It’s quite possible the email will end up in the junk folder.

Worse still, if lots of emails come via customdomain.com Gmail might start thinking the entire domain is spammy. This is partly why I suspect the hosting company want to avoid forwarding. Ending up on spam blacklists is a costly, damaging affair.

This might appear like a weakness of SPF, but there is a fairly reasonable solution to this. I use Google Apps for Work (i.e. I pay for Gmail) and as part of my configuration I can tell Gmail which mail servers are trustworthy.

If Joe did this, he would add his mail server for customdomain.com to the trusted list. Now, when Gmail checks where an email is coming from it will see that customdomain.com is a trusted server, so will look further up the history to the previous mail server. In this case it’ll be bigcorp.com, which is valid, and the mail will be allowed through.

Unfortunately I’m not sure this is available on the freebie Gmail server, nor whether other services support it.

The hosting company didn’t give their reasons for limiting the feature, but this seems like a pretty likely (and logical) reason.

Sidenote: there’s also a bit on Wikipedia about rewriting the envelope sender. Looks a little hacky, but might be a viable alternative for those without whitelisting.

So You Wanna Be an IT Contractor?

As part of my work I mentor and support people looking to run their own businesses, or who want to become an IT contractor. This means I get a fair few people come up to me with excitement, ‘I’ve decided – I want to go contracting.

Disclaimer: I am no accountant, nor am I a lawyer. The calculations below are rough and based on basic research, and there are big differences between sole trading and running Limited companies, so please take appropriate advice.

Update: Self-employed people might be entitled to Statutory Pay for sick & parental leave. For specific advice the above rules apply – speak with an expert about your specific circumstances.

Money

The most common reason is money. Employees see contractors come in, they find out how much these people are paid, and envy creeps in - They’re being paid so much more than me.

The wannabe forgets some important points:

1. The price you hear is most likely the pre-tax price. After all the earnings contractors are going to have to pay National Insurance, employee tax and (in some circumstances) employer NI. They’re also going to have to pay more into pensions, something your employer would likely otherwise help with. Employers usually pay 13.8% NI on top of the employee’s contributions. That’s going to be your problem now.

2. There’s no sick pay. No holiday entitlement. By my rough calculation, if you take just the statutory holiday entitlement of 5.6 weeks with bank holidays rolled into that, that’s nearly 10% of your available working time. To reverse that out, you need to be earning 7.6% more than a full-time salaried person just to cover statutory holidays. That might not sound a lot, but it all adds up – and this is just the bare minimum.

3. Thinking of starting a family? No maternity or paternity pay for you.

4. Insurance. You’re going to need to pay that yourself. Accountants’ fees. Yup, that’s your responsibility as well. Legal costs – you’re going to want a solicitor to check your contracts.

5. Finding work. A good number of people turn to contracting because they’ve found an opportunity. This is fine and dandy for the first few months but you’ll quickly realise why businesses like contractors. They’re easy to get rid of.

Come the end of your contract you’ll need to try and renew it, or find another gig. You need to become a salesperson. For all the time you’re selling; trying to find a new job, you’re not earning. If you’re in a full-time contract that means looking around in your spare time or sneakily getting calls in at lunchtime. If that contract ends and you’ve not found a new one, you will earn nothing, zero, zilch. You get no redundancy pay, and termination clauses on your contracts might bring an unexpected early end to what you thought was a cushty little job.

To summarise: contractors get paid more because they have to spend more. Their liabilities are far greater than employees’. The rate they’re paid is not even comparable to your gross salary, because they’ll have employer liabilities and costs as well. The risks are high, and you need to do your sums carefully before becoming a contractor. It’s likely less lucrative than you think.

Freedom

The second most common reason is freedom. I want to be my own boss.

Contractors aren’t bosses. They’re expendable labourers. Put some effort into your personal brand (yes – marketing) and you can become a consultant, usually with a better rate and slightly higher esteem among your clients, but you’re going to have to work for it and have a higher work turnover.

I might pick up the contractors vs. consultants issue on a separate post, since it’s an issue close to my heart. But it’s not for now.

Simply put, if you want to be your own boss, be very careful about how you present yourself. Contractors fill gaps, have bosses and generally get told what to do. Consultants come in (usually after stuff hits the fan) and get told to figure out how to fix things. Both have limited autonomy, and your decisions can be vetoed, overruled or simply ignored.

So – you still wanna be an IT Contractor?

iOS 8 to thwart Wifi tracking

Apple have recently announced the latest version of their mobile operating system, iOS 8, including changes which might affect tracking systems used in supermarkets and shopping centres.

In the last few years a number of companies have sprung up with the offer of tracking customers for the benefit of analytics. Asda are doing this in the UK, as are shopping centres and plenty of US companies.

What is wifi tracking?

The basic principle is that your smartphone is constantly looking for wifi networks to connect to and – in doing so – sends out a unique identifier (a “MAC address”) which can be used to trace you. With enough network access points it becomes possible to work out where somebody is, based on which points their phone tries to connect to.

The outcome of this data is pretty beneficial for shop owners, estate managers and so on. It tells them how people move around their stores, which areas are hot-spots and which others might be ideal places for promotions.

Apple’s announcement is short on detail, but it appears that they are preparing to scramble the MAC address when the phone tries to search for wifi signal, making it more difficult to track movements between one place and another. The key is how often they do this – if the MAC address is scrambled once a day (for example) the drawbacks will only be for long-term tracking, i.e. watching for repeat visits. If it changes every few seconds the entire capability is gone.

Arguably wifi tracking is an opportunistic service anyway, making use of an artefact of technology that was never designed for tracking purposes. Its utility, however, has been enormous and companies will certainly miss this data if it goes.

Privacy

Are Apple doing this for privacy’s sake? It would seem so. In the right hands, Wifi tracking is anonymous. Try hard enough and – in the right circumstances – it becomes possible to identify people. Most companies won’t try to do this for fear of running foul of data protection laws, but that won’t stop everybody.

In any case, this won’t prevent opt-in tracking such as the Asda scheme mentioned earlier. The offer of free wifi comes with strings attached – they can track and identify you, and once connected to a ‘known’ wifi hotspot your phone will no longer scramble its MAC address, giving the company full access to your individual movements.

Direct selling need not be an option either – shops and districts offering free wifi can still benefit from tracking, but this will likely be forced into terms and conditions. Whether they use that to gather personal information and market to individuals is entirely a question for them and their visitors (the latter always has the option to decline the invitation).

That might see the opening of more free wifi hotspots, as companies will need to offer this to retain the all-important MAC address. I’m not sure if Apple see things this way as well, but it’s certainly an interesting possibility.

The Google/EU Search Removal Issue

Google are starting to remove results when searching for individuals, if that individual requests – provided the result is no longer relevant.

It seems to me that this is simply a logical extension of the Data Protection act – here are the principles that underline the UK act. In particular, take a look at parts 3, 4 & 5. Google, having been established as a data controller for individuals, are obliged to keep the data they hold about individuals proportional and relevant.

That’s also – surely – Google’s goal: to have relevant results, which includes prioritising accuracy and timeliness. Therefore, is the Removal Form simply achieving something Google’s algorithms haven’t been able to do for themselves?

 

Not Smart Enough

We’re still getting post for a former householder who moved out ten years ago (plus daily post for previous owners, 1yr on). Surely they’ve noticed either the lack of response, or my “return to sender” messages aren’t getting through. The former owners had a redirect for a while as well, so the data’s in the system. It’s tiring and wasteful on all parts.

Tom Morris has a long list of things Google thinks he is interested in – except he’s not (he really doesn’t like country music!).

Dave Winer’s dad would be surprised if he got an iPad – as recommended by Apple for Father’s Day – because he passed away several years ago. “With these companies doing such a great job of Big Data, you’d think they could get a clue about this one.”Â