The last subway line in Japan

On December 6, 2015, the Sendai Subway’s Tōzai (East-West) Line opened. The second line of a provincial Japanese city, this event passed entirely under the radar of the Western media and even Japanese coverage faithfully followed the dulcet tones of the press release (時短 jitan, faster! 楽乗 rakunori, fun to ride!), a few pausing to note that the completion of the line had been delayed by the 2011 Tōhoku Earthquake. Almost all stories approvingly added that the line uses linear motor technology, the meaning of which is so unclear to the average Japanese that it’s regularly confused with magnetic levitation, but which imparts the sheen of futuristic technology so necessary for a large Japanese infrastructure project. A future this or any other transport technology lacks in Japan, for the Tōzai line is almost certainly the last subway line Japan will ever build.

This seems a bold claim to make, but the Future Railway DB and Wikipedia agree: in the entire archipelago of 126 million, there are neither any new subways under construction nor any even being seriously planned. Not just subways, mind you; but new railway lines of any kind, with the notable exception of three that we’ll get to later.

So does Japan need more lines? Arguably not, and it’s certainly had a good run. In the 88 years that elapsed from the opening of Tokyo’s Ginza Line in December 1927 to today, 9 Japanese cities drilled, dug and blasted 750 kilometers of mostly-underground subways, with another 6 opting for more exotic options like monorails. In central Tokyo alone, in the ten years since my first visit in 1997, great big slabs of the Namboku, Ō-Edo and Fukutoshin opened up, not to mention the all-new suburban Rinkai, Nippori-Toneri, Minato Mirai and Yokohama Green lines.

And then the Leviathan of Japanese public works, grown fat from decades of suckling on the teat of the taxpayer, finally ran out of the inertia that had sustained it all the way from the bursting of the Bubble in 1991 to the 2008 financial crisis. It ground to a shuddering halt and lay still, and even the twin shocks of Abenomics and Tokyo winning the 2020 Olympics, applied to the heap like defibrillator paddles, threw up only airy bubbles of froth with next to no hope of realization.

The reason why Japan is no longer building lines is twinfold:

Japan’s population peaked in 2010 and is set to decline at -0.7%/year, a figure that masks the gaping trench between the more or less stable cities (although even Tokyo will tip into decline in 2020) and the rapidly collapsing countryside, where some districts have already lost over 90% of their peak population.
Japan’s public debt continues to soar into the stratosphere, with this year’s burden projected to hit 246% of GDP.

It’s a simple, crushing equation: ever more debt to pay back, ever less people to pay it. In Shrinking-Population Economics, Professor Matsutani Akihiko of Japan’s National Graduate Institute for Policy Studies computes that by 2023, Japan’s bill for infrastructure maintenance will exceed its total budget for capital investment.

Matsutani’s suggested cure is shutting down excess capacity to free up scarce labor, paring back investment, increasing wages and freezing taxes. Under Abe, Japan has done precisely the opposite, jacking up taxes and encouraging companies to spend, spend, spend on the three prestige rail projects still under way:

In 2016, the first stage of the Hokkaidō Shinkansen bullet train to Hokkaido, Japan’s most rapidly depopulating island, opens at a cost of ¥1.5 trillion (US$12 billion).
By 2022, the Hokuriku Shinkansen will creep down Honshu’s already rattlingly empty Japan Sea coast from Kanazawa to Tsuruga (pop. 68,000) at a cost of ¥700 billion (US$6b), plus another ¥1.4 trillion if they complete the loop back to Kyoto.
Last but certainly not least, the crowning lunacy of them all, the Linear Chūō Shinkansen (there’s that word again!), which is basically 500 kilometres of tunnel duplicating the existing Tokaidō Shinkansen route between Tokyo and Osaka, only lined with superconducting magnets allowing speeds up to 505 km/h and built at a staggering-even-for-Japan cost of ¥9 trillion (US$72 billion). The line is to be completed by 2045 and hopes to generate a total of ¥5 trillion in profit in the next 50 years, meaning that if all goes according to plan, by 2095 they will have paid back half the construction cost. Now that is planning for the future.

Toy train, Railway Museum, Saitama, Japan Back in reality, Japan is simultaneously embarking on the equally mammoth task of dismantling its railways; not just the once-extensive rural network, which as elsewhere has been comprehensively obsoleted by cars, but urban transport akin to those recently-built subways. Over half of Japan’s railways operate at a loss, including virtually all of those serving less than 2000 passengers/km/day, an increasingly unachievable number when ridership has falling faster than the population since 1995.

The first casualty was the Peachliner in 2006, a once-fancy automated line built to service an exurb of a Nagoyan suburb that carried only 6% of the people originally projected (Shinkansen planners, take heed) with operating costs that equalled rebuilding it from scratch every three years. Next in line may be the Linimo, Japan’s sole operating (slow-speed) maglev line, which stops running when there’s a gust of wind and whose finances, losing ¥1.9 billion/year or nearly $1000 per daily passenger, are on equally shaky ground.

One rural railway managed to resurrect its sagging fortunes for a while by naming a cat as its stationmaster, but the rest won’t be so lucky. It’s time, long past time, to let the zombies die and invest Japan’s dwindling money and manpower elsewhere.

Crashes only on Wednesdays

Sit back and grab a drink: it’s time for the story of one of my favorite bugs.

‘Twas my very first IT job, a software developer summer internship at a manufacturer of very serious medical equipment, primarily anesthesia delivery systems and hospital patient monitors. The latter are the boxes you see bleeping next to bed-ridden patients, drawing graphs of pulse, blood pressure, respiration etc and summoning nurses when they flatline. The office was full of 2-meter cylinders of laughing gas, luxuriously bearded gurus expert in embedded systems, and entire rooms devoted to the documentation needed for the FDA certification of various devices. Hushed voices still spoke of a bug that had slipped through testing ten years ago in one piece of anesthesia kit and caused the machine to reset in the middle of surgery, and needless to say a teenage greenhorn like myself was kept well clear of all production systems.

Instead I was given a plum prototyping project to test out the hot buzzwords of 1997: build a C++ server that listens to the serial port of a patient monitor and dumps a selection of interesting events to a SQL Server database, then send the data via CORBA to a Java applet, so doctors, relatives etc could see how the patient was doing via the Internet, with both the ability to see realtime data and browse previous records. Spiffy! Especially since I had zero practical experience in any of these languages or systems!

After a few weeks of butcherous hackery, much of it spent on inhaling the migraine-inducing Visibroker ORB manual and squishing utterly mundane type conversion bugs, I had my “Simpson” system more or less running, with “Homer” recording and serving the data and “Bart” displaying it. Along the way I had learned out that CORBA was hopelessly overcomplicated, that AWT was painful (GridBagLayouts, brr), that applets were slow as molasses, and that Java otherwise seemed like a pretty decent language. But there was a niggling bug: every now and then, the C++ server would crash hard, and I set about figuring why.

Since patient monitor whose output I was listening to was in another cubicle, I had done most of my development and testing using the handy “Demo” mode where it happily played back simulated cardiac arrests and such in a loop, and as far as I could tell my server never crashed while using this. It did sometimes crash when I or somebody else twiddled with the controls manually, especially during live demos, but for the life of me I couldn’t work out a way to make it crash repeatably, no matter what I did. I logged all events to disk to try to find out what had happened just before a crash, but slow and careful manual replication of the exact preceding event sequence (“set filter to X, twist control knob exactly three notches to the right, press button…”), complete with running back and forth because I couldn’t see my PC’s logs when using the patient monitor, failed to elicit a crash. Whatever “the evil event”, as I took to calling it, was, it seemed to cause crashes while eluding all logging. Was there some sort of serial I/O or hardware issue corrupting the events? Were cosmic rays flipping bits in my PC?!

After several weeks of frustration, where entire days devoted to experimentation had produced no results, I ended up basically adding printf statements to every single line between receiving the event from the serial port and writing it in the database… and in the process, as I revisited every line of that code, a sinking realization dawned on me.

When setting up the database schema, in a misguided attempt at saving space, I had made the amateur mistake of using the event timestamp as the primary key — so if two events happened to come in at the same millisecond, the DB would throw a unique key constraint violation. I noticed this early on, but determined that this only seemed to happen in unusual and uninteresting circumstances like when futzing about in the monitor’s internal configuration settings, so I added a catch block that logged a warning and continued on its merry way.

But. This was old-school code, and the log string was written C-style into a fixed-length 80-character buffer. While the length of the unique key violation message itself was invariable, the log timestamp happened to be formatted so that it used the long-form English weekday name (%E), so the output looked like “Monday, July 17, 1997, 10:38:47.123”. Turns out English weekday names have an interesting property:

Name	Length
Sunday	6
Monday	6
Friday	6
Tuesday	7
Thursday	8
Saturday	8
Wednesday	9

Caught on yet? On Wednesdays, and only on Wednesdays, if somebody manually twiddled certain bits in the monitor settings in a certain way, two events would occur during the same millisecond and cause the DB to throw an exception, and the error message that logged this would be exactly 81 bytes long including the null terminator, overflowing the 80-char buffer and causing the program to crash!

Ever since, I’ve made sure to use a dedicated and preferably auto-incremented integer ID as the primary key of every database table I need, and to log everything in ISO format, YYYY-MM-DD, no weekdays needed. And over the years, I’ve learned that no matter how seemingly random and unpredictable the behavior of the bug, there’s always a logical explanation for it that you can find if you dig deep enough; that “unrelated” errors rarely are; and that it’s almost certainly your own damn fault.

Update: This story is now available in Russian! Когда программа падает только по средам

The Opal or Not Report: The hidden $98 (or $54) million fare hike

Krinklewood Carriages, Pokolbin, NSW About half a year ago, I launched the Opal or Not website that compares the fares of Sydney’s new Opal transit smartcard and the paper tickets it replaces. As I’d hoped, this quickly went viral, bringing attention to Opal’s hidden costs and ending up in the Sydney Morning Herald.

Now getting my mug in the SMH is all well and good, but my end goal was to “make Transport for NSW to come up with a saner fare scheme”. Instead, TfNSW sent me a rather bizarre good cop/bad cop letter telling me to they’d love to cooperate, but implying they will sue me if I don’t stop using the Opal logo. When I offered to play ball, changing the logo as a token of good faith, they sent me a 26-page report about all the things with my site that they were unhappy with; quite handy, as you’ll see later, but not exactly the kind of action I was hoping for.

Meanwhile, as they prattled on about how they’re committed to transparency yadda yadda, another arm of the same bureaucracy has been steadfastly stonewalling my request for a copy of that 7% report. At time of writing, after half a year of increasingly absurd effort, they have yet to confirm in writing that the “7%” report they keep talking about even exists. And maybe it doesn’t, because in other places they claim the figure is 4%!

Well, fine. If TfNSW is not going to release their report, based on their vague projections, I’m going to write my own report, based on the cold hard data of the over 130,000 fare comparisons made on the site. And here it is.

The short version

	Who will pay more with Opal?	Cost of Opal to Sydney commuters?
Transport for NSW report	4%? 7%? ^{[citation needed]}	They’re not telling
Opal or Not report, model A: optimal tickets	61% ^[1]	$98 million/year ^[1]
Opal or Not report, model B: estimated tickets	26% ^[1]	$54 million/year ^[1]

Here’s how you can help fix this.

The long version

Wagon, Felton, CA, USA

Now there’s two basic requirements for Opal or Not’s data for it to be of any use: it has to be representative, and it has to be correct.

So the first question is, are the 130,000+ fare comparisons made on Opal or Not a representative sample of the Sydney commuter, instead of (say) an escapist chimpanzee recalculating his fare from Taronga Zoo to Bondi over and over again? In terms of sheer sample size, it should certainly qualify: 812,000 people use Sydney’s public transport on an average weekday, so even if we discount every other comparison, we’re reaching a good tenth or so. For comparison, the Bureau of Transport Statistics Household Travel Survey 2011/12 (HTS) is based on the survey responses of about 3,000 households.

A more useful comparison of the quality, though, is to compare Opal or Not’s mode distribution against the HTS. This isn’t entirely straightforward, since Opal or Not compares door-to-door ‘journeys’ (eg. bus & train) while HTS counts single ‘trips’, but by making the not unreasonable assumption that a bus & train journey equals one bus trip plus one train trip, we get a 70/26/4% split for train/bus/ferry, while HTS has 59/37/4%. This is unsurprising, since Opal for buses remains very much a work in progress, but it does skew the results a bit since the average train commuter loses more from Opal (an extra $65 a year, to be precise) than the average bus commuter. Long story short, if we adjust the mode split to match the HTS, this shaves $5.7m off the paper advantage.

“But wait”, I hear you say, “even if the sample population is representative, what if the calculation itself riddled with errors”? To make sure all potential problems are covered, I’m going to go through TfNSW’s llst of 18 “issues” with the site and analyze their impact on the results one by one. You might want to a brew a stiff coffee before you read on.

1. Incorrect fares shown for some train journeys

Opal or Not computes train distance bands based on actual distance, not wacky-world fare distance, which can throw off estimates by a few km. This means that users whose trip lengths are right on a fare boundary (10 km, 25 km, etc) may be quoted fares for the wrong fare band.

However, Opal and MyTrain bands are defined identically by distance. This means that, even if a user is quoted the wrong zone, they are quoted the correct fares for those zones, and the comparison is thus fair. No adjustment needed.

2. Train journeys that cannot be calculated

There are some cases where the underlying Google Directions API only offers buses, because they’re much faster, so Opal or Not cannot estimate the train distance. But if the site can’t calculate the route, it can’t get the comparison wrong either. No adjustment needed.

3. Incorrect morning peak times for Intercity trains

Opal or Not assumes that morning peak is always 7am to 9am, when it’s actually 6am to 8am when traveling from Intercity stations. However, the website does not ask for exact times, it only asks for time bands, so the relatively few users affected can self-adjust.

But yes, those who do enter 6:30 Intercity trip as “before 7 AM” will get quoted an off-peak Opal fare when they should be paying a peak fare. Since there are more of these people than there are 8:30 Intercity travellers who get quoted peak for off-peak, the net effect of the error is in Opal’s favor: the true cost of Opal is higher.

4. Weekend off-peak fares not included

Opal or Not does not implement train off-peak fares all day on Saturday (or Sunday) for either Opal or paper tickets, instead applying the weekday off-peak rules. This has two effects:

Peak period travel on Saturday is incorrectly calculated at weekday peak fares. This inflates the cost of both Opal and paper equally, producing no net effect.
Off-peak travel is calculated with weekday off-peak periods. This gives an unfair advantage to Opal, since this grants ~26% of travelers the discount, vs. ~16% for paper travel. (The actual figure is, of course, 100%.)

All told, the net effect is again in Opal’s favor.

5. Description of MyBus and MyFerry tickets as zonal

This is a wording quibble: MyBus and MyFerry tickets are based on distance travelled, and strictly speaking should not be called “zones”, since that usually implies geographical areas. (By comparison, MyMulti is a true zonal ticket.) Fair enough, but obviously no impact on fares.

6. Change in method of calculating bus fares

The website currently assumes that that MyBus sectors map one-for-one to Opal’s distance-based fares, which is not the case. However, without access to the exact sector data, which TfNSW does not make public, it’s not possible to compute what difference this makes in practice. Opal claims some fares will be cheaper, but the SMH has some cases of people whose fares nearly double, and as we know, while TfNSW probably has their own estimates for this, they refuse to release them. I’ll give them the benefit of the doubt and call it a draw: no impact.

7. Incorrect MyMulti ticket recommended for some journeys

Opal or Not’s determination of what MyMulti ticket to use combines bus and train distance bands, meaning issues 1 and 6 (train and bus distances) apply and users may be quoted the incorrect MyMulti zone. Again, since there is no systemic bias in either direction, this alone does not bias results.

There is, however, one important source of error in the MyMulti calculations: they assume that train fares bands map directly to MyMulti zones. This is correct for trips originating or terminating from the CBD, but will incorrectly compare to a MyMulti 1 for someone traveling from (say) Blacktown to Parramatta, actually MyMulti 2. The diagram to the left attempts to demonstrate this, with the top ‘ovals’ representing how Opal or Not calculates trip costs and the underlying colored bands showing how the actual MyMulti zones apply. The bits marked “+1/2” show undercharging, whereas “-1/2” shows overcharging: for example, somebody traveling from Hurstville to Parramatta would have been computed as MyMulti 3, when actually a MyMulti 2 would suffice.

We can attempt to correct for the error by applying the known distribution of train travel distances to major destinations in and out of the CBD (both courtesy the Compendium of Sydney Rail Travel Statistics, 8th Edition, RTS), times the number of train travellers using MyMultis and finally the cost incurred (or saved) from using the wrong MyMulti zone. The math is fairly hairy and makes a few rather generous assumptions (eg. that each station’s patronage follows the general distance distribution and that the Inner Sydney statistical area is close enough to the rail CBD fare zone), but because the under- and overcharging tend to balance out, the net effect is fairly minor: $300k off paper tickets.

For completeness, I’ll note that the same issue theoretically applies to ferries. However, since ferries are a relatively minor form of transport and the overwhelming majority of ferry travel is to/from the CBD, the approximation used by the website is more than good enough for our purposes: Ferry 1 = MyMulti 2, Ferry 2 = MyMulti 3.

8. No provision for customers with complex travel patterns

Opal or Not is not a general-purpose fare calculator, but a single-purpose tool geared solely at making it easy to check how Opal will affect the cost of your commute. Per the HTS (Table 4.3.3), we know that commuting to work and school/uni account for the vast majority of public transport usage, so this means the results also represent the vast majority of Sydney’s public transport users. And once again, if the site doesn’t support a given travel pattern, then it’s not giving inaccurate results, it just doesn’t support it. No adjustment needed.

9. No consideration of the customer’s current ticket

Locomotive #7 "Sonora", Felton, CA, USA TfNSW, and quite a few of my readers, think it “unfair” to compare Opal to monthly or quarterly tickets, for two distinct reasons: 1) people do not use them very much, and 2) they are not a good fit for some people. Let’s analyze these in detail.

First, do people actually use them? TfNSW loves to trot out the statistic that only 4% of ticket sales are periodicals, but this is highly misleading: if person A buys a yearly ticket once a year, and person B buys two singles every working day, only 0.2% sales will be for yearly tickets, even though 50% out of A and B use them. The best statistics for actual usage that I’m aware of are in RTS Table 20, which extrapolates from ticket sales and covers trains only: according to this, periodical tickets (weekly or longer) account for 50% of usage, with 8% using monthlies or longer.

At the end of the day, though, this is kind of irrelevant: Opal or Not is all about showing people the best possible fare, and I can only assume that its users are rational economic actors who choose the cheapest fare. In the same way, I’ll give Opal the benefit of the doubt and assume that its users are religious about tapping on and tapping off, so they never get accidentally charged the maximum fare — a lip-smacking $8.10 if you forget to tap off the train!

The second objection is that people are rational and choose not to opt for periodicals because, when accounting for vacation etc, they cost more. However, since a train quarterly covers 13 weeks of travel but costs the price of 10 weeklies, virtually everybody who uses the train would actually be better off buying them, even if they take two weeks off that quarter. For buses and ferries, TravelTens are unequivocally more flexible and cheaper than Opal if you have even slightly irregular commute patterns.

What’s more, comparisons for monthly and quarterly tickets were introduced only after launch, meaning the initial 5,000 or so users did not benefit from them, and a rounding error meant that the next 10,000 or so had quarterlies incorrectly computed as 12 weeks instead of the correct 12.85, again a bias in Opal’s favor.

All in all, I’m going to call it a draw and say no adjustment needed. If you have better stats, or thoughts on how to quantify this, I’m all ears.

Update: By popular demand, I’ve taken a stab at adjusting the pricing comparisons to be in line with ticket usage from RTS Table 20. This is rather rough and ready, because Opal or Not does not collect the user’s actual ticket types or record non-optimal paper fares, but long story short, this shaves $44m off the paper advantage and converts 280k people into the Opal camp, meaning only 26.46% pay more with Opal. This, mind you, is still about 4x more than what TfNSW is claiming, and it goes to underline what we’ve seen from previously: those who win from Opal win a bit, while those who lose generally lose a lot.

The previous calculation is now Model A, and this version is now Model B.

10. Opal trip advantage not always applied

More specifically, the site does not cater to the case where there is less than 60 minutes between the trip out and the trip back. Again, since Opal or Not is a commute comparison tool and most people work/study more than 60 minutes a day, this scenario is irrelevant. No adjustment needed.

11. No provision for journeys with multiple transfers

Setting aside ferries, there are three possible realistic multiple transfer combinations: train/train/bus, bus/bus/train, bus/train/bus. The first and most common is already catered for, since Opal or Not can compute the fares between any two train stations, including transfers. For the second, Opal would be cheaper if you were currently foolish enough to purchase separate tickets, but as we know MyMulti is nearly always the better option even for a single bus/train transfer, much less two; not covering this thus tilts the balance in Opal’s favor.

So that leaves the third option, bus/train/bus. Getting good data on this is hard, but Bus Users in Sydney, 2002 p5 (not a typo, that’s the latest available) tells us only around 7% of bus journeys involve two buses, and Rail Travel Statistics 2012 section 3.2 tells us 17.7% of train users transfer to bus, from which we can determine a conservative upper bound of 1.2% travellers affected. Not that this helps Opal’s case any, as MyMulti will almost certainly wipe the floor with Opal for these poor people, but I’ll graciously not dock Opal’s score for this. In Opal’s favor.

12. Many NSW TrainLink Intercity stations not included

That’s because they were not covered by Opal at the time the service was launched, although support has since been added both to Opal and Opal or Not. Once again, if you can’t compare it you can’t get it wrong, so no impact on accuracy.

14. $2.50 Sunday cap not included

Neither Opal’s all-day-for-$2.50 offer nor the more limited Family Fun Day ticket it replaces is covered by Opal or Not. While this is clearly a win in Opal’s column, for the vast majority of Sydney’s commuters it’s a nice bonus, not an actual commute cost-saver.

The second complication is that since the Travel Reward is accounted for, this applies to only people who travel on Sundays without accruing 8 weekday journeys. Since Opal or Not does not ask about days of week, and I’m not aware of any TfNSW data for this, quantifying the size of this group is difficult.

All in all, no adjustment made, but underlying bias is in paper tickets’ favor.

13. Domestic Airport and International Airport stations not included
15. Opal Child/Youth fares not included
16. Opal Senior/Pensioner fares not included

Opal or Not is a weekly commute calculator, all these are explicitly out of scope and do not affect the accuracy of results. No adjustment needed.

17. Daily tickets not included

The Opal $15 daily cap is included in the site; however, the $23 MyMulti day ticket is not. This skews the results in Opal’s favor.

18. Fares for services and classes of Opal cards that have not yet been announced

I had to read this one twice: TfNSW apparently considers it a flaw that Opal or Not does not cover Opal fares that have not yet been announced? I’ll be happy to address this once somebody lends me a time machine. No adjustment needed.

We’re done! Almost, anyway, since magnanimous soul that I am, I’m going to include a large source of error that TfNSW did not notice:

Bonus. Off-peak fares on trains

Off-peak fares are frequently cited as a major win for Opal, and while Opal or Not now supports them, this option was not available initially, affecting the first 26,925 comparisons. Looking at the additional statistics kept for off-peak fares, we can see three general trends:

The 15% of train travellers who travel outside both peak periods are unequivocal winners, Opal saving them around $200/year.
The 20% who have one peak and one off-peak trip come out a wash, gaining or losing less than $100/year.
The 65% who hit both peak periods pay on average $270/year more.

So while the majority still lose, the gains of the small group of winners are so large that they translate to a 30% decrease in the overall cost. Adjusting the data accordingly (see spreadsheet), this docks some $14 million off paper’s advantage. Better start waking up before 7 AM!

Conclusion

Roaring Camp Railroad, Felton, CA, USA Of the 20 potential issues identified by TfNSW and yours truly:

4 bias the results in paper tickets’ favor, and are accounted for by adjusting down by $20m,
3 bias the results in Opal’s favor, which means Opal’s actually even costlier than this report indicates, and
13 have no measurable impact.

In other words, this report bends over backwards to give Opal the benefit of the doubt, and it still comes up a cropper to the tune of $54 to 98 million dollars.

Call to action

So what should Transport for NSW do to fix Opal? I have two simple suggestions:

Allow bus, train and ferry transfers. A wide array of wise men all agree: if Melbourne, Perth, Brisbane and Adelaide can all do it, there’s no sane reason why Sydney can’t. Penalizing modal transfers makes Opal a huge leap backwards for Sydney transport.
Lower the base fare and drop the Travel Reward. The “Travel Reward” is supposed to encourage using public transport, but it ends up discouraging it for many people, and only encourages gaming the system instead.

If you agree, let the Minister for Transport know. Gladys, the buck stops with you here: it’s not too late to fix this, and turning Opal from a PR disaster into a win for your Sydney voters will only take a stroke of your pen.

And if you’re a commuter who wants to hedge your bets, lock in your savings before Opalcalypse on September 1st, 2014, and buy a quarterly or yearly ticket today.

How SMS set back the mobile internet by ten years

Stop me if you’ve heard this one before: an obsessively perfectionist corporate maverick defies conventional wisdom and launches a radical new mobile “i”-phone that offers built-in email integration, an app store with simple payments and, finally, a decent way to surf the web, all tied together with revolutionary ease of use. Sales go gangbusters, the company mints billions, mobile Internet usage skyrockets and competitors are left scrambling to catch up.

I am, of course, taking about Japanese operator NTT DoCoMo’s launch of i-mode in 1999, spearheaded by Mari Matsunaga (pictured), who clawed her way up in the intensely male world of Japanese business and whose insistence of putting user experience first almost scuttled the project several times. i-mode phones were equipped with an “i” button that opened up a browser in one click, allowing users to surf in a curated portal of i-mode sites, some for subscribers only, most free but with in-site purchases charged directly to the user’s mobile bill, or to access the wider internet by entering URLs manually. Downloadable “i-appli” Java apps were launched shortly thereafter. Each phone also came with the “i-mail” email service that was completely interoperable with the general Internet (and other Japanese phones), delta some fairly drastic limitations on message size since we’re talking 320×200 screens and download speeds measured in kilobits. And since this was all running on packet-switched data, the billing scheme couldn’t have been simpler: pay per byte sent or received.

“So”, I hear you ask, “if this was so awesome, why have I never heard of this?” There are many theories, but I’m going to lay out the case for an unconventional one today: it’s all SMS’s fault.

Why SMS was awesome

SMS is the GSM mobile standard’s Short Message Service, and by telco standards it has a venerable pedigree, being first dreamed up in 1984 (yes, that’s thirty years ago). Fundamentally, SMS is a hack; a glorious hack, to be sure, but a hack nonetheless. SMS is built on the Signalling System 7 (SS7) protocol, designed in the 1970s for controlling landline calls. Unlike most previous telco signalling systems, which controlled calls by inserting tones directly into the voice stream and let phone phreaks run wild with blue boxes, SS7 has a separate control channel used primarily to signal when calls start and end, means it transmits blank frames the rest of the time. SMS squeezes text content into those empty gaps, which explains both why telcos were so fast to adopt it — it needed precisely zero additional hardware to roll out — and where that famous size limitation comes from: the frame size is exactly 140 octets, which is enough for 160 7-bit characters.

The other reason telcos love SMS is the money. Remember, if you’ve already got a GSM network, SMS costs nothing at all to transmit. (Yes, there is some cost involved in setting up SMS store-and-forward centers and interconnects and whatnot, and I made my living for close to a decade working in the field, but compared to actually building a network, this stuff is a rounding error.) However, people in the 1990s were already well-trained to pay for each call, letter or fax, so charging (say) a flat dollar a piece for each SMS still sounded like a bargain, and the telcos, buying SMS for $0 and reselling them at $6500/MB, weren’t about to disillusion them. In 2013, US operators alone minted 21 billion dollars from SMS, and you can still today get charged that $1 a piece if you happen to send an SMS while roaming in the wrong place.

So far, so awesome (if you’re a telco, at least), and Worse is Better when it comes to victory in the marketplace. But SMS’s sheer hackiness made it an evolutionary dead end, despite increasingly Sisyphean efforts to build more and more complicated infrastructure on top. For example, while SMS was originally meant purely for transmitting text, some clever noggins came up with the idea of using six of those precious 140 bytes for a User Data Header that, bit by bit, could say things like “I am a logo!”, so you could squeeze in the 1008 bits needed for a 72×14 monochrome bitmap representing the Vodafone logo. Or if you were so greedy that 134 bytes wasn’t enough, you could use a few more of those bits to say “I’m piece 3 of 7!”, and get the phone to concatenate those SMSes together into a chain that might contain as much as a kilobyte of info, just enough for a Crazy Frog ringtone (that’ll be $4.99, thank you). Unfortunately, SMS is a UDP-style best-effort-only service with no guarantees of delivery, meaning that the longer the message, the more likely it was that some pieces would fall by the wayside. And while there is a “delivery notification” bit that’s supposed to confirm a message was received, and whose absence can trigger redelivery, those notifications are also SMSes and are themselves often dropped along the way. Remember when, in the olden days, you’d occasionally get 3 or 5 copies of a message? Somebody was dropping your DNs.

Meanwhile, in Japan, they didn’t have SMS, because they didn’t have GSM. The Japanese telcos naturally noticed that this SMS thing was taking off in Europe and built their equivalents for the Japanese 2G standard PDC, but the operators never managed to sort out talking to each other, and by 1996 or so they just said “fuck it” and deployed good old e-mail instead. This was easy enough technically, since PDC offered packet-switched data and WAP 1.0 didn’t, and easy enough financially, since they didn’t have to worry about cannibalizing non-existent SMS revenue. The standard undiscounted data rate in Japan circa 1997 was around 0.3 yen per kilobyte, and because a short e-mail is about 3 kB (headers and all), it cost around one US cent to send/receive one to/from any device in the world that groks SMTP.

Why MMS is so bad

In 2000, NTT’s arch-competitor J-Phone launched Sha-Mail, the world’s first mobile photo messaging service, which (then as now) was an instant killer app quickly duplicated by NTT as “i-shot”. This was implemented using standard e-mail attachments and charged for at packet data rates, meaning around 10 yen ($0.10) for the 30 kB images taken by camera phones circa 2001, and phones and pictures sold like hotcakes.

Back in Europe, the WAP hype wave had just crested and crashed when it turned out people weren’t willing to pay one euro per minute to surf an unusably sluggish crude approximation of the few Internet sites that had bothered to completely rewrite themselves with WML. Yet the first GSM packet data scheme, GPRS, was starting to gain traction and it started to dawn on operators that SMS just wasn’t going to cut it for images.

So they came up with the replacement: MMS, the Multimedia Messaging Service, and you can already sense what a kludge it would be from that very first buzzword, which has killed all it touches. To grossly oversimplify, MMS works by sending magical SMSes called “WAP pushes” that tell the receiving phone to open a WAP connection to an MMS server, from which they then proceed to download what are essentially e-mails crammed full of bizarre custom MIME parts, which are then rejiggled into an approximation of the original message, defined with a laughably overcomplicated custom slideshow language not too distant from Powerpoint. For additional hilarity, the protocol provides no way of checking what the receiving device supports: instead, it’s the operator‘s job to attempt to manually track which phone number corresponds to which phone model. (Thought experiment: imagine if the Internet worked this way, and instead of User-Agent strings, you had to phone up your ISP every time you changed device, browser or window size.) This grotesque Rude Goldberg contraption is defined by eleven (11) different specifications, imaginatively named MM1 through MM11, each laying out the edges of one piece of the puzzle, and (unlike SMS) requiring very considerable resources to deploy. Unsurprisingly, nobody could agree on how to actually implement the damn thing, making MMS hysterically unreliable to the point that, in A.D. 2014, I still consider it a minor miracle to successfully transmit one.

But MMS did have one feature that operators considered very, very important: it let them continue charging per piece, although many operators socked you for data usage as well. So if you were visiting Mexico and Aunt Tilly sent you a 300 kB picture of her cat, she’d pay a flat $2 or something to send it, and you’d pay $30 to download it at $100/MB. Or, more likely, she’d pay $2 to send it, but her phone or your phone or your home operator or your roaming network or any of the gazillion boxes talking incompatible variants of MMn along the way would barf and lose it. And then Aunt Tilly would get pissed that you didn’t reply to her message, and never touch MMS again, which is in fact pretty much what happened.

Why i-mode never took off elsewhere

Now operators around the world did notice that this i-mode thing looked like a license to print money, and 17 of them ended up eventually launching it around the world, offering i-phones (small p) with little i-buttons that opened i-sites that charged i-ncredible amounts of money and otherwise looked a whole lot like the Japanese i-mode i-cosystem. Yet every single one of them failed.

Partly this was technical: not everybody/everywhere had GPRS, and i-mode over plain WAP, which essentially involves a dialup connection every time you press a button, is atrociously slow. Partly this was timing: so many fingers were burned in the bonfire of WAP malinvestment that few were willing to go all-in on another whiz-bang mobile invention, making the chicken-and-egg problem of getting sites to create i-sites tagged with i-mode’s “compact HTML” extensions that much harder to hatch. Partly this was the cutthroat nature of the market: at least one operator I know of launched i-mode more or less entirely as a cynical ploy to extract better terms from Nokia. But mostly it was just because none of those operators had the guts to go all-in on i-mail, because they were all so wedded to sucking on the SMS teat and could not bear to exchange $1 now in easy SMS profit for $0.01 forever in data revenue. And the straw that broke the back of the few brave operators that did launch it? The fallback mechanism for sending e-mail to non-i-mode phones was… MMS.

Instead, it was BlackBerry who grabbed that market opportunity in the West, letting you read business e-mails on your CrackBerry and later launching BlackBerry Messenger (BBM), giving its own users a first taste of rock-solid, instant, all-you-can-eat messaging. What’s more, it wouldn’t be too far-fetched to argue that BBM was such a hit in North America only because SMS was so bad there: half the operators in the US and Canada were using CDMA, which doesn’t natively support SMS. But this was a completely proprietary protocol, not an ecosystem others could use, and BlackBerry did its best to make sure BBM only worked properly on their own devices, also opting for easy money now over a chance of future relevance later.

Jobs delivers us the keys to the new kingdom

So that was the state of the rest of the world until 2007: SMS ubiquitous for lack of interoperable alternatives, MMS strangled in the cot, BBM minting money in its little niche. Only then came along the iPhone and the Cambrian burst of mobile internet evolution, which saw Android make mobile e-mail a commodity and the rise of alternative messaging systems like Skype, Facebook, Snapchat, WhatsApp, etc. In telco lingo, equal parts dismissive and terrified, these are to this day called “Over-the-Top” (OTT) because they finally scaled over that charge-per-piece wall put in place by SMS in 1984, and brought the rest of the world to where Japan was in 1999.

Meanwhile, in Japan, Matsunaga was railroaded out of NTT barely a year after launching her creation, and is now a director at Bandai. NTT lost over $10 billion on its i-modal overseas misadventures but grabbed nearly two thirds of the Japanese market, with around 50 million i-mode users at its peak in 2008. But they refused to sell the iPhone until 2013, by which time it had lost over 20% of that market share to nimbler rivals like SoftBank, which went on to buy Sprint.

And that’s where my story ends. Now, I’m not saying i-mode had any real chance to rule the world, GSM was not a good fit technically and SMS was already entrenched by the time it came along. But if SMS hadn’t existed, the world would almost certainly have had an i-mode equivalent much sooner, and you can only wonder how the world would look today if it had.

Sydney’s screwed-up smartcard, or why I wrote “Opal or Not”

The backstory behind Opal or Not.

Last year, Sydney started trialing its new “Opal” transit smartcard. As a regular commuter on Sydney Ferries, the first service to roll out Opal, I awaited its arrival eagerly. After all, they couldn’t possibly screw it up worse than Melbourne’s Myki, whose ludicrous cost overruns and sheer technical incompetence I had witnessed first hand earlier.

Alas, while Opal has indeed been lighter on the government’s purse and is mostly capable of registering card taps, Transport for NSW still managed to completely stuff up something that Melbourne didn’t: the fare structure. For many users including me, Opal is much more expensive, in my case translating to $332.80 more every year for the same commute.

Virtually every smart card in the world prices individual trips lower than the equivalent single fare, meaning it always make financial sense to use the card. Not Opal: for ferries and buses, a single fare costs more than a trip on the TravelTen paper ticket, and only on your 10th trip of the week does the cap finally make Opal cheaper again. And if, like me, you occasionally bike to work or work from home, meaning you use the ferry or bus less than 10 times a week? You fall into the Opal Fail Zone, shown in red above: that’s the premium you pay for the privilege of using Opal.

But believe it or not, I soon found out that there were others even worse off than me. Say you live in Dee Why, take a bus to Wynyard, and switch to the train to Central. (Substitute with bus/train combo of your choice.) Because Opal has no replacement for MyMulti, your commute is going to rocket up $728 a year, even if you travel five days a week!

Just look at that thick red slab of Opal Fail: if you’re commuting by bus and train, unless you’re doing it exactly 6 times a week, it never makes sense to switch to Opal.

Is it easy to figure this out? Hell no, it takes an intimate understanding of Sydney’s convoluted fare structure and a whole lot of flipping between browser tabs to come up with the actual numbers. The Opal website has some contrived examples, every single one of which shows Opal as cheaper, but lacks even a basic Opal fare calculator, never mind any way to compare to non-Opal fares.

Now I could have written feedback to Opal, which would have gotten me a form letter response with sneering thanks before getting chucked in the bin. Or I could have written an angry blog post (well, I am writing one), which with some luck would have been retweeted a few times before being overtaken by Justin Bieber’s latest drunken antics. But neither would have had any real impact.

Instead, I wanted something that would:

Let people see exactly how the switch to Opal will hit their wallet
Collect statistics on how many people are positively or negatively impacted by Opal, and by how much
Ultimately make Transport for NSW to come up with a saner fare scheme that encourages all public transport use and does not penalize transfers.

So I spent a few evenings coding up a fare calculation engine (and Jesus Christ that was a pain, just look at this shit) and a few more slapping a web interface on top, and the result is Opal or Not. Here’s hoping it was worth it!

Half the Donut: Why an entrepreneur earning $100k gets to keep over $99k in Singapore but under $57k in San Francisco

You’re about to quit your job and start living the life of a one-man entrepreneur. Living in your basement, you have no rent and no employees, and since you’re selling your skills you don’t need venture capita yet, but you’re still dreaming big. All things being equal, where should you base your business?

Tip: Probably not in the place where the taxman takes a red bite like this out of your income donut.

More specifically, in each of San Francisco, Helsinki, Sydney, Tokyo and Singapore, if your business earns $100,000 in net revenue and pays you a “ramen-profitable” $2000 monthly net salary:

How much post-tax profit will your company make in a year?
If all of this profit is paid out in dividends, how much does will you have left after taxes?

If you’re wondering why I picked those five cities, part of the reason is that it’s a nice geographical, cultural and political spread, but mostly it’s because I’ve either founded a company or worked for a company based in each of them. The $100k net revenue/$2k net salary model, funding product development on the side, is basically what I modeled my one-man consulting company on back in 2006.

To jump straight to the answers, click here. If dividend imputation and municipal inhabitant taxes get you all tingly and excited, read on. Hardcore masochists who’d like to double-check my math are also invited to examine the gruesome innards of this Google Docs spreadsheet.

Disclaimers

This is a hypothetical exercise, so I’m going to simplify as much as I can, the accounting is still stupidly complicated and there’s a whole lot of cramming square pegs into round holes going on. In particular:

I ignore all non-financial considerations. Visas, availability of talent, infrastructure, economic prospects, legal and political environment etc are all important real-world factors for locating a business, but out of scope for today.
$100,000 is net revenue, we ignore all expenses aside from salary and tax. In other words, $100,000 is what’s left over after business registration, accounting, stationery and whatnot, and we also assume that those costs are the same across all countries.
We assume sales tax does not apply. This is not as unfair as it seems, since most countries exempt companies until they reach fairly high yearly sales thresholds and exclude online and/or international/interstate sales.
The business has no tax deductible expenses. This is particularly unrealistic for the United States, where 72,500 pages of federal tax code means creative deductions are a national pastime, but them’s the breaks.
Pensions are accounted for on a defined contribution (what-you-pay-is-what-you-get) basis, so $1 put in now is worth (at least) $1 later. This is true for Singapore, Australia, US 401(k)/IRAs and some Japanese corporate plans; this is manifestly not true for US Social Security or the national plans in Finland and Japan, but for lack of a better measure we assume it is anyway.
Taxes are computed assuming that pension and health insurance contributions are tax-free, which allows me to ignore the distinction between employer and employee contributions.
The owner is a full-fledged local resident under 40 for the purpose of tax, pension, insurance etc brackets.
For the sole purpose of minimizing futzing about with exchange rates, for income tax thresholds etc I’m going to merrily assume that 1 USD = 1 SGD = 1 AUD = 1 EUR = 100 JPY. This is obviously not correct, but is not all that much worse than picking actual exchange rates that’ll be out of date in moments anyway.

Last but not least, this is a work in progress, a revision history is at the bottom of the article. Now, let’s roll up our sleeves, channel the spirit of the late great Herbert Kornfeld, and balance this shit wit’ a quickness.

Round 1: Paying a salary

$2000 x 12 = $24,000, leaving $76,000, right? Not so fast. There are four main things to worry about here: income tax, pension fund contributions, payroll taxes and health insurance. Both pensions and health insurance are tricky since our countries’ systems vary so widely, so we’ll just attempt to standardize the legislative minimum. And since all these fees are usually percentages of salary, we have to work our way backwards starting from desired post-tax income to get to the employer’s cost.

In San Francisco, I’m going to cheat a bit and outsource the otherwise horribly complex income tax computation to the MIT Living Wage Survey, which figures that for a single adult, it takes pre-tax earnings of $26,692 to have $1,929/mo left over after tax, an effective tax rate of 13.28%. Optimistically assumes that rate stays the same at $2,000/mo, we now need $27,675. Next, we add in a 6% employee pension contribution with 6% employer matching (+$3203) and boost medical from $149/mo to a post-Obamacare estimate of $368/mo (+$2628). On top of this, employers have to pay 6.2% of wages for Social Security, which I’ll lump as a “pension” for the purposes of this article; 1.45% for Medicare; 3.4% for CA unemployment, plus 1.2% and 0.1% of the first $7,000 only for federal unemployment and employment training tax respectively. (Huge props to ZenPayroll.) In a rare bit of good news, SF’s payroll tax (1.5%) only applies once total salaries exceed $150,000, so we can ignore this. We thus arrive at $36,773.

In Helsinki, personal income tax is so complicated the only sane way to compute the effective rate is to punch numbers into the official tax calculator. A gross income of €29,000 and zeroes for everything else, including being a godless pagan who avoids church tax, nets income of €23,978, for an effective rate of 17.32%.

As a >30% shareholder of his own company, our hero can apply the lower “YEL” pension employer contribution of 17.55% for two years. On top of this, we have the employee side pension contribution of 5.15%, 2.04% mandatory health insurance, and 0.80% employer/0.60% employee unemployment insurance. We thus arrive at €36,615, which is, rather incredibly, a hundred bucks less than SF, and this gets you a cradle-to-grave Scandinavian welfare state!

In Tokyo, national income tax (shotokuzei) is 5% for the first Y1.95 million and 10% above, plus a flat 4% prefectural and 6% municipal tax (which combine to form jūminzei, resident tax), which works out to 16.6%. For pension, you can pick the national plan (kokumin nenkin) at a fixed Y14,980/month regardless of income, or a standardized company pension (kōsei nenkin) at 17.120% split evenly between employer and employee; being cheapskates, we pick the national plan. National health insurance is 80% of municipal and prefectural tax paid, which in Tokyo equates to 9.97% split between employer and employee. Phew! (Dōmo arigatō to Japan Consult.) Final damage 33,448 hectoyen, and I’m delighted to find out that I’m apparently the fifth person in the history of the Internet to use that lovably awkward term.

In Sydney, the Tax Office says the first $18,200 of income are tax-free and the next bracket until $37k is 19%, for an effective rate of 5.36%. The minimum pension (superannuation) contribution is 9.25% and Medicare (public health insurance) levy is 1.5%, and since our total income is under $84k, we’re not required to pay the Medicare levy surcharge for not having private insurance. Payroll taxes in Australia are administered at the state level, but New South Wales only applies its rate of 5.45% once payroll exceeds $750,000, so that particular bullet is dodged, and at $28,087, the end result is easily the lowest of the bunch.

In Singapore, income tax is easy: 0% for the first $20k, 1.4% for the next $10k (usually 2%, but they’re discounting 30% in 2013), for a scarcely believable effective rate of 0.24%. Mandatory pension (CPF) contributions for employees under 50, which include a health insurance component, are 16% for the employer and the 20% for the employee, with a maximum contribution of $1800/mo (not applicable here). The only other obligatory cost is the Skills Development Levy at 0.25% of salary, capped at $11.25/mo. This equates to $32,777.

We can now compare the total cost of employment, ie. how much it costs the company to pay the owner that $2000/mo salary:

But while taxes disappear into the gaping jaw of Leviathan, pensions are retained by the owner in some form of another (see Disclaimer), so a more instructive comparison deducts salary and pension from total cost to arrive at what I’m calling “employment overhead”:

At the end of Round 1, Singapore is the clear winner ($117!) and Sydney a close second, followed by Helsinki and, at the back of the bus, Tokyo and San Francisco, where nearly $8k disappear in a puff of smoke.

Round 2: Corporate tax and dividend taxation

In our idealized mini-company, everything that was not paid out to the employee is pure profit. If the company wants to dish them out to its sole shareholder, how much do they get after the taxman takes his share?

In California, a C corporation pays 8.84% corporate tax. (A pity we’re not in Nevada, where it would be zero.) Pile on federal taxes at 15% to $50k and 25% to 75k, noting that you can deduct your state corporate tax first, and we get 23.7%. Since California treats dividend income as ordinary taxable income, in the $36k-to-87k tax bracket you’re looking at 25% to Uncle Sam, 12.3% to California, 3.8% to Medicare and 1.2% for “deduction phaseouts” (wat?), totaling a whopping 42.30%. Tot these up, and our entrepreneur is left with 44% of what the company earned, or $27,828. And if that sounds bad, here’s a calculation that arrives at 26% when you max out both tax brackets!

That said, if you’re going to do this in real life and do not have ambitions to grow to be the next Google, you should almost certainly opt for an S corporation or an LLC instead. These don’t pay corporate taxes: instead, they pass their income (or loss) onto their shareholders, who then pay normal income taxes. This is good if you’re keeping the money, but terrible if you were planning to reinvest it. However, since all other corporation types in this little survey are “real” companies that can choose to reinvest or issue dividends, I felt that a C corporation is a fairer comparison point.

In Helsinki, both corporate tax and the tax treatment of dividends was revamped in 2013. From January 2014 onwards, corporate tax is 20%, leaving €63,385 in post-tax profits. The basic capital gains tax on dividends is 30% (but 32% above €40k), with a tax break of 75% on the first 8% of “free capital”, which for our newborn company equates to the total yearly profit. This works out to an effective rate of 28.59%, leaving €36,211 in our hero’s bank account.

In Tokyo, corporate taxation makes Japanese payroll and income taxes look simple. Based on this handy summary from JETRO, for “a small company in Tokyo”, corporate tax is 15%, “restoration corporate surtax” (read: Fukushima surcharge) is 1.5%, prefectural and municipal “inhabitant taxes” in Tokyo are 0.75% and 1.85% respectively, “enterprise tax” is 2.70% (I presume this is meant to discourage entrepreneurship?) and as a cherry on top “special local corporate tax” is 2.19%, for a total of 23.99% but an effective rate of 22.86% for reasons I won’t even claim to understand. But wait! Once your earnings top Y4m, new rates apply and you now get socked for 24.56%. In effect, the effectively effective rate is an ineffective 23.54%.

So that’s taxes, now dividends, which are even more like tentacle porn. To quote Japan Tax, “As so often seems the case with Japanese individual taxation, what would be expected to be a simple tax matter is made overly complicated by a range of expiring tax benefits (that are often revised or extended) and a range of alternative obscure reporting elections, deductions or exemptions.” The short of it is that the withholding tax rate for unlisted shares is 20%. The long of it is that you can then choose to report or not report it as ordinary income. If you do report, you have to pay income tax, but receive a dividend deduction of 12.8% for overall income of Y10m or less but 6.4% above which offsets the withholding tax and, you know what, fuggedaboudit. Pay 20%, end of story, and keep 40,709 hectoyen.

In Sydney, company tax is a flat 30%, no if ands or buts, and the highest rate in our survey. Dividend income is treated the same as ordinary income and taxed at same rate, except that thanks to dividend imputation you get “franking benefits” that are meant to offset the corporate tax already paid. Acting on the optimistic assumption that they do, this means all that’s left is the difference between the corporate tax rate and the receiver’s marginal income tax rate, in our case 32.5% (for the $37k+ tax bracket) plus 1.5% levy, or 4%. This leaves a rather juicy $48,326.

In Singapore, corporate tax is usually 15%, but new companies in many sectors including IT pay zero (0%) tax for the first three years for their first $100k in profit, no special applications needed. There’s also a whole slew of other tax incentives, including 4x deductions of IT gear and, most incredibly, cash transfers of up to $5,000/year from the tax man for companies that make a loss despite earning over $100,000 in revenue, but that’s another story and our rules exclude this kind of thing anyway.

Singapore has no capital gains tax, so Singapore company dividends are also tax-free. The company can thus pay out $67,223 in dividends, and the shareholder gets every last cent. This means our entrepreneur’s yearly earnings after tax are $91,223, and adding in the $8,640 sitting in their pension fund, they have managed to hold on to $99,883 of it.

Dear reader, if you made it this far, I salute you. Now all that’s left is to sum up your salary, your pension and what’s left of your dividends to see how much of your $100,000 you still have left over.

Conclusion

Singapore romps home as the undisputed winner, with the entrepreneur keeping a scarcely credible 99.9% of what they started with. It’s just a real shame that, for political reasons, the government has recently gutted the EntrePass scheme and thus made it close to impossible for a foreign one-man entrepreneur to set up shop. (If you’re keen anyway, check out the Singapore Expats “Business in Singapore” forum or drop me a line.)

Somewhat to my own surprise, Sydney rocks up in second place with 74.6% left over. Australia’s not what you’d call a low-tax (much less low-cost) country, but income tax is highly progressive and the dividend franking system means you only pay tax once on dividends, meaning that at low incomes you get to keep most of what you earn. The calculus would change pretty rapidly if you earned $200k and found yourself in the 45% income tax bracket.

Helsinki and Tokyo show up neck and neck in 3rd and 4th place, with 66.8% and 66.5% respectively. Both have heavy taxation of income, payroll, corporate profit and dividends, but no total clangers. (Except Finland’s 24% value-added tax, the exclusion of which makes Helsinki unrealistically rosy, unless you can source all your income from outside the EU.) The canny entrepreneur could nudge up both of those figures: in Tokyo, you’d actually want to pay yourself more salary since income tax is lower than corporate tax, while in Helsinki the reverse applies.

And trailing the pack with barely half left over is poor old San Francisco with 56.9%. Now obviously there’s more to deciding where a company sets up than taxes, because otherwise Silicon Valley would be empty… but unless you really need to be there, from a financial point of view setting up pretty much anywhere else seems to make a whole lot more dollars and cents.

Acknowledgements

Thanks to corporate law ninja Joe & serial entrepreneur Juha for a sanity check of the US, Finland and Japan calculations, Tuomas Talola for corrections to Finnish calculations, and my Singaporean accountant, the infatigable Ms Tan, for clueing me onto this stuff back in the day. If you spot any mistakes, drop me a line.

Incidentally, Joe says that a fairer version of this exercise would model the corporate and personal taxes of each country and then solve for the optimal combination of salary and dividends. Any takers?

Revision history

31 Oct 2013

Recomputed Finnish income taxes with tax office calculator.
Adjusted Finnish employer pension to use YEL rate instead of TyEL rate.
Corrected Japanese pension to pay national pension (kokumin nenkin) only.

Lonely Planet: This is Not the End

Commenting on the affairs of past employers is bad karma, but the media circus surrounding Lonely Planet’s recent restructuring, with Skift’s hash of disgruntled misinformation and the Guardian’s premature obituary, is sufficiently misguided to warrant an unsolicited opinion.

Lonely Planet is and has always been a print publishing operation. Despite their carefully cultivated hippy-dippy image, the Wheelers ran a tight ship and LP was known in the industry for being able to produce and distribute more guidebooks of higher quality at a lower cost than anyone else in the business. This was achieved by a relentless focus on tweaking the publishing machine, and during my time there were regular mini-celebrations for (say) switching to a new printer that allowed cheaper color pages or trimming editing time by 10% by automating tasks that were previously done by hand in layout.

Yet being a print house left the company unprepared for the digital era, and despite its early web presence, it never seized the chance to become Expedia or TripAdvisor. Two anecdotes illustrate why:

Early on, one of the publishing execs was taking me through The Spreadsheet, which forecast in minute detail and often with stunning accuracy how much a book would cost to create and how much it would sell, taking into account everything from the cost of public transport in the destination to the impact of upcoming titles from the competition. Offhand, she remarked, “I don’t think we should be investing in digital until its revenues exceed print.”

Taken at face value, this seemed absurd. How would digital ever grow without any investment? Only later did it dawn on me: “investment” for her meant doing what the spreadsheet measures, which is putting money into books. Digital revenue would come anyway from e-books, which would be faithful replicas of print books, and once the magical 50% tipping point was reached, they could start by adding video clips of the Eiffel Tower to page 294 in the e-book.

But what if people don’t want e-books?

Later on, I mentioned the travel potential of Google Glass to one of the people in the product development team, responsible for dreaming up Lonely Planet’s future products. “Yes!”, he enthused, “just imagine if somebody wearing Glass looked at our guidebook, and they could see the latest edits superimposed on top!”

But what if people stop buying printed guidebooks?

Mind you, these were both consummate publishing professionals who live and breathe print. So at the end of the day, even though they and others at LP knew in their bones that print was falling, and that e-books and apps weren’t making up the slack, they simply didn’t know what to do about it, other than to cut more costs and churn out more books.

The new CTO Gus, on the other hand, does. LP’s asset is its independent content beholden to no-one, which drives its website and its brand. Despite debacles like BBC’s catastrophic mismanagement of Thorn Tree, at 100m+ visitors/year LP’s digital footprint remains head and shoulders above its print competitors, and its vetted content has no match (yet) elsewhere in the digital world. What’s more, they’ve already spent years putting in the hard yards to bring their technical backend up to speed as well. A relentless focus on digital is LP’s best shot at survival, and last week’s layoffs, far from being a portent of doom, are the most concrete sign yet that NC2 Media gets this as well.

Particularly important is the unheralded switch to a “destination editor” model, which finally breaks the stranglehold the book publishing schedule has had on the operations of the entire company. For example, this will allow the website to be updated continuously, instead of having to wait for the next book edition to roll around. Far from giving up on content, this puts it front and center, and the move parallels The Guardian‘s digital transformation that has seen the newspaper grab a sizable online audience far outside its native UK market.

None of this diminishes the human tragedy of letting go people who have poured years of their lives into what was indeed for many more of a family than a company. But as the only alternative is slow and inexorable decline guaranteed to lead to the elimination of every single job, this is the best hand the company can play. As the last page of LP’s guidebooks used to proclaim: “THIS IS NOT THE END”.

Time is money: How Clash of Clans earns $500,000 a day with in-app purchases

Confession time: While I like to rationalize my blog’s recent silence with changing jobs and moving to a new city, the truth is, the single biggest drain on my free time lately has been Supercell’s Clash of Clans. While this apparently puts me in good company, I decided it’s about time I shared what I’ve learned about how this “free” game apparently manages to spin well over $500,000 a day for its creator Supercell. (Update: Forbes reports Supercell now earns over $2.4m a day, the majority of that from Clash of Clans.)

Overview

The core of Clash of Clans is a bog-standard resource management game: mine “gold” and “elixir”, use gold and elixir to buy improvements to your town so you can build stronger armies, raid other players in order to loot their gold and elixir, rinse and repeat. If you’ve ever played Starcraft, Age of Empires or pretty much any other real-time strategy game, you’ll know the drill, and the buildings and units come off as almost painfully derivative. There’s a Barracks for new troops and Archer Towers for defense, you’ve got Zerg-like cheap and disposable Barbarians, weak but ranged Archers, slow Giant tanks for soaking up damage, etc. But formulas are formulas because they work, and it’s fun to set up your little village, win your first battles and watch your (thoroughly meaningless) levels and experience points rack up. The touch-screen interface is a pleasure to use and the smoothly zoomable 3D graphics are beautifully animated with cute little touches; for example, when you tap to select an army camp, every unit salutes their leader in a different way.

Show Me the Gems

“So that’s all well and good”, I hear you say, “but where’s the money coming from?”

In the standard Zynga playbook for making money off with freemium games, you would let players buy gold, elixir or the items they want directly. But Clash of Clans adds a twist: you can’t buy anything directly, but you can buy a third resource called “gems”. Unlike gold and elixir, gems are not necessary for building anything functional, they’re simply a type of “power-up” that serve as a shortcut. Need more gold to finish a building? Buy it with gems. Need more elixir to add a dragon to your army? Buy it with gems. Don’t want to wait a week for a building to finish? Complete it instantly with gems. In other words, gems mean instant satisfaction. What’s more, their cost is neatly obfuscated: purchased gems come in big, oddly numbered stashes (500, 1200, 2500, 6500, 14000), and once you have the pile sitting in your account, it’s easier to whittle it away 834 gems at time, whereas you’d probably think twice if asked to punch in your credit card details and confirm that you really want to pay $6.98 (just a sliver under the U.S. federal minimum wage) to upgrade your Wizard Tower.

Yet this formula’s beauty is that none of this is immediately apparent. You start the game with 500 gems, which is plenty for the initial stages, and there are many easily earned “achievements” that reward you with more, so that you don’t initially appreciate their value. The initial buildings are fast to build, with some building instantly and others taking a minute here or five minutes there. And you’re shielded from enemy attacks for three days, so you can take your time building up your base and raiding the AI’s goblin bases for easy loot.

As you advance through the levels, though, the time and expense of everything ramps up exponentially. A level 2 Town Hall takes 5 minutes to build and costs 1,000 gold; a level 8 Town Hall takes 8 days and costs 2,000,000 gold. And you soon encounter the next twist: on the later levels, patience is no longer enough. A maxed-out set of Gold Mines can produce 360,000 gold a day, meaning you could theoretically accumulate the sum neeeded for that Town Hall upgrade in 6 days. However, you’re being continually raided by other players, and since other players can see your wealth before they choose to attack, a fat bank account means you’re a fat target. What’s more, since successful raids award percentages of your wealth, a single “three-star” attack worth 25% can see 500,000 gold disappear in a flash.

There’s more. In your typical RTS, collected resources immediately go into your central storage. In Clash of Clans, though, they stay in the collectors, vulnerable to attack both by location and design (up to 50% can be stolen, vs. 20% for central storage), until you log in to manually transfer them to relative safety with a tap. This, too, makes it difficult to accumulate large amounts and encourages you to login at least several times a day. The chart shows why: if you start with 1,000,000 gold in storage, earn 360,000 daily, and get attacked once daily, losing 50% from your collectors and 20% from central storage, the player who doesn’t bother logging in for two weeks will see their pile drop 75% to under 250,000, while the player who logs in religiously four times a day will increase their wealth by over 50% to 1,550,000 — but even their earnings flatline well before two million.

What this means is that, once your bankroll is over 1.5 million or so, the only free way to keep the balance growing is grinding, a tedious non-stop cycle of raiding and army rebuilding, with nervous logins every five minutes to keep raiders at bay (you cannot be attacked while online). In their grandmotherly kindness, though, Supercell provides you a whole wealth of alternatives. Can you spot the pattern?

You can use gems to buy “shields” that stop you from getting raided: 250 gems ($2.50). Lest that seem too cheap, you’re preventing from using more than one week of shield per month.
You can use gems to fill up your gold or elixir storages instantly: 834 gems ($8.34)
You can use gems to upgrade your gold mines to the next level, where they will work faster: 966 gems ($9.66)
You can use gems to buy additional “workers”, so you can upgrade your production faster and earn gold/elixir faster: 1000 gems ($9.98)
You can use gems to double your production of gold or elixir for a 6-hour period. Repeated across six mines for three days: 1368 gems ($13.68)
You can use gems to rebuild your armies instantly, so you can keep raiding and racking up loot without anybody having a chance to steal their money back via the handy “Revenge” button. Assuming 50k elixir to rebuild and 50k profit per raid: 2760 gems ($27.60)

And once you’ve finally earned those 2 million and clicked the “Build” button, it takes another 8 days to build the thing — unless, that is, you fork out another 1123 gems ($11.23) for instant completion. It’s little wonder most players start “to gem” (it’s a verb in Clash of Clans parlance) by the time they reach Town Hall level 7 or so, the stage when most costs are measured in millions and building virtually anything unassisted takes days. Jorge Yao, the game’s undisputed champion, figures he has spent north of $2500 in real money on buying gems, and according to back-of-the-envelope calculations, the cost of fully fitting out your virtual village is on the order of $5000 when you include walls. It’s little wonder the top clans leaderboard is full of players like “>< Royal ><” from Kuwaiti clan “Q8 FORCE” and clan UAE’s “khalifa” (presumably from Bahrain’s ruling House of Khalifa).

Your Pain is Supercell’s Gain

Unsurprisingly, the entire game has been warped in subtle ways to encourage buying and using gems.

For example, in your average real-time strategy game, you have fine-grained control deploying and directing your troops, and units that survive can be used in the next battle. Not so in Clash of Clans: once deployed, units fight according to their hardcoded strategy (most commonly the harebrained “bash closest building”, regardless of what it is or who is firing at them), and every unit deployed disappears at the end of the battle, even if they are victorious. This means it’s essential to rebuild huge armies and to attack with massive force every time. And since building those dragons can take several hours, during which time you’re wide open to attack, there’s another massive inducement to solve the problem with a few gems.

Probably the most blatant case of tilting the table is the recent introduction of “Dark Elixir”, a third in-game resource geared squarely at high-level players. Collected at the glacially slow pace of 20 units per hour, even in an raidless pacifist world it would take 21 days of waiting to accrue the 10,000 units needed purchase its main selling point, the Barbarian King, and the table is stacked further yet by subjecting collectors to 75% raid losses. Who wouldn’t pay $6 to skip the tedium and uncertainty?

In comparison, the “clans” of the name seem almost like an afterthought. Their primary function is to be a gifting circle, where players donate units to others in their clan, and receive units in return. And that’s it: clan players cannot share gold or elixir, much less gems. But they do provide another handy lever of extra social pressure to ensure you log in regularly, since clan troops defend your base, die when attacked and can only be received on explicit request, and since most clans enforce minimum per week donations and kick out “freeloaders” who have not paid their dues.

But It Could Be Worse…

Some credit where credit is due: unlike Zynga’s notoriously annoying games, Clash of Clans does not require Facebook signup, cram the game full of ads, spam you and your friends, or pimp your personal information to random third parties. And while you’ll be reminded that “Hey, you could use gems for this” whenever you try to do something you can’t afford, if you stay within your means and have the patience of an ascetic saint, you’ll never even get asked for money.

Conclusion

At the end of the day, my feelings towards Clash of Clans are distinctly mixed. Being a penny-pincher whose in-game purchases have been limited to a single $4.99 gem pack, even that largely as a token of appreciation to the game’s makers, I can’t really complain about the hours of entertainment I’ve gotten in exchange. Yet I still can’t help but cringe as I run into all the ways the game is intentionally crippled to get you to pay up, and the way its Pavlovian triggers to come back for more operate on fear. Would Minecraft have been any fun if it required you to log in every six hours or you’d lose parts of your inventory? And how much more awesome would Clash of Clans be if the effort of squeezing every last cent out had been put into improving the game itself instead?

Down, down, down: Books, e-books and apps all trending to zero

“We expect DVD Subscribers to decline every quarter.. forever.”
— Reed Hastings, CEO, Netflix

Does your business model rely on selling paid content? Welcome to Mr. Hastings’s world.

Books

Many publishers continue to operate under the assumption that printed book sales are declining gradually or perhaps even plateauing. Unfortunately the data tells a different story: the decline appears to be accelerating. Here’s Nielsen Bookscan for the travel market:

That’s from the “Guidebook Category Report, Rolling, Period 13” for 2006 to 2012. The trendline is a simple polynomial (n=2) best fit, and if it’s accurate, the market will halve by 2015. And while that sounds drastic, it’s by no means unprecedented, as the sales of CDs did pretty much the same thing in 2006-2009.

Of course, the market’s not quite homogenous: Lonely Planet’s been beating the trend, mostly by continuing to invest in print and absorbing customers from the rapidly-disappearing Frommers. But that just makes LP an even-bigger fish in an ever-shrinking pond.

So can the white knights of digital paid content, e-books and mobile apps, save the publishing industry?

E-Books

Finding good data for e-books is a pain, so I ended up rolling my own: I grabbed Amazon’s Kindle Store Top 100 bestsellers lists since 2007, using snapshots from the Internet Archive that record the actual prices at the time, and computed average prices and the proportion of under-$5 titles, the vast majority of which are self-published. (Source code in Ruby here.)

Despite that December 2012 spike, the trend is clear, and while the decline looks gentle, my personal suspicion is that the trend line is too optimistic and that there’s a collapse looming. For one thing, the data above is only for best sellers, meaning new books by well-known authors who command a distinct price premium; the average price of an average e-book is both lower and falling faster. Yet even in the Top 100, the share of “cheap” books, the vast majority of which are self-published, is growing exponentially:

While I didn’t use their data directly, I owe tips of the hat to Piotr Kowalczyk, who wrote a very detailed report on the growth of self-published books on Kindle, and Digital Book World, which has been keeping tabs for the past half year (albeit looking only at the top 25). DBW also has a credible explanation for the spike, which boils down to publishers yanking up prices in the period before they had to start allowing discounting, and further reinforces that shrinking prices are the new normal.

Apps

The collapse of prices in the mobile app world has been even more drastic, to the point that outside an elite circle of bestsellers and the odd very specific niche, making money by selling the app itself is a pipe dream. (All data courtesy of 148Apps.biz and the Internet Archive.)

By the end of the year, the average app will cost under 99 cents, and even that’s pulled up by every $999 BarMax and wannabe in the store. The median price is already zero:

In other words, over half of all apps are already free. On current trends (and look how beautifully that line fits the data!), that will be over 80% within two years. This is for the “premium” Apple Store widely opined to have less stingy customers; the equivalent figures for Android will be even more brutal.

What to do then? The only answer is to figure out a way to make money that doesn’t involve readers paying for content. Here are some ideas:

Farewell Lonely Planet Melbourne, hello Google Sydney

At 18, I spent the summer delivering mail at minimum wage minus 15% (it was “training”, you see), and promptly blew my meager savings on a frenzied one-month Interrail trip through Europe. When my parents read through the angsty, near-incoherent notes I’d scribbled into a diary while waiting for trains in Holesovice or Ljubljana, complaining about expensive yogurt and Hungarian orthography, they ruffled my hair the same way I now praise our two-year-old for going potty and said “This is amazing! You should go work for Lonely Planet!”

“Ha”, the cynical teen thought. “Fat chance of that ever happening.”

Sri Veeramakaliamman Temple, Little India, Singapore

Yet 15 years later in Singapore, as I sat warming my hands over the dying embers of Wikitravel Press and glumly contemplated a return to the grim meathook world of telco billing systems, I received an e-mail from Lonely Planet. A few days later, I took Gus to Komala Vilas for roti pratas, and he outlined the vision for what would become the Shared Publishing Platform and why they could really use a travel wiki kind of geek for it. A few weeks later, I was in the Melbourne suburb of Footscray, staring at the world’s largest accumulation of travel knowledge in the Void and pinching myself: “Holy crap. This is for real, I’m standing inside the HQ of Lonely effing Planet, and these people want me to come work here.”

Industrial Science Laboratory, U. of Tokyo, Japan

Now I sit here in equal disbelief, voluntarily saying farewell to the best company and best team I’ve ever had the privilege of working for, and that’s not just the kind of hyperbole expected for these public farewells. The past three years of replacing a jet’s engines in mid-flight have been an intense learning experience, and the work is nowhere near done, but unlike the company’s three previous attempts, it’s now over the hump. All authors are now writing directly into the content management system, where editors and curators weave their magic, with printed books, e-books, apps and the website being pumped out the other end, and Lonely Planet can now start fully focusing on its shiny digital future.

And me? I’m joining Google’s Geo team in Sydney, where I’ll be working with the world’s most popular travel application, Google Maps.

I plan to continue to write this blog, although there will be less idle speculation about what the Big G is up to next and less of a focus on the print publishing business I now depart. That said, my next post ought to give some food for thought to those in the industry, so don’t unsubscribe just yet!

Gyrovague

vaguely heretical musings on travel and technology