The Opal or Not Report: The hidden $98 (or $54) million fare hike

Krinklewood Carriages, Pokolbin, NSWAbout half a year ago, I launched the Opal or Not website that compares the fares of Sydney’s new Opal transit smartcard and the paper tickets it replaces.  As I’d hoped, this quickly went viral, bringing attention to Opal’s hidden costs and ending up in the Sydney Morning Herald.

Now getting my mug in the SMH is all well and good, but my end goal was to “make Transport for NSW to come up with a saner fare scheme”. Instead, TfNSW sent me a rather bizarre good cop/bad cop letter telling me to they’d love to cooperate, but implying they will sue me if I don’t stop using the Opal logo.  When I offered to play ball, changing the logo as a token of good faith, they sent me a 26-page report about all the things with my site that they were unhappy with; quite handy, as you’ll see later, but not exactly the kind of action I was hoping for.

Meanwhile, as they prattled on about how they’re committed to transparency yadda yadda, another arm of the same bureaucracy has been steadfastly stonewalling my request for a copy of that 7% report.  At time of writing, after half a year of increasingly absurd effort, they have yet to confirm in writing that the “7%” report they keep talking about even exists.  And maybe it doesn’t, because in other places they claim the figure is 4%!

Well, fine.  If TfNSW is not going to release their report, based on their vague projections, I’m going to write my own report, based on the cold hard data of the over 130,000 fare comparisons made on the site.  And here it is.

The short version

Who will pay more with Opal? Cost of Opal to Sydney commuters?
Transport for NSW report 4%? 7%? [citation needed] They’re not telling
Opal or Not report, model A: optimal tickets
61% [1] $98 million/year [1]
Opal or Not report, model B: estimated tickets
26% [1] $54 million/year [1]

Here’s how you can help fix this.

The long version

Wagon, Felton, CA, USA

Now there’s two basic requirements for Opal or Not’s data for it to be of any use: it has to be representative, and it has to be correct.

So the first question is, are the 130,000+ fare comparisons made on Opal or Not a representative sample of the Sydney commuter, instead of (say) an escapist chimpanzee recalculating his fare from Taronga Zoo to Bondi over and over again?  In terms of sheer sample size, it should certainly qualify: 812,000 people use Sydney’s public transport on an average weekday, so even if we discount every other comparison, we’re reaching a good tenth or so.  For comparison, the Bureau of Transport Statistics Household Travel Survey 2011/12 (HTS) is based on the survey responses of about 3,000 households.

A more useful comparison of the quality, though, is to compare Opal or Not’s mode distribution against the HTS.  This isn’t entirely straightforward, since Opal or Not compares door-to-door ‘journeys’ (eg. bus & train) while HTS counts single ‘trips’, but by making the not unreasonable assumption that a bus & train journey equals one bus trip plus one train trip, we get a 70/26/4% split for train/bus/ferry, while HTS has 59/37/4%.   This is unsurprising, since Opal for buses remains very much a work in progress, but it does skew the results a bit since the average train commuter loses more from Opal (an extra $65 a year, to be precise) than the average bus commuter.  Long story short, if we adjust the mode split to match the HTS, this shaves $5.7m off the paper advantage.

“But wait”, I hear you say, “even if the sample population is representative, what if the calculation itself riddled with errors”?  To make sure all potential problems are covered, I’m going to go through TfNSW’s llst of 18 “issues” with the site and analyze their impact on the results one by one.  You might want to a brew a stiff coffee before you read on.

1. Incorrect fares shown for some train journeys

Opal or Not computes train distance bands based on actual distance, not wacky-world fare distance, which can throw off estimates by a few km.  This means that users whose trip lengths are right on a fare boundary (10 km, 25 km, etc) may be quoted fares for the wrong fare band.

However, Opal and MyTrain bands are defined identically by distance.  This means that, even if a user is quoted the wrong zone, they are quoted the correct fares for those zones, and the comparison is thus fair.  No adjustment needed.

2. Train journeys that cannot be calculated

There are some cases where the underlying Google Directions API only offers buses, because they’re much faster, so Opal or Not cannot estimate the train distance.  But if the site can’t calculate the route, it can’t get the comparison wrong either.  No adjustment needed.

3. Incorrect morning peak times for Intercity trains

Opal or Not assumes that morning peak is always 7am to 9am, when it’s actually 6am to 8am when traveling from Intercity stations.  However, the website does not ask for exact times, it only asks for time bands, so the relatively few users affected can self-adjust.

But yes, those who do enter 6:30 Intercity trip as “before 7 AM” will get quoted an off-peak Opal fare when they should be paying a peak fare.   Since there are more of these people than there are 8:30 Intercity travellers who get quoted peak for off-peak, the net effect of the error is in Opal’s favor: the true cost of Opal is higher.

4. Weekend off-peak fares not included

Opal or Not does not implement train off-peak fares all day on Saturday (or Sunday) for either Opal or paper tickets, instead applying the weekday off-peak rules. This has two effects:

  1. Peak period travel on Saturday is incorrectly calculated at weekday peak fares.  This inflates the cost of both Opal and paper equally, producing no net effect.
  2. Off-peak travel is calculated with weekday off-peak periods.  This gives an unfair advantage to Opal, since this grants ~26% of travelers the discount, vs. ~16% for paper travel.  (The actual figure is, of course, 100%.)

All told, the net effect is again in Opal’s favor.

5. Description of MyBus and MyFerry tickets as zonal

This is a wording quibble: MyBus and MyFerry tickets are based on distance travelled, and strictly speaking should not be called “zones”, since that usually implies geographical areas.  (By comparison, MyMulti is a true zonal ticket.)  Fair enough, but obviously no impact on fares.

6. Change in method of calculating bus fares

The website currently assumes that that MyBus sectors map one-for-one to Opal’s distance-based fares, which is not the case. However, without access to the exact sector data, which TfNSW does not make public, it’s not possible to compute what difference this makes in practice.  Opal claims some fares will be cheaper, but the SMH has some cases of people whose fares nearly double, and as we know, while TfNSW probably has their own estimates for this, they refuse to release them.  I’ll give them the benefit of the doubt and call it a draw: no impact.

7. Incorrect MyMulti ticket recommended for some journeys

Opal or Not’s determination of what MyMulti ticket to use combines bus and train distance bands, meaning issues 1 and 6 (train and bus distances) apply and users may be quoted the incorrect MyMulti zone.  Again, since there is no systemic bias in either direction, this alone does not bias results.

MyMulti zone shiftingThere is, however, one important source of error in the MyMulti calculations: they assume that train fares bands map directly to MyMulti zones.  This is correct for trips originating or terminating from the CBD, but will incorrectly compare to a MyMulti 1 for someone traveling from (say) Blacktown to Parramatta, actually MyMulti 2.  The diagram to the left attempts to demonstrate this, with the top ‘ovals’ representing how Opal or Not calculates trip costs and the underlying colored bands showing how the actual MyMulti zones apply.  The bits marked “+1/2″ show undercharging, whereas “-1/2″ shows overcharging: for example, somebody traveling from Hurstville to Parramatta would have been computed as MyMulti 3, when actually a MyMulti 2 would suffice.

We can attempt to correct for the error by applying the known distribution of train travel distances to major destinations in and out of the CBD (both courtesy the Compendium of Sydney Rail Travel Statistics, 8th Edition, RTS), times the number of train travellers using MyMultis and finally the cost incurred (or saved) from using the wrong MyMulti zone.  The math is fairly hairy and makes a few rather generous assumptions (eg. that each station’s patronage follows the general distance distribution and that the Inner Sydney statistical area is close enough to the rail CBD fare zone), but because the under- and overcharging tend to balance out, the net effect is fairly minor: $300k off paper tickets.

For completeness, I’ll note that the same issue theoretically applies to ferries.  However, since ferries are a relatively minor form of transport and the overwhelming majority of ferry travel is to/from the CBD, the approximation used by the website is more than good enough for our purposes: Ferry 1 = MyMulti 2, Ferry 2 = MyMulti 3.

8. No provision for customers with complex travel patterns

Opal or Not is not a general-purpose fare calculator, but a single-purpose tool geared solely at making it easy to check how Opal will affect the cost of your commute.  Per the HTS (Table 4.3.3), we know that commuting to work and school/uni account for the vast majority of public transport usage, so this means the results also represent the vast majority of Sydney’s public transport users.  And once again, if the site doesn’t support a given travel pattern, then it’s not giving inaccurate results, it just doesn’t support it.  No adjustment needed.

9. No consideration of the customer’s current ticket

Locomotive #7 "Sonora", Felton, CA, USATfNSW, and quite a few of my readers, think it “unfair” to compare Opal to monthly or quarterly tickets, for two distinct reasons: 1) people do not use them very much, and 2) they are not a good fit for some people.  Let’s analyze these in detail.

First, do people actually use them?  TfNSW loves to trot out the statistic that only 4% of ticket sales are periodicals, but this is highly misleading: if person A buys a yearly ticket once a year, and person B buys two singles every working day, only 0.2% sales will be for yearly tickets, even though 50% out of A and B use them. The best statistics for actual usage that I’m aware of are in RTS Table 20, which extrapolates from ticket sales and covers trains only: according to this, periodical tickets (weekly or longer) account for 50% of usage, with 8% using monthlies or longer.

At the end of the day, though, this is kind of irrelevant: Opal or Not is all about showing people the best possible fare, and I can only assume that its users are rational economic actors who choose the cheapest fare.  In the same way, I’ll give Opal the benefit of the doubt and assume that its users are religious about tapping on and tapping off, so they never get accidentally charged the maximum fare — a lip-smacking $8.10 if you forget to tap off the train!

The second objection is that people are rational and choose not to opt for periodicals because, when accounting for vacation etc, they cost more.  However, since a train quarterly covers 13 weeks of travel but costs the price of 10 weeklies, virtually everybody who uses the train would actually be better off buying them, even if they take two weeks off that quarter.  For buses and ferries, TravelTens are unequivocally more flexible and cheaper than Opal if you have even slightly irregular commute patterns.

What’s more, comparisons for monthly and quarterly tickets were introduced only after launch, meaning the initial 5,000 or so users did not benefit from them, and a rounding error meant that the next 10,000 or so had quarterlies incorrectly computed as 12 weeks instead of the correct 12.85, again a bias in Opal’s favor.

All in all, I’m going to call it a draw and say no adjustment needed.  If you have better stats, or thoughts on how to quantify this, I’m all ears.

Update: By popular demand, I’ve taken a stab at adjusting the pricing comparisons to be in line with ticket usage from RTS Table 20.  This is rather rough and ready, because Opal or Not does not collect the user’s actual ticket types or record non-optimal paper fares, but long story short, this shaves $44m off the paper advantage and converts 280k people into the Opal camp, meaning only 26.46% pay more with Opal.  This, mind you, is still about 4x more than what TfNSW is claiming, and it goes to underline what we’ve seen from previously: those who win from Opal win a bit, while those who lose generally lose a lot.

The previous calculation is now Model A, and this version is now Model B.

10. Opal trip advantage not always applied

More specifically, the site does not cater to the case where there is less than 60 minutes between the trip out and the trip back. Again, since Opal or Not is a commute comparison tool and most people work/study more than 60 minutes a day, this scenario is irrelevant.  No adjustment needed.

11. No provision for journeys with multiple transfers

Setting aside ferries, there are three possible realistic multiple transfer combinations: train/train/bus, bus/bus/train, bus/train/bus.  The first and most common is already catered for, since Opal or Not can compute the fares between any two train stations, including transfers.  For the second, Opal would be cheaper if you were currently foolish enough to purchase separate tickets, but as we know MyMulti is nearly always the better option even for a single bus/train transfer, much less two; not covering this thus tilts the balance in Opal’s favor.

So that leaves the third option, bus/train/bus.  Getting good data on this is hard, but Bus Users in Sydney, 2002 p5 (not a typo, that’s the latest available) tells us only around 7% of bus journeys involve two buses, and Rail Travel Statistics 2012 section 3.2 tells us 17.7% of train users transfer to bus, from which we can determine a conservative upper bound of 1.2% travellers affected.  Not that this helps Opal’s case any, as MyMulti will almost certainly wipe the floor with Opal for these poor people, but I’ll graciously not dock Opal’s score for this.  In Opal’s favor.

12. Many NSW TrainLink Intercity stations not included

That’s because they were not covered by Opal at the time the service was launched, although support has since been added both to Opal and Opal or Not.  Once again, if you can’t compare it you can’t get it wrong, so no impact on accuracy.

14. $2.50 Sunday cap not included

Neither Opal’s all-day-for-$2.50 offer nor the more limited Family Fun Day ticket it replaces is covered by Opal or Not.  While this is clearly a win in Opal’s column, for the vast majority of Sydney’s commuters it’s a nice bonus, not an actual commute cost-saver.

The second complication is that since the Travel Reward is accounted for, this applies to only people who travel on Sundays without accruing 8 weekday journeys.  Since Opal or Not does not ask about days of week, and I’m not aware of any TfNSW data for this, quantifying the size of this group is difficult.

All in all, no adjustment made, but underlying bias is in paper tickets’ favor.

13. Domestic Airport and International Airport stations not included
15. Opal Child/Youth fares not included
16. Opal Senior/Pensioner fares not included

Opal or Not is a weekly commute calculator, all these are explicitly out of scope and do not affect the accuracy of results.  No adjustment needed.

17. Daily tickets not included

The Opal $15 daily cap is included in the site; however, the $23 MyMulti day ticket is not.  This skews the results in Opal’s favor.

18. Fares for services and classes of Opal cards that have not yet been announced

I had to read this one twice: TfNSW apparently considers it a flaw that Opal or Not does not cover Opal fares that have not yet been announced?  I’ll be happy to address this once somebody lends me a time machine.  No adjustment needed.

We’re done!  Almost, anyway, since magnanimous soul that I am, I’m going to include a large source of error that TfNSW did not notice:

Bonus. Off-peak fares on trains

Off-peak fares are frequently cited as a major win for Opal, and while Opal or Not now supports them, this option was not available initially, affecting the first 26,925 comparisons.  Looking at the additional statistics kept for off-peak fares, we can see three general trends:

  1. The 15% of train travellers who travel outside both peak periods are unequivocal winners, Opal saving them around $200/year.
  2. The 20% who have one peak and one off-peak trip come out a wash, gaining or losing less than $100/year.
  3. The 65% who hit both peak periods pay on average $270/year more.

So while the majority still lose, the gains of the small group of winners are so large that they translate to a 30% decrease in the overall cost.  Adjusting the data accordingly (see spreadsheet), this docks some $14 million off paper’s advantage.  Better start waking up before 7 AM!

Conclusion

Roaring Camp Railroad, Felton, CA, USAOf the 20 potential issues identified by TfNSW and yours truly:

  • 4 bias the results in paper tickets’ favor, and are accounted for by adjusting down by $20m,
  • 3 bias the results in Opal’s favor, which means Opal’s actually even costlier than this report indicates, and
  • 13 have no measurable impact.

In other words, this report bends over backwards to give Opal the benefit of the doubt, and it still comes up a cropper to the tune of $54 to 98 million dollars.

 

Call to action

So what should Transport for NSW do to fix Opal?  I have two simple suggestions:

  1. Allow bus, train and ferry transfers.  A wide array of wise men all agree: if Melbourne, Perth, Brisbane and Adelaide can all do it, there’s no sane reason why Sydney can’t. Penalizing modal transfers makes Opal a huge leap backwards for Sydney transport.
  2. Lower the base fare and drop the Travel Reward.  The “Travel Reward” is supposed to encourage using public transport, but it ends up discouraging it for many people, and only encourages gaming the system instead.

If you agree, let the Minister for Transport know.  Gladys, the buck stops with you here: it’s not too late to fix this, and turning Opal from a PR disaster into a win for your Sydney voters will only take a stroke of your pen.

And if you’re a commuter who wants to hedge your bets, lock in your savings before Opalcalypse on September 1st, 2014, and buy a quarterly or yearly ticket today.

 

How SMS set back the mobile internet by ten years

Mari Matsunaga, the yin to Steve Jobs' yangStop me if you’ve heard this one before: an obsessively perfectionist corporate maverick defies conventional wisdom and launches a radical new mobile “i”-phone that offers built-in email integration, an app store with simple payments and, finally, a decent way to surf the web, all tied together with revolutionary ease of use.  Sales go gangbusters, the company mints billions, mobile Internet usage skyrockets and competitors are left scrambling to catch up.

I am, of course, taking about Japanese operator NTT DoCoMo’s launch of i-mode in 1999, spearheaded by Mari Matsunaga (pictured), who clawed her way up in the intensely male world of Japanese business and whose insistence of putting user experience first almost scuttled the project several times.  i-mode phones were equipped with an “i” button that opened up a browser in one click, allowing users to surf in a curated portal of i-mode sites, some for subscribers only, most free but with in-site purchases charged directly to the user’s mobile bill, or to access the wider internet by entering URLs manually.  Downloadable “i-appli” Java apps were launched shortly thereafter.  Each phone also came with the “i-mail” email service that was completely interoperable with the general Internet (and other Japanese phones), delta some fairly drastic limitations on message size since we’re talking 320×200 screens and download speeds measured in kilobits.  And since this was all running on packet-switched data, the billing scheme couldn’t have been simpler: pay per byte sent or received.

“So”, I hear you ask, “if this was so awesome, why have I never heard of this?”  There are many theories, but I’m going to lay out the case for an unconventional one today: it’s all SMS’s fault.

Why SMS was awesome

SMS is the GSM mobile standard’s Short Message Service, and by telco standards it has a venerable pedigree, being first dreamed up in 1984 (yes, that’s thirty years ago).  Fundamentally, SMS is a hack; a glorious hack, to be sure, but a hack nonetheless.  SMS is built on the Signalling System 7 (SS7) protocol, designed in the 1970s for controlling landline calls.  Unlike most previous telco signalling systems, which controlled calls by inserting tones directly into the voice stream and let phone phreaks run wild with blue boxes, SS7 has a separate control channel used primarily to signal when calls start and end, means it transmits blank frames the rest of the time.  SMS squeezes text content into those empty gaps, which explains both why telcos were so fast to adopt it — it needed precisely zero additional hardware to roll out — and where that famous size limitation comes from: the frame size is exactly 140 octets, which is enough for 160 7-bit characters.

The other reason telcos love SMS is the money.  Remember, if you’ve already got a GSM network, SMS costs nothing at all to transmit.  (Yes, there is some cost involved in setting up SMS store-and-forward centers and interconnects and whatnot, and I made my living for close to a decade working in the field, but compared to actually building a network, this stuff is a rounding error.)  However, people in the 1990s were already well-trained to pay for each call, letter or fax, so charging (say) a flat dollar a piece for each SMS still sounded like a bargain, and the telcos, buying SMS for $0 and reselling them at $6500/MB, weren’t about to disillusion them.  In 2013, US operators alone minted 21 billion dollars from SMS, and you can still today get charged that $1 a piece if you happen to send an SMS while roaming in the wrong place.

Crazy Frog Crazy HitsSo far, so awesome (if you’re a telco, at least), and Worse is Better when it comes to victory in the marketplace.  But SMS’s sheer hackiness made it an evolutionary dead end, despite increasingly Sisyphean efforts to build more and more complicated infrastructure on top.  For example, while SMS was originally meant purely for transmitting text, some clever noggins came up with the idea of using six of those precious 140 bytes for a User Data Header that, bit by bit, could say things like “I am a logo!”, so you could squeeze in the 1008 bits needed for a 72×14 monochrome bitmap representing the Vodafone logo.  Or if you were so greedy that 134 bytes wasn’t enough, you could use a few more of those bits to say “I’m piece 3 of 7!”, and get the phone to concatenate those SMSes together into a chain that might contain as much as a kilobyte of info, just enough for a Crazy Frog ringtone (that’ll be $4.99, thank you).  Unfortunately, SMS is a UDP-style best-effort-only service with no guarantees of delivery, meaning that the longer the message, the more likely it was that some pieces would fall by the wayside.  And while there is a “delivery notification” bit that’s supposed to confirm a message was received, and whose absence can trigger redelivery, those notifications are also SMSes and are themselves often dropped along the way.  Remember when, in the olden days, you’d occasionally get 3 or 5 copies of a message?  Somebody was dropping your DNs.

Meanwhile, in Japan, they didn’t have SMS, because they didn’t have GSM.  The Japanese telcos naturally noticed that this SMS thing was taking off in Europe and built their equivalents for the Japanese 2G standard PDC, but the operators never managed to sort out talking to each other, and by 1996 or so they just said “fuck it” and deployed good old e-mail instead.  This was easy enough technically, since PDC offered packet-switched data and WAP 1.0 didn’t, and easy enough financially, since they didn’t have to worry about cannibalizing non-existent SMS revenue.  The standard undiscounted data rate in Japan circa 1997 was around 0.3 yen per kilobyte, and because a short e-mail is about 3 kB (headers and all), it cost around one US cent to send/receive one to/from any device in the world that groks SMTP.

Why MMS is so bad

In 2000, NTT’s arch-competitor J-Phone launched Sha-Mail, the world’s first mobile photo messaging service, which (then as now) was an instant killer app quickly duplicated by NTT as “i-shot”.  This was implemented using standard e-mail attachments and charged for at packet data rates, meaning around 10 yen ($0.10) for the 30 kB images taken by camera phones circa 2001, and phones and pictures sold like hotcakes.

Back in Europe, the WAP hype wave had just crested and crashed when it turned out people weren’t willing to pay one euro per minute to surf an unusably sluggish crude approximation of the few Internet sites that had bothered to completely rewrite themselves with WML.  Yet the first GSM packet data scheme, GPRS, was starting to gain traction and it started to dawn on operators that SMS just wasn’t going to cut it for images.

The MMS network architectureSo they came up with the replacement: MMS, the Multimedia Messaging Service, and you can already sense what a kludge it would be from that very first buzzword, which has killed all it touches.   To grossly oversimplify, MMS works by sending magical SMSes called “WAP pushes” that tell the receiving phone to open a WAP connection to an MMS server, from which they then proceed to download what are essentially e-mails crammed full of bizarre custom MIME parts, which are then rejiggled into an approximation of the original message, defined with a laughably overcomplicated custom slideshow language not too distant from Powerpoint.   For additional hilarity, the protocol provides no way of checking what the receiving device supports: instead, it’s the operator‘s job to attempt to manually track which phone number corresponds to which phone model.  (Thought experiment: imagine if the Internet worked this way, and instead of User-Agent strings, you had to phone up your ISP every time you changed device, browser or window size.)  This grotesque Rude Goldberg contraption is defined by eleven (11) different specifications, imaginatively named MM1 through MM11, each laying out the edges of one piece of the puzzle, and (unlike SMS) requiring very considerable resources to deploy.  Unsurprisingly, nobody could agree on how to actually implement the damn thing, making MMS hysterically unreliable to the point that, in A.D. 2014, I still consider it a minor miracle to successfully transmit one.

But MMS did have one feature that operators considered very, very important: it let them continue charging per piece, although many operators socked you for data usage as well.  So if you were visiting Mexico and Aunt Tilly sent you a 300 kB picture of her cat, she’d pay a flat $2 or something to send it, and you’d pay $30 to download it at $100/MB.  Or, more likely, she’d pay $2 to send it, but her phone or your phone or your home operator or your roaming network or any of the gazillion boxes talking incompatible variants of MMn along the way would barf and lose it.  And then Aunt Tilly would get pissed that you didn’t reply to her message, and never touch MMS again, which is in fact pretty much what happened.

Why i-mode never took off elsewhere

3G_NormalNow operators around the world did notice that this i-mode thing looked like a license to print money, and 17 of them ended up eventually launching it around the world, offering i-phones (small p) with little i-buttons that opened i-sites that charged i-ncredible amounts of money and otherwise looked a whole lot like the Japanese i-mode i-cosystem.  Yet every single one of them failed.

Partly this was technical: not everybody/everywhere had GPRS, and i-mode over plain WAP, which essentially involves a dialup connection every time you press a button, is atrociously slow.  Partly this was timing: so many fingers were burned in the bonfire of WAP malinvestment that few were willing to go all-in on another whiz-bang mobile invention, making the chicken-and-egg problem of getting sites to create i-sites tagged with i-mode’s “compact HTML” extensions that much harder to hatch.  Partly this was the cutthroat nature of the market: at least one operator I know of launched i-mode more or less entirely as a cynical ploy to extract better terms from Nokia.  But mostly it was just because none of those operators had the guts to go all-in on i-mail, because they were all so wedded to sucking on the SMS teat and could not bear to exchange $1 now in easy SMS profit for $0.01 forever in data revenue.  And the straw that broke the back of the few brave operators that did launch it?  The fallback mechanism for sending e-mail to non-i-mode phones was…  MMS.

Instead, it was BlackBerry who grabbed that market opportunity in the West, letting you read business e-mails on your CrackBerry and later launching BlackBerry Messenger (BBM), giving its own users a first taste of rock-solid, instant, all-you-can-eat messaging.  What’s more, it wouldn’t be too far-fetched to argue that BBM was such a hit in North America only because SMS was so bad there: half the operators in the US and Canada were using CDMA, which doesn’t natively support SMS.  But this was a completely proprietary protocol, not an ecosystem others could use, and BlackBerry did its best to make sure BBM only worked properly on their own devices, also opting for easy money now over a chance of future relevance later.

Jobs delivers us the keys to the new kingdom

So that was the state of the rest of the world until 2007: SMS ubiquitous for lack of interoperable alternatives, MMS strangled in the cot, BBM minting money in its little niche.  Only then came along the iPhone and the Cambrian burst of mobile internet evolution, which saw Android make mobile e-mail a commodity and the rise of alternative messaging systems like Skype, Facebook, Snapchat, WhatsApp, etc.  In telco lingo, equal parts dismissive and terrified, these are to this day called “Over-the-Top” (OTT) because they finally scaled over that charge-per-piece wall put in place by SMS in 1984, and brought the rest of the world to where Japan was in 1999.

Meanwhile, in Japan, Matsunaga was railroaded out of NTT barely a year after launching her creation, and is now a director at Bandai. NTT lost over $10 billion on its i-modal overseas misadventures but grabbed nearly two thirds of the Japanese market, with around 50 million i-mode users at its peak in 2008.  But they refused to sell the iPhone until 2013, by which time it had lost over 20% of that market share to nimbler rivals like SoftBank, which went on to buy Sprint.

And that’s where my story ends.  Now, I’m not saying i-mode had any real chance to rule the world, GSM was not a good fit technically and SMS was already entrenched by the time it came along.  But if SMS hadn’t existed, the world would almost certainly have had an i-mode equivalent much sooner, and you can only wonder how the world would look today if it had.

Sydney’s screwed-up smartcard, or why I wrote “Opal or Not”

The backstory behind Opal or Not.

Last year, Sydney started trialing its new “Opal” transit smartcard.  As a regular commuter on Sydney Ferries, the first service to roll out Opal, I awaited its arrival eagerly.  After all, they couldn’t possibly screw it up worse than Melbourne’s Myki, whose ludicrous cost overruns and sheer technical incompetence I had witnessed first hand earlier.

Screen Shot 2014-02-21 at 10.39.12 PM

Alas, while Opal has indeed been lighter on the government’s purse and is mostly capable of registering card taps, Transport for NSW still managed to completely stuff up something that Melbourne didn’t: the fare structure. For many users including me, Opal is much more expensive, in my case translating to $332.80 more every year for the same commute.

Virtually every smart card in the world prices individual trips lower than the equivalent single fare, meaning it always make financial sense to use the card.  Not Opal: for ferries and buses, a single fare costs more than a trip on the TravelTen paper ticket, and only on your 10th trip of the week does the cap finally make Opal cheaper again.  And if, like me, you occasionally bike to work or work from home, meaning you use the ferry or bus less than 10 times a week?  You fall into the Opal Fail Zone, shown in red above: that’s the premium you pay for the privilege of using Opal.

Screen Shot 2014-02-21 at 11.05.16 PM

But believe it or not, I soon found out that there were others even worse off than me.  Say you live in Dee Why, take a bus to Wynyard, and switch to the train to Central.  (Substitute with bus/train combo of your choice.)  Because Opal has no replacement for MyMulti, your commute is going to rocket up $728 a year, even if you travel five days a week!

chart_2(2)

Just look at that thick red slab of Opal Fail: if you’re commuting by bus and train, unless you’re doing it exactly 6 times a week, it never makes sense to switch to Opal.

Is it easy to figure this out?  Hell no, it takes an intimate understanding of Sydney’s convoluted fare structure and a whole lot of flipping between browser tabs to come up with the actual numbers.  The Opal website has some contrived examples, every single one of which shows Opal as cheaper, but lacks even a basic Opal fare calculator, never mind any way to compare to non-Opal fares.

Now I could have written feedback to Opal, which would have gotten me a form letter response with sneering thanks before getting chucked in the bin.  Or I could have written an angry blog post (well, I am writing one), which with some luck would have been retweeted a few times before being overtaken by Justin Bieber’s latest drunken antics.  But neither would have had any real impact.

Instead, I wanted something that would:

  1. Let people see exactly how the switch to Opal will hit their wallet
  2. Collect statistics on how many people are positively or negatively impacted by Opal, and by how much
  3. Ultimately make Transport for NSW to come up with a saner fare scheme that encourages all public transport use and does not penalize transfers.

So I spent a few evenings coding up a fare calculation engine (and Jesus Christ that was a pain, just look at this shit) and a few more slapping a web interface on top, and the result is Opal or Not.  Here’s hoping it was worth it!

Half the Donut: Why an entrepreneur earning $100k gets to keep over $99k in Singapore but under $57k in San Francisco

The donut of doom: Total earnings vs total tax in California

You’re about to quit your job and start living the life of a one-man entrepreneur. Living in your basement, you have no rent and no employees, and since you’re selling your skills you don’t need venture capita yet, but you’re still dreaming big. All things being equal, where should you base your business?

Tip: Probably not in the place where the taxman takes a red bite like this out of your income donut.

More specifically, in each of San Francisco, Helsinki, Sydney, Tokyo and Singapore, if your business earns $100,000 in net revenue and pays you a “ramen-profitable” $2000 monthly net salary:

  1. How much post-tax profit will your company make in a year?
  2. If all of this profit is paid out in dividends, how much does will you have left after taxes?

If you’re wondering why I picked those five cities, part of the reason is that it’s a nice geographical, cultural and political spread, but mostly it’s because I’ve either founded a company or worked for a company based in each of them.  The $100k net revenue/$2k net salary model, funding product development on the side, is basically what I modeled my one-man consulting company on back in 2006.

To jump straight to the answers, click here. If dividend imputation and municipal inhabitant taxes get you all tingly and excited, read on. Hardcore masochists who’d like to double-check my math are also invited to examine the gruesome innards of this Google Docs spreadsheet.

Disclaimers

This is a hypothetical exercise, so I’m going to simplify as much as I can, the accounting is still stupidly complicated and there’s a whole lot of cramming square pegs into round holes going on.  In particular:

  • I ignore all non-financial considerations. Visas, availability of talent, infrastructure, economic prospects, legal and political environment etc are all important real-world factors for locating a business, but out of scope for today.
  • $100,000 is net revenue, we ignore all expenses aside from salary and tax. In other words, $100,000 is what’s left over after business registration, accounting, stationery and whatnot, and we also assume that those costs are the same across all countries.
  • We assume sales tax does not apply.  This is not as unfair as it seems, since most countries exempt companies until they reach fairly high yearly sales thresholds and exclude online and/or international/interstate sales.
  • The business has no tax deductible expenses. This is particularly unrealistic for the United States, where 72,500 pages of federal tax code means creative deductions are a national pastime, but them’s the breaks.
  • Pensions are accounted for on a defined contribution (what-you-pay-is-what-you-get) basis, so $1 put in now is worth (at least) $1 later. This is true for Singapore, Australia, US 401(k)/IRAs and some Japanese corporate plans;  this is manifestly not true for US Social Security or the national plans in Finland and Japan, but for lack of a better measure we assume it is anyway.
  • Taxes are computed assuming that pension and health insurance contributions are tax-free, which allows me to ignore the distinction between employer and employee contributions.
  • The owner is a full-fledged local resident under 40 for the purpose of tax, pension, insurance etc brackets.
  • For the sole purpose of minimizing futzing about with exchange rates, for income tax thresholds etc I’m going to merrily assume that 1 USD = 1 SGD = 1 AUD = 1 EUR = 100 JPY. This is obviously not correct, but is not all that much worse than picking actual exchange rates that’ll be out of date in moments anyway.

Last but not least, this is a work in progress, a revision history is at the bottom of the article.   Now, let’s roll up our sleeves, channel the spirit of the late great Herbert Kornfeld, and balance this shit wit’ a quickness.

Round 1: Paying a salary

$2000 x 12 = $24,000, leaving $76,000, right? Not so fast. There are four main things to worry about here: income tax, pension fund contributions, payroll taxes and health insurance. Both pensions and health insurance are tricky since our countries’ systems vary so widely, so we’ll just attempt to standardize the legislative minimum. And since all these fees are usually percentages of salary, we have to work our way backwards starting from desired post-tax income to get to the employer’s cost.

Golden Gate BridgeIn San Francisco, I’m going to cheat a bit and outsource the otherwise horribly complex income tax computation to the MIT Living Wage Survey, which figures that for a single adult, it takes pre-tax earnings of $26,692 to have $1,929/mo left over after tax, an effective tax rate of 13.28%.  Optimistically assumes that rate stays the same at $2,000/mo, we now need $27,675.  Next, we add in a 6% employee pension contribution with 6% employer matching (+$3203) and boost medical from $149/mo to a post-Obamacare estimate of $368/mo (+$2628).  On top of this, employers have to pay 6.2% of wages for Social Security, which I’ll lump as a “pension” for the purposes of this article; 1.45% for Medicare; 3.4% for CA unemployment, plus 1.2% and 0.1% of the first $7,000 only for federal unemployment and employment training tax respectively. (Huge props to ZenPayroll.)  In a rare bit of good news, SF’s payroll tax (1.5%) only applies once total salaries exceed $150,000, so we can ignore this.  We thus arrive at $36,773.

Ice, HelsinkiIn Helsinki, personal income tax is so complicated the only sane way to compute the effective rate is to punch numbers into the official tax calculator.  A gross income of €29,000 and zeroes for everything else, including being a godless pagan who avoids church tax, nets income of €23,978, for an effective rate of 17.32%.

As a >30% shareholder of his own company, our hero can apply the lower “YEL” pension employer contribution of 17.55% for two years.  On top of this, we have the employee side pension contribution of 5.15%, 2.04% mandatory health insurance, and 0.80% employer/0.60% employee unemployment insurance.  We thus arrive at €36,615, which is, rather incredibly, a hundred bucks less than SF, and this gets you a cradle-to-grave Scandinavian welfare state!

Expressways in Shinjuku, TokyoIn Tokyo, national income tax (shotokuzei) is 5% for the first Y1.95 million and 10% above, plus a flat 4% prefectural and 6% municipal tax (which combine to form jūminzei, resident tax), which works out to 16.6%.  For pension, you can pick the national plan (kokumin nenkin) at a fixed Y14,980/month regardless of income, or a standardized company pension (kōsei nenkin) at 17.120% split evenly between employer and employee; being cheapskates, we pick the national plan.  National health insurance is 80% of municipal and prefectural tax paid, which in Tokyo equates to 9.97% split between employer and employee. Phew!  (Dōmo arigatō to Japan Consult.)   Final damage 33,448 hectoyen, and I’m delighted to find out that I’m apparently the fifth person in the history of the Internet to use that lovably awkward term.

Sydney Opera HouseIn Sydney, the Tax Office says the first $18,200 of income are tax-free and the next bracket until $37k is 19%, for an effective rate of 5.36%.  The minimum pension (superannuation) contribution is 9.25% and Medicare (public health insurance) levy is 1.5%, and since our total income is under $84k, we’re not required to pay the Medicare levy surcharge for not having private insurance.  Payroll taxes in Australia are administered at the state level, but New South Wales only applies its rate of 5.45% once payroll exceeds $750,000, so that particular bullet is dodged, and at $28,087, the end result is easily the lowest of the bunch.

Shophouses in Katong, SingaporeIn Singapore, income tax is easy: 0% for the first $20k, 1.4% for the next $10k (usually 2%, but they’re discounting 30% in 2013), for a scarcely believable effective rate of 0.24%.  Mandatory pension (CPF) contributions for employees under 50, which include a health insurance component, are 16% for the employer and the 20% for the employee, with a maximum contribution of $1800/mo (not applicable here). The only other obligatory cost is the Skills Development Levy at 0.25% of salary, capped at $11.25/mo.  This equates to $32,777.

We can now compare the total cost of employment, ie. how much it costs the company to pay the owner that $2000/mo salary:

Employer cost vs take-home pay

But while taxes disappear into the gaping jaw of Leviathan, pensions are retained by the owner in some form of another (see Disclaimer), so a more instructive comparison deducts salary and pension from total cost to arrive at what I’m calling “employment overhead”:

Employment overhead

At the end of Round 1, Singapore is the clear winner ($117!) and Sydney a close second, followed by Helsinki and, at the back of the bus, Tokyo and San Francisco, where nearly $8k disappear in a puff of smoke.

Round 2: Corporate tax and dividend taxation

In our idealized mini-company, everything that was not paid out to the employee is pure profit. If the company wants to dish them out to its sole shareholder, how much do they get after the taxman takes his share?

Cable car, San FranciscoIn California, a C corporation pays 8.84% corporate tax.  (A pity we’re not in Nevada, where it would be zero.)  Pile on federal taxes at 15% to $50k and 25% to 75k, noting that you can deduct your state corporate tax first, and we get 23.7%. Since California treats dividend income as ordinary taxable income, in the $36k-to-87k tax bracket you’re looking at 25% to Uncle Sam, 12.3% to California, 3.8% to Medicare and 1.2% for “deduction phaseouts” (wat?), totaling a whopping 42.30%. Tot these up, and our entrepreneur is left with 44% of what the company earned, or $27,828.  And if that sounds bad, here’s a calculation that arrives at 26% when you max out both tax brackets!

That said, if you’re going to do this in real life and do not have ambitions to grow to be the next Google, you should almost certainly opt for an S corporation or an LLC instead.  These don’t pay corporate taxes: instead, they pass their income (or loss) onto their shareholders, who then pay normal income taxes.  This is good if you’re keeping the money, but terrible if you were planning to reinvest it.  However, since all other corporation types in this little survey are “real” companies that can choose to reinvest or issue dividends, I felt that a C corporation is a fairer comparison point.

Geese in Hietaniemi Bay, HelsinkiIn Helsinki, both corporate tax and the tax treatment of dividends was revamped in 2013.  From January 2014 onwards, corporate tax is 20%, leaving €63,385 in post-tax profits.  The basic capital gains tax on dividends is 30% (but 32% above €40k), with a tax break of 75% on the first 8% of “free capital”, which for our newborn company equates to the total yearly profit.  This works out to an effective rate of 28.59%, leaving €36,211 in our hero’s bank account.

Cherry blossoms in Korakuen Garden, TokyoIn Tokyo, corporate taxation makes Japanese payroll and income taxes look simple.  Based on this handy summary from JETRO, for “a small company in Tokyo”, corporate tax is 15%, “restoration corporate surtax” (read: Fukushima surcharge) is 1.5%, prefectural and municipal “inhabitant taxes” in Tokyo are 0.75% and 1.85% respectively, “enterprise tax” is 2.70% (I presume this is meant to discourage entrepreneurship?) and as a cherry on top “special local corporate tax” is 2.19%, for a total of 23.99% but an effective rate of 22.86% for reasons I won’t even claim to understand. But wait! Once your earnings top Y4m, new rates apply and you now get socked for 24.56%.  In effect, the effectively effective rate is an ineffective 23.54%.

So that’s taxes, now dividends, which are even more like tentacle porn. To quote Japan Tax, “As so often seems the case with Japanese individual taxation, what would be expected to be a simple tax matter is made overly complicated by a range of expiring tax benefits (that are often revised or extended) and a range of alternative obscure reporting elections, deductions or exemptions.” The short of it is that the withholding tax rate for unlisted shares is 20%. The long of it is that you can then choose to report or not report it as ordinary income. If you do report, you have to pay income tax, but receive a dividend deduction of 12.8% for overall income of Y10m or less but 6.4% above which offsets the withholding tax and, you know what, fuggedaboudit. Pay 20%, end of story, and keep 40,709 hectoyen.

Sydney Monorail on Darling BridgeIn Sydney, company tax is a flat 30%, no if ands or buts, and the highest rate in our survey.  Dividend income is treated the same as ordinary income and taxed at same rate, except that thanks to dividend imputation you get “franking benefits” that are meant to offset the corporate tax already paid.  Acting on the optimistic assumption that they do, this means all that’s left is the difference between the corporate tax rate and the receiver’s marginal income tax rate, in our case 32.5% (for the $37k+ tax bracket) plus 1.5% levy, or 4%.  This leaves a rather juicy $48,326.

Singapore city at duskIn Singapore, corporate tax is usually 15%, but new companies in many sectors including IT pay zero (0%) tax for the first three years for their first $100k in profit, no special applications needed.  There’s also a whole slew of other tax incentives, including 4x deductions of IT gear and, most incredibly, cash transfers of up to $5,000/year from the tax man for companies that make a loss despite earning over $100,000 in revenue, but that’s another story and our rules exclude this kind of thing anyway.

Singapore has no capital gains tax, so Singapore company dividends are also tax-free. The company can thus pay out $67,223 in dividends, and the shareholder gets every last cent. This means our entrepreneur’s yearly earnings after tax are $91,223, and adding in the $8,640 sitting in their pension fund, they have managed to hold on to $99,883 of it.

Company profit vs dividends after tax

Dear reader, if you made it this far, I salute you. Now all that’s left is to sum up your salary, your pension and what’s left of your dividends to see how much of your $100,000 you still have left over.

Conclusion

Total left over from $100k

Singapore romps home as the undisputed winner, with the entrepreneur keeping a scarcely credible 99.9% of what they started with. It’s just a real shame that, for political reasons, the government has recently gutted the EntrePass scheme and thus made it close to impossible for a foreign one-man entrepreneur to set up shop.  (If you’re keen anyway, check out the Singapore Expats “Business in Singapore” forum or drop me a line.)

Somewhat to my own surprise, Sydney rocks up in second place with 74.6% left over.  Australia’s not what you’d call a low-tax (much less low-cost) country, but income tax is highly progressive and the dividend franking system means you only pay tax once on dividends, meaning that at low incomes you get to keep most of what you earn.  The calculus would change pretty rapidly if you earned $200k and found yourself in the 45% income tax bracket.

Helsinki and Tokyo show up neck and neck in 3rd and 4th place, with 66.8% and 66.5% respectively. Both have heavy taxation of income, payroll, corporate profit and dividends, but no total clangers. (Except Finland’s 24% value-added tax, the exclusion of which makes Helsinki unrealistically rosy, unless you can source all your income from outside the EU.)  The canny entrepreneur could nudge up both of those figures: in Tokyo, you’d actually want to pay yourself more salary since income tax is lower than corporate tax, while in Helsinki the reverse applies.

And trailing the pack with barely half left over is poor old San Francisco with 56.9%.  Now obviously there’s more to deciding where a company sets up than taxes, because otherwise Silicon Valley would be empty… but unless you really need to be there, from a financial point of view setting up pretty much anywhere else seems to make a whole lot more dollars and cents.

Acknowledgements

Thanks to corporate law ninja Joe & serial entrepreneur Juha for a sanity check of the US, Finland and Japan calculations, Tuomas Talola for corrections to Finnish calculations, and my Singaporean accountant, the infatigable Ms Tan, for clueing me onto this stuff back in the day.  If you spot any mistakes, drop me a line.

Incidentally, Joe says that a fairer version of this exercise would model the corporate and personal taxes of each country and then solve for the optimal combination of salary and dividends.  Any takers?

Revision history

31 Oct 2013

  • Recomputed Finnish income taxes with tax office calculator.
  • Adjusted Finnish employer pension to use YEL rate instead of TyEL rate.
  • Corrected Japanese pension to pay national pension (kokumin nenkin) only.

Lonely Planet: This is Not the End

not-the-endCommenting on the affairs of past employers is bad karma, but the media circus surrounding Lonely Planet’s recent restructuring, with Skift’s hash of disgruntled misinformation and the Guardian’s premature obituary, is sufficiently misguided to warrant an unsolicited opinion.

Lonely Planet is and has always been a print publishing operation. Despite their carefully cultivated hippy-dippy image, the Wheelers ran a tight ship and LP was known in the industry for being able to produce and distribute more guidebooks of higher quality at a lower cost than anyone else in the business. This was achieved by a relentless focus on tweaking the publishing machine, and during my time there were regular mini-celebrations for (say) switching to a new printer that allowed cheaper color pages or trimming editing time by 10% by automating tasks that were previously done by hand in layout.

Yet being a print house left the company unprepared for the digital era, and despite its early web presence, it never seized the chance to become Expedia or TripAdvisor. Two anecdotes illustrate why:

Industrial History Museum, Merrickville, CanadaEarly on, one of the publishing execs was taking me through The Spreadsheet, which forecast in minute detail and often with stunning accuracy how much a book would cost to create and how much it would sell, taking into account everything from the cost of public transport in the destination to the impact of upcoming titles from the competition. Offhand, she remarked, “I don’t think we should be investing in digital until its revenues exceed print.”

Taken at face value, this seemed absurd. How would digital ever grow without any investment? Only later did it dawn on me: “investment” for her meant doing what the spreadsheet measures, which is putting money into books. Digital revenue would come anyway from e-books, which would be faithful replicas of print books, and once the magical 50% tipping point was reached, they could start by adding video clips of the Eiffel Tower to page 294 in the e-book.

But what if people don’t want e-books?

Later on, I mentioned the travel potential of Google Glass to one of the people in the product development team, responsible for dreaming up Lonely Planet’s future products. “Yes!”, he enthused, “just imagine if somebody wearing Glass looked at our guidebook, and they could see the latest edits superimposed on top!”

But what if people stop buying printed guidebooks?

Mind you, these were both consummate publishing professionals who live and breathe print. So at the end of the day, even though they and others at LP knew in their bones that print was falling, and that e-books and apps weren’t making up the slack, they simply didn’t know what to do about it, other than to cut more costs and churn out more books.

The new CTO Gus, on the other hand, does. LP’s asset is its independent content beholden to no-one, which drives its website and its brand. Despite debacles like BBC’s catastrophic mismanagement of Thorn Tree, at 100m+ visitors/year LP’s digital footprint remains head and shoulders above its print competitors, and its vetted content has no match (yet) elsewhere in the digital world. What’s more, they’ve already spent years putting in the hard yards to bring their technical backend up to speed as well. A relentless focus on digital is LP’s best shot at survival, and last week’s layoffs, far from being a portent of doom, are the most concrete sign yet that NC2 Media gets this as well.

National Arboretum, Canberra, AustraliaParticularly important is the unheralded switch to a “destination editor” model, which finally breaks the stranglehold the book publishing schedule has had on the operations of the entire company.  For example, this will allow the website to be updated continuously, instead of having to wait for the next book edition to roll around.  Far from giving up on content, this puts it front and center, and the move parallels The Guardian‘s digital transformation that has seen the newspaper grab a sizable online audience far outside its native UK market.

None of this diminishes the human tragedy of letting go people who have poured years of their lives into what was indeed for many more of a family than a company. But as the only alternative is slow and inexorable decline guaranteed to lead to the elimination of every single job, this is the best hand the company can play.  As the last page of LP’s guidebooks used to proclaim: “THIS IS NOT THE END”.

Time is money: How Clash of Clans earns $500,000 a day with in-app purchases

Confession time: While I like to rationalize my blog’s recent silence with changing jobs and moving to a new city, the truth is, the single biggest drain on my free time lately has been Supercell’s Clash of Clans.  While this apparently puts me in good company, I decided it’s about time I shared what I’ve learned about how this “free” game apparently manages to spin well over $500,000 a day for its creator Supercell.  (Update: Forbes reports Supercell now earns over $2.4m a day, the majority of that from Clash of Clans.)

Overview

The core of Clash of Clans is a bog-standard resource management game: mine “gold” and “elixir”, use gold and elixir to buy improvements to your town so you can build stronger armies, raid other players in order to loot their gold and elixir, rinse and repeat.  If you’ve ever played Starcraft, Age of Empires or pretty much any other real-time strategy game, you’ll know the drill, and the buildings and units come off as almost painfully derivative.  There’s a Barracks for new troops and Archer Towers for defense, you’ve got Zerg-like cheap and disposable Barbarians, weak but ranged Archers, slow Giant tanks for soaking up damage, etc.  But formulas are formulas because they work, and it’s fun to set up your little village, win your first battles and watch your (thoroughly meaningless) levels and experience points rack up.  The touch-screen interface is a pleasure to use and the smoothly zoomable 3D graphics are beautifully animated with cute little touches; for example, when you tap to select an army camp, every unit salutes their leader in a different way.

Show Me the Gems

“So that’s all well and good”, I hear you say, “but where’s the money coming from?”

In the standard Zynga playbook for making money off with freemium games, you would let players buy gold, elixir or the items they want directly.  But Clash of Clans adds a twist: you can’t buy anything directly, but you can buy a third resource called “gems”.   Unlike gold and elixir, gems are not necessary for building anything functional, they’re simply a type of “power-up” that serve as a shortcut.  Need more gold to finish a building? Buy it with gems.  Need more elixir to add a dragon to your army?  Buy it with gems.  Don’t want to wait a week for a building to finish?  Complete it instantly with gems.  In other words, gems mean instant satisfaction.  What’s more, their cost is neatly obfuscated: purchased gems come in big, oddly numbered stashes (500, 1200, 2500, 6500, 14000), and once you have the pile sitting in your account, it’s easier to whittle it away 834 gems at time, whereas you’d probably think twice if asked to punch in your credit card details and confirm that you really want to pay $6.98 (just a sliver under the U.S. federal minimum wage) to upgrade your Wizard Tower.

Yet this formula’s beauty is that none of this is immediately apparent.  You start the game with 500 gems, which is plenty for the initial stages, and there are many easily earned “achievements” that reward you with more, so that you don’t initially appreciate their value.  The initial buildings are fast to build, with some building instantly and others taking a minute here or five minutes there.  And you’re shielded from enemy attacks for three days, so you can take your time building up your base and raiding the AI’s goblin bases for easy loot.

As you advance through the levels, though, the time and expense of everything ramps up exponentially.  A level 2 Town Hall takes 5 minutes to build and costs 1,000 gold; a level 8 Town Hall takes 8 days and costs 2,000,000 gold.  And you soon encounter the next twist: on the later levels, patience is no longer enough.  A maxed-out set of Gold Mines can produce 360,000 gold a day, meaning you could theoretically accumulate the sum neeeded for that Town Hall upgrade in 6 days.  However, you’re being continually raided by other players, and since other players can see your wealth before they choose to attack, a fat bank account means you’re a fat target.  What’s more, since successful raids award percentages of your wealth, a single “three-star” attack worth 25% can see 500,000 gold disappear in a flash.

coc-gold-vs-loginsThere’s more.  In your typical RTS, collected resources immediately go into your central storage.  In Clash of Clans, though, they stay in the collectors, vulnerable to attack both by location and design (up to 50% can be stolen, vs. 20% for central storage), until you log in to manually transfer them to relative safety with a tap.  This, too, makes it difficult to accumulate large amounts and encourages you to login at least several times a day.  The chart shows why: if you start with 1,000,000 gold in storage, earn 360,000 daily, and get attacked once daily, losing 50% from your collectors and 20% from central storage, the player who doesn’t bother logging in for two weeks will see their pile drop 75% to under 250,000, while the player who logs in religiously four times a day will increase their wealth by over 50% to 1,550,000 — but even their earnings flatline well before two million.

What this means is that, once your bankroll is over 1.5 million or so, the only free way to keep the balance growing is grinding, a tedious non-stop cycle of raiding and army rebuilding, with nervous logins every five minutes to keep raiders at bay (you cannot be attacked while online).  In their grandmotherly kindness, though, Supercell provides you a whole wealth of alternatives.  Can you spot the pattern?

  • You can use gems to buy “shields” that stop you from getting raided: 250 gems ($2.50).  Lest that seem too cheap, you’re preventing from using more than one week of shield per month.
  • You can use gems to fill up your gold or elixir storages instantly: 834 gems ($8.34)
  • You can use gems to upgrade your gold mines to the next level, where they will work faster: 966 gems ($9.66)
  • You can use gems to buy additional “workers”, so you can upgrade your production faster and earn gold/elixir faster: 1000 gems ($9.98)
  • You can use gems to double your production of gold or elixir for a 6-hour period.  Repeated across six mines for three days: 1368 gems ($13.68)
  • You can use gems to rebuild your armies instantly, so you can keep raiding and racking up loot without anybody having a chance to steal their money back via the handy “Revenge” button. Assuming 50k elixir to rebuild and 50k profit per raid: 2760 gems ($27.60)

And once you’ve finally earned those 2 million and clicked the “Build” button, it takes another 8 days to build the thing — unless, that is, you fork out another 1123 gems ($11.23) for instant completion.  It’s little wonder most players start “to gem” (it’s a verb in Clash of Clans parlance) by the time they reach Town Hall level 7 or so, the stage when most costs are measured in millions and building virtually anything unassisted takes days.  Jorge Yao, the game’s undisputed champion, figures he has spent north of $2500 in real money on buying gems, and according to back-of-the-envelope calculations, the cost of fully fitting out your virtual village is on the order of $5000 when you include walls.  It’s little wonder the top clans leaderboard is full of players like “>< Royal ><” from Kuwaiti clan “Q8 FORCE” and clan UAE’s “khalifa” (presumably from Bahrain’s ruling House of Khalifa).

Your Pain is Supercell’s Gain

Unsurprisingly, the entire game has been warped in subtle ways to encourage buying and using gems.

For example, in your average real-time strategy game, you have fine-grained control deploying and directing your troops, and units that survive can be used in the next battle.  Not so in Clash of Clans: once deployed, units fight according to their hardcoded strategy (most commonly the harebrained “bash closest building”, regardless of what it is or who is firing at them), and every unit deployed disappears at the end of the battle, even if they are victorious.  This means it’s essential to rebuild huge armies and to attack with massive force every time.   And since building those dragons can take several hours, during which time you’re wide open to attack, there’s another massive inducement to solve the problem with a few gems.

Probably the most blatant case of tilting the table is the recent introduction of “Dark Elixir”, a third in-game resource geared squarely at high-level players.  Collected at the glacially slow pace of 20 units per hour, even in an raidless pacifist world it would take 21 days of waiting to accrue the 10,000 units needed purchase its main selling point, the Barbarian King, and the table is stacked further yet by subjecting collectors to 75% raid losses.  Who wouldn’t pay $6 to skip the tedium and uncertainty?

In comparison, the “clans” of the name seem almost like an afterthought.  Their primary function is to be a gifting circle, where players donate units to others in their clan, and receive units in return.  And that’s it: clan players cannot share gold or elixir, much less gems.  But they do provide another handy lever of extra social pressure to ensure you log in regularly, since clan troops defend your base, die when attacked and can only be received on explicit request, and since most clans enforce minimum per week donations and kick out “freeloaders” who have not paid their dues.

But It Could Be Worse…

Some credit where credit is due: unlike Zynga’s notoriously annoying games, Clash of Clans does not require Facebook signup, cram the game full of ads, spam you and your friends, or pimp your personal information to random third parties.  And while you’ll be reminded that “Hey, you could use gems for this” whenever you try to do something you can’t afford, if you stay within your means and have the patience of an ascetic saint, you’ll never even get asked for money.

Conclusion

At the end of the day, my feelings towards Clash of Clans are distinctly mixed.  Being a penny-pincher whose in-game purchases have been limited to a single $4.99 gem pack, even that largely as a token of appreciation to the game’s makers, I can’t really complain about the hours of entertainment I’ve gotten in exchange.  Yet I still can’t help but cringe as I run into all the ways the game is intentionally crippled to get you to pay up, and the way its Pavlovian triggers to come back for more operate on fear.   Would Minecraft have been any fun if it required you to log in every six hours or you’d lose parts of your inventory?   And how much more awesome would Clash of Clans be if the effort of squeezing every last cent out had been put into improving the game itself instead?

Down, down, down: Books, e-books and apps all trending to zero

“We expect DVD Subscribers to decline every quarter.. forever.”
— Reed Hastings, CEO, Netflix

Does your business model rely on selling paid content?  Welcome to Mr. Hastings’s world.

Books

Many publishers continue to operate under the assumption that printed book sales are declining gradually or perhaps even plateauing.  Unfortunately the data tells a different story: the decline appears to be accelerating.  Here’s Nielsen Bookscan for the travel market:

Printed guidebook sales, millions/year

That’s from the “Guidebook Category Report, Rolling, Period 13″ for 2006 to 2012.  The trendline is a simple polynomial (n=2) best fit, and if it’s accurate, the market will halve by 2015.  And while that sounds drastic, it’s by no means unprecedented, as the sales of CDs did pretty much the same thing in 2006-2009.

Of course, the market’s not quite homogenous: Lonely Planet’s been beating the trend, mostly by continuing to invest in print and absorbing customers from the rapidly-disappearing Frommers. But that just makes LP an even-bigger fish in an ever-shrinking pond.

So can the white knights of digital paid content, e-books and mobile apps, save the publishing industry?

E-Books

Finding good data for e-books is a pain, so I ended up rolling my own: I grabbed Amazon’s Kindle Store Top 100 bestsellers lists since 2007, using snapshots from the Internet Archive that record the actual prices at the time, and computed average prices and the proportion of under-$5 titles, the vast majority of which are self-published.  (Source code in Ruby here.)

Average price of e-books in Kindle Store Top 100

Despite that December 2012 spike, the trend is clear, and while the decline looks gentle, my personal suspicion is that the trend line is too optimistic and that there’s a collapse looming.  For one thing, the data above is only for best sellers, meaning new books by well-known authors who command a distinct price premium; the average price of an average e-book is both lower and falling faster.  Yet even in the Top 100, the share of “cheap” books, the vast majority of which are self-published, is growing exponentially:

Books Under $5 in the Kindle Store Top 100

While I didn’t use their data directly, I owe tips of the hat to Piotr Kowalczyk, who wrote a very detailed report on the growth of self-published books on Kindle, and Digital Book World, which has been keeping tabs for the past half year (albeit looking only at the top 25).  DBW also has a credible explanation for the spike, which boils down to publishers yanking up prices in the period before they had to start allowing discounting, and further reinforces that shrinking prices are the new normal.

Apps

The collapse of prices in the mobile app world has been even more drastic, to the point that outside an  elite circle of bestsellers and the odd very specific niche, making money by selling the app itself is a pipe dream.  (All data courtesy of 148Apps.biz and the Internet Archive.)

Average price of paid applications in the Apple Store

By the end of the year, the average app will cost under 99 cents, and even that’s pulled up by every $999 BarMax and wannabe in the store.  The median price is already zero:

Share of free applications in Apple Store

In other words, over half of all apps are already free.  On current trends (and look how beautifully that line fits the data!), that will be over 80% within two years.  This is for the “premium” Apple Store widely opined to have less stingy customers; the equivalent figures for Android will be even more brutal.

What to do then?  The only answer is to figure out a way to make money that doesn’t involve readers paying for content.  Here are some ideas: