Handling Fraudulent In-App Purchases on iOS

The Audiogalaxy mobile apps for iOS and Android have some in-app purchases which allow users to buy add-ons to augment the free service. I've written previously about some interesting observations about implementing each. Continuing that series, here are some more interesting things I've seen when dealing with in-app purchases on iOS, specifically related to fraudulent purchases.

Apple's in-app purchases programming guide describes the steps needed to verify store receipts once you're notified of a successful transaction in the SKRequestDelegate callback. It's tempting to just unlock the feature when you get notified of a successful transaction being complete, but you really should do the extra work required to send over the receipt to Apple for verification, or you risk getting defrauded pretty easily. Ideally, you want to send the receipt up to your own server first (over SSL, obviously) and then verify the receipt using Apple's iTunes receipt verification service.

Since releasing the in-app purchase for iOS, I've seen several fraudulent purchase attempts. For the first version, I took a conservative approach - when notified by StoreKit that a transaction was successfully completed, I unlocked the feature and then sent the receipt up to my server for verification. Additionally, I recorded verification responses from Apple's service but didn't make the device reject the add-on if it failed. The reason I did this was a combination of naivete and being unsure of how things would really work in the wild. What if my server could not be reached? What if Apple's service was down? I didn't want the user to be charged money and be left hanging because of connectivity problems. (Not knowing better, I also assumed that the 'purchase complete' callback was pretty bullet-proof and meant that the user was charged, and that verification was just added formality).

Well, it turns out that StoreKit on jailbroken iOS devices can be easily fooled into believing that a local transaction completed successfully. So with my conservative approach, users with jailbroken phones could easily get in-app features unlocked without paying for them. Thankfully, I caught this before it did serious damage and pushed an app update. Because of the dumb-client architecture of my system, I was also able to easily revoke the purchases made fraudulently (need more reasons to have a server be in control?) What I had originally failed to account for is that the system is built to handle verification failures - if you legitimately fail to verify a transaction because of connectivity problems on either end, just don't mark the transaction as 'finished' and it will automatically get restored the next time StoreKit's queue is initialized (which is why it's also a good idea to write code to handle restored transactions, even if your in-app purchases are only of the consumable variety). This might mean that an add-on won't be available to a user for a while if they paid and you failed to verify the transaction because of networking problems, but they won't be re-charged for it and it will eventually be unlocked correctly.

In case you're interested, here are some examples of fake base64 encoded receipts:

Y29tLnVydXMuaWFwLjcxNDA4NTE0
Y29tLnVydXMuaWFwLjMwODEwMjk5
Y29tLnVydXMuaWFwLjk2NTQyNjQ3

The response from Apple, when these types of receipts are sent for verification, looks like:

{"status":21002, "exception":"java.lang.ClassCastException"}

FWIW, Apple's documentation states that any non-0 response is considered a failure for purchases other than auto-renewing subscriptions. For those, they have a set of well-defined error codes. In a future post, I'll try to cover my experiences implementing auto-renewing subscriptions (and the non-technical, app approval overhead that goes with it).

If you're curious, here's the distribution of where fraudulent purchase attempts came from:

United States - 38.46%
Israel - 14.11%
Canada - 5.45%
United Kingdom - 4.49%
Saudi Arabia - 3.81%
Netherlands - 3.53%
Germany - 1.60%
Mexico - 1.60%
United Arab Emirates - 1.28%
Japan - 1.28%
Others - rest

Customer Support Rule #1 - Don't be a Robot

A recent user support interaction, pictured below, is one of many examples I've experienced where being honest and human goes a long way in winning over users, and humor doesn't hurt. You don't have to be 'nice', don't have to follow a strict template or be formal. You just have to communicate with your user like a real human being.

Users love it when they realize there's a human on the other end, and not a robot that's spewing stock replies crafted by corporate types. That is, often, the start of a loyal relationship.

Don't Let Apple's Latency Mess With Your iOS App

Last week, I released an update for the iOS Audiogalaxy app which featured its first in-app purchase that allows users to 'pin' songs to their devices. Users can try the feature by pinning a limited number of songs for free, and are then prompted to purchase if they wished to continue pinning more songs. As suggested in Apple's SDK documentation, I built and tested the in-app purchase against the sandbox. After everything checked out as I expected, I submitted both the app update and the in-app purchase for approval by Apple.

The application and the in-app purchase were concurrently approved. After verifying that the status of in-app purchase product in iTunes Connect was "Ready for Sale", I released the updated application. It usually takes about an hour for app updates to start propagating to users, so I anxiously waited to get the update on my iPhone and verify that in-app purchasing worked fine. Once I got the update, I tried to purchase the in-app product but it failed. I had written code to handle potential errors, and looking at the error message I realized that the StoreKit API wasn't able to identify the in-app product I was requesting.

I went back and verified that the product identifier was correct, and everything still worked fine in the sandbox. Users had started hitting the same problem - not good! It's definitely not the best first impression for users to upgrade, try new features and immediately see an error. I scoured the web for clues and found a StackOverflow question which suggested that it can take up to 24-36 hours for in-app products to 'go live'. Uh-oh!

There is no mention of such a delay anywhere in Apple's documentation, and neither is it reflected in the status of the in-app purchase product. If I had known, I obviously would have waited to release the update. Apple doesn't provide the ability to roll back to previous versions, so basically I was SOL. If indeed it took ~24 hours for the in-app product to show up, it could be disastrous. The fallout could include:

• Disappointed users. There is nothing more disappointing that looking at the "What's New" section of an app before upgrading, getting excited and then watching the feature just not work as expected.
• Negative reviews/ratings. App Store ratings/reviews provide a medium for users to vent their frustrations or to shower praise. For developers, it's really a matter of making sure that the people with exceedingly pleasurable experiences with your application exceed the number of people with even remotely dissatisfying experiences. Audiogalaxy was a 5-star rated app, but this snafu could potentially affect that negatively, really fast.
• Personal disappointment. I take a lot of pride in making sure that I deliver delightful experiences to users, and this would be the exact opposite of delightful. The hurt would be tempered by the fact that this was beyond my control, but it would hurt nonetheless.
• Lost revenue. From my experience with in-app purchases on Android (and now also on iOS) it was abundantly clear to me that purchase volume is very high in the first few days of a new in-app purchase being available. Users upgrade, try what's new and buy it if they like it. If they're unable to, they might never come back to buy it.

I was really hoping that the problem was the delay-to-propagate issue and not anything more serious. Damage would be limited if the in-app purchase went live before too many users upgraded and started using the new feature, and I was secretly hoping that this was a matter of a few hours instead of 24-36 as suggested in the SO question.

I started monitoring how many users were pulling down the update and trying this new feature (yay for building simple monitoring tools that are fully in your control) and also dumping these userIDs into a table so I could keep track - a subset of these users were (hopefully) trying to purchase the add-on, but were failing to do so. The number started growing rapidly, and I began to panic. 5-6 hours later, the in-app product still wasn't live, negative reviews had started showing up and folks had started emailing us to let us know.

Watching user behavior, I knew something needed to be done. We decided that the best option was to make it impossible to even attempt to purchase the add-on. I updated the server to make it believe that all iOS users had already purchased the add-on - the app would then automatically hide all purchase related options. The plan was to just let everyone use the feature for free until the in-app purchase showed up for real. I wrote the code, tested it and pushed it to production. I watched it work for a bit, and then went to sleep.

The next morning, 14 hours after the status was "Ready for Sale", I was finally able to purchase the in-app product from the app. Thankfully, the server tweaks had helped and there were no new negative reviews or support emails. I rolled back the code from the previous night and waited for purchases to start showing up. Within minutes, I could see folks buying so I knew things were working fine now.

The last thing that needed to be done was informing users who might have previously had trouble and might potentially never try buying again. I wrote a script to run through the list of all users who had tried the feature and to send them a personal note thanking them for their support, explaining what had happened and letting them know that the in-app purchase was now available if they wished to purchase it. A surprisingly large percentage of users went ahead and purchased it, and a few also went back and updated their reviews.

I took away three lessons from this experience -

• Design your apps/clients to be as dumb as possible and the server as much in control as possible. We've always tried to do that with Audiogalaxy, and it certainly helped in this situation (as it has on a few previous occasions).
• Spamming all your users isn't a good idea, but it's perfectly OK to email a subset of users directly impacted by specific issues. Users are receptive to personally addressed emails with honest dialogue, and don't consider it spam.
• Wait for 24 hours before releasing iOS app updates if you have added new in-app purchases.

All in all, this didn't turn out as badly for us as it could have - but it shouldn't have happened. I still believe Apple is at fault for not communicating this information to developers, but that's not a rare occurrence for Apple - it's best to anticipate and prepare for the unexpected. I hope this blog post helps other who might be planning to add in-app purchases to their apps. In addition to better documenting such gotchas, it would also be awesome if Apple added the ability for developers to roll back to previous versions (or at least 1 previous version) like Google's Android Market recently did.

A Capitalistic Fix for America's Broken Immigration System?

Immigration in America is broken. There is no way to sugarcoat it, unfortunately. Failure of the administration to separate legal and illegal immigration concerns and to treat the politics of both as distinct from each other is jeopardizing the future of America.

There are several articles, memos and blog posts which articulate what is wrong in a lot of detail, so I won't repeat those. To illustrate the problem though, here is a short blurb about my own situation.

I have been a legal immigrant in America since I arrived to study Computer Science at the University of Texas at Austin in 2000. I graduated with a BS in Computer Science, interned at National Instruments for many months and went on to work at Microsoft for several years. I now work at a software startup. I'm a legal resident alien, but I'm still awaiting my permanent resident card ("green card"). I've been in the US for over 11 years now. I pay my taxes (TurboTax computes that ~16% of all income I've earned over the last 10 years has gone towards taxes, and this doesn't include local sales taxes), I occasionally speed on freeways and get pulled over, I spend money to stimulate the local economy (Mint.com says I spend ~2.5x what the average American my age spends), I eat junk food, I save money for retirement and invest it in American companies, I volunteer at a local high school, I support the Texas Longhorns and the Seattle Seahawks, I drink a lot of coffee, I donate to local and international causes, I whine about the weather... basically, I do what most average Americans do. I have a decent life here in America, and I'm thankful for it. But, barring any change in legislation, the estimated date for my green card being issued is 2024. Yep, that's not a typo. If nothing changes, I will get my green card 24 years after I legally entered the United States of America. Not citizenship, just permanent residence!

I assume I don't need to explain to any reasonable person how ridiculous and ironic this situation is, given that American was built by immigrants. I won't get into how my personal and professional life are impacted by this silly situation. Instead, I want to propose a possible solution, one that I haven't heard mentioned previously.

What if one of the criteria for issuing a green card was - if your stay in America has been legal and you have paid more in federal+local taxes than the average American pays over his/her entire lifetime then your green card application should be fast-tracked.

If this sounds too simple, it's because it is -a simple, capitalist, pragmatic approach to immigration. This rule ensures that prospective immigrants enter America using legal means and remain legal, that they are employable and have documented income that is in line with or exceeds that of average Americans, that they will not be a burden to society but an asset and that they have an incentive to contribute to the success of their adopted nation.

Thoughts?

Unfortunately, immigration reform is a political issue instead of a practical one, and a solution to legal immigration doesn't seem possible without a solution to illegal immigration (which just baffles me - anyone know why the two are joined at the hip?). America and Americans will eventually have to suffer the consequences of such nearsightedness, I fear. Illegal immigration is a political minefield though, so I don't have much hope for any real reform anytime soon, but then again... "hope and change", right?

Some Notes on Implementing In-App Billing on Android

About a year ago, I implemented in-app payments for iOS, and it was painful. Technically, it wasn't too hard but there were a number of hoops to jump though to get everything working. Lack of clear documentation and undocumented error responses seemed to be the primary problem - I think a hold up with verifying tax information on the developer account resulted in strange, undocumented failure codes in my case. There were several dozen blog posts complaining about many such "magic" issues back then. It was almost comical, to be honest.

This last weekend, I decided to add in-app billing to an app that's already shipping for Android, and I ran into a couple of hiccups that I think are worth documenting here to save others some time/pain. Overall, I think the Android folks learnt from some of the pitfalls that early developers hit with in-app payments on iOS and worked to fix those. The documentation is a lot clearer and precise compared to where iOS was a year ago. Android Market also has a nifty static test/response facility which makes it easy to test code without worrying about external influences. iOS was desperately lacking something of this sort (not sure if it's better now) - I even wrote a dummy implementation of the iOS in-app payment API and the different response conditions to let me test my code in an emulator, which was really helpful. I'm glad Android got this right (still no testing in an emulator though, which is a bummer because of the inability to test with different Android OS versions).

At a high level, the in-app billing protocol on Android is extremely chatty (see the diagrams on this page). It seems particularly so when compared to PayPal's rival offering. PayPal's in-app billing module handles payment using startActivityForResult and onActivityResult methods which can be used for communicating between child/parent activities. This is very similar to how Facebook's Android SDK handles single sign on. That model is extremely simple to understand and implement for developers - you could probably have it coded up and tested faster than you can finish reading this blog post. I think part of this chattiness is to ensure a higher level of security and also to handle managed purchases, which are both likely harder to do using PayPal's approach. I assume they had good reason to make it as chatty as it is now, but my gut says that Google loses out on a lot of developers because of this apparent complexity. Having said that, the sample application Google provides does a really good job of illustrating the various communications that need to occur, so studying that code makes the chattiness easier to understand (for some reason, code registers more easily in my brain than verbose documentation - good code and good documentation together are very rare, and Android seems to have gotten this right - kudos!)

One of the biggest red flags with using Android's in-app billing, however, is the development teams non-responsiveness to critical bugs. This issue, marked critical and accepted, talks about transactions failing 4-6% of the time, and has been active for over two months without an owner being assigned. An equally disturbing issue with unacceptably long latency when handling authorization has been open for almost two months without any comment from Google or even an owner being assigned. In fact, browsing the issues list certainly doesn't inspire confidence in the platform. It would be a big mistake by Google to not take these issues seriously. In my case, I'm certainly debating the merit of jumping into this as an early adopter.

Some other notes about implementing in-app billing on Android:

• It sounds obvious in hindsight, but you need to make sure that your application is signed with the release key before testing in-app billing, even the static response stuff. Developer's aren't used to doing this while debugging, so it's worth calling out in the documentation.

• If you want valid signatures for the JSON responses as suggested by the table here, the "debuggable" attribute in the app manifest should be set to false, even if you're using a release-key-signed APK and static responses. I was a little annoyed by this un-documented restriction because it basically meant that I had to resort to using logging as the only real means of debugging my code. If there is indeed some way to debug this using a debugger (and yet receive valid signatures), it’s definitely not obvious to me. Again, not a terribly big deal once you realize that this attribute actually seems to affect signatures, but that isn't documented anywhere.

• If you're using a server to validate signatures (as you should), be aware that the public key you obtain from the Android Market profile page is a base-64 encoded string. You will almost certainly need to convert it to PEM/DER format or use the x509 certificate for it to be usable with most server openssl implementations. For instance, this command can be used to convert the base-64 encoded key into a key usable by PHP:

  openssl enc -base64 -d -in publickey.base64 -A | openssl rsa -inform DER -pubin > publickey.pem

  Also, if you're using the PHP function openssl_verify, remember that $signature needs to be the binary signature, not the base-64 encoded signature string that the Android Market app sends (I found out the long way).

• Unavailability of subscriptions/auto-billing is a big gap in the offering currently, but I assume support for it is coming soon, since iOS already supports it.

• I hit an issue when testing payments using carrier billing, where accepting the carrier ToS basically causes the transaction to get lost somewhere between the carrier and Android Market. I logged this bug to track the issue.

Well, I hope that the web-search spiders do their job and this post helps others out there looking at implementing in-app billing on Android. And making Google get a move on those bugs.

Discuss this post on Hacker News

The India-Trip Penalty

Vacations to India from America have a pretty high fixed-cost that significantly eats into the time that can 'actually' count as vacation. Here's the composition of this 'penalty':

• Average flying time to get to India, if you don't need a connecting flight at the destination, is 18-22 hours each way depending on where you fly from, the route and layovers. Assuming the actual flight time is an optimistic 20 hours, let's add the 3 hours that it takes to head out to the airport/check-in/security/etc each way, and another 2 hours to clear immigration/customs/etc each way. So, the time to get to and from the vacation is 50 hours.

• When you fly half way around the world, it's hard not to get jet-lagged. Optimistically, you've got to account for at least a half day (a very optimistic assumption) wasted on both ends of the trip. So you have another 24 hours spent jet-lagged.

• A special perk of visiting India is the high probability that you'll fall ill. Over the years, I've noticed my immunity to the Indian environment (food, water, air, etc) fade. This is true for most people I know who have lived outside of India for > 5 years. During each trip, it's very likely that you'll catch a stomach flu, get food poisoning, a fever, or generally fall sick. If you're lucky, it takes a day of complete rest and another half day of extreme caution to get back on track. So that's another 36 hours of lost vacation right there.

• Packing for a trip back from India isn't like packing for a quick flight to San Diego. A good deal of your last day is usually spent planning to pack, and then packing, and then re-packing to make sure your bags are all within weight limits. For those who don't live in metro areas, there's the additional time of travelling to metro areas. So let's throw in another 6 hours for all this.

So the total penalty of a trip to India is, roughly, 116 hours (a little under 5 days). Regardless of how long your vacation is, you're likely to incur this penalty, give-or-take a few hours. Heck, I would consider a 5-day trip within the US, or even a quick 5-day trip to Mexico, to be a decent vacation in itself.

It's pretty amazing to think that for many, the penalty of a vacation to visit friends and family is a whole other vacation. It's no wonder that most people who go to India need/want at least 3 weeks for the vacation to be worth it, and even then they feel gypped (rightly so?).

This also puts into perspective things that you otherwise might not think about. How much would you pay to avoid a 6-hour layover in Amsterdam en-route to India? Would a business-class ticket be worth the time it helps you shave off with airport formalities, jet-lag and worrying about luggage limits?

Hey Bing, iOS isn't Windows Phone 7

The Bing app for iOS was decent, until recent updates. It looks like someone at Microsoft decided they needed to make the Bing app for iOS look and feel more like a Windows Phone 7 app. So they did, and the result is an ugly, unusable disaster.

On Windows Phone 7, buttons look like traditional 'labels', but in lower-case font. Clickable elements are very subtly different from text, but yet it's obvious what is clickable and what isn't. The hardware 'back' button, standard on all phones, is used to dismiss modal dialogs. That's how the native interface of the phone (called 'Metro', I believe) is, and it all comes together quite beautifully. Here's what a menu on a native Windows Phone 7 app looks like:



On iOS, buttons looks like, well, buttons. Clickable elements have depth and curves and are strikingly different from text/labels. There is no hardware back button, so Apple has provided built-in buttons with system fonts and colors which are placed on the top right corner to dismiss modal dialogs. Just like Windows Phone 7, it's well thought out, coherent and comes together beautifully. Here's how menus, buttons and dialogs look on iOS.

Good apps follow Human Interface Guidelines of the platform they're built for, which gives them consistency with the rest of the system and makes interactions with the application intuitive. For some reason, the developers/designers of the Bing iOS app decided to follow the Human Interface Guidelines for Windows Phone 7 for their iOS app, instead of following the iOS HIG guidelines, and the result is just awful.

Below is what a modal menu on the Bing app looks like on my iPhone (the rest of the app looks similar, and has the same flaws). I have been using apps on my iPhone for two years and yet it took me several minutes to figure out what I needed to do on that dialog. It wasn't obvious how to toggle options on the menu, nor was it obvious how I could dismiss the dialogs. It took me a good two minutes to realize that the 'done' label was infact a button, and that almost everything on that screen was clickable, except the tiny header that reads "MENU". The font size on that label (page header) is smaller than that on the label-buttons, which sounds inverted, and makes the page look ugly.

Compare this with the image below and notice how obvious and beautiful some of those same options are on the native iPhone mapping application. (This isn't a modal dialog on the native app - if it was, I can guarantee that there would be a blue 'Done' button in the top right corner).

In fact, the entire experience of using the Bing app is so jarring that I refuse to use it at all, in spite of some of its nifty features. And I would be very surprised if I'm the only one who feels that way. It literally feels like Microsoft ported an app that is likely beautiful and usable on Windows Phone 7 to iOS without putting any thought into iOS usability. In fact, my guess is that the developers spent more time trying to make those buttons and table-views looks like Windows Phone 7 than they would have if they just used what the system provided with minor tweaks.

I cannot come up any reasonable explanation for why the designers of Bing chose to do this. If they want their application to be used, I sincerely hope they fix it soon.