Handling Fraudulent In-App Purchases on iOS

The Audiogalaxy mobile apps for iOS and Android have some in-app purchases which allow users to buy add-ons to augment the free service. I've written previously about some interesting observations about implementing each. Continuing that series, here are some more interesting things I've seen when dealing with in-app purchases on iOS, specifically related to fraudulent purchases.

Apple's in-app purchases programming guide describes the steps needed to verify store receipts once you're notified of a successful transaction in the SKRequestDelegate callback. It's tempting to just unlock the feature when you get notified of a successful transaction being complete, but you really should do the extra work required to send over the receipt to Apple for verification, or you risk getting defrauded pretty easily. Ideally, you want to send the receipt up to your own server first (over SSL, obviously) and then verify the receipt using Apple's iTunes receipt verification service.

Since releasing the in-app purchase for iOS, I've seen several fraudulent purchase attempts. For the first version, I took a conservative approach - when notified by StoreKit that a transaction was successfully completed, I unlocked the feature and then sent the receipt up to my server for verification. Additionally, I recorded verification responses from Apple's service but didn't make the device reject the add-on if it failed. The reason I did this was a combination of naivete and being unsure of how things would really work in the wild. What if my server could not be reached? What if Apple's service was down? I didn't want the user to be charged money and be left hanging because of connectivity problems. (Not knowing better, I also assumed that the 'purchase complete' callback was pretty bullet-proof and meant that the user was charged, and that verification was just added formality).

Well, it turns out that StoreKit on jailbroken iOS devices can be easily fooled into believing that a local transaction completed successfully. So with my conservative approach, users with jailbroken phones could easily get in-app features unlocked without paying for them. Thankfully, I caught this before it did serious damage and pushed an app update. Because of the dumb-client architecture of my system, I was also able to easily revoke the purchases made fraudulently (need more reasons to have a server be in control?) What I had originally failed to account for is that the system is built to handle verification failures - if you legitimately fail to verify a transaction because of connectivity problems on either end, just don't mark the transaction as 'finished' and it will automatically get restored the next time StoreKit's queue is initialized (which is why it's also a good idea to write code to handle restored transactions, even if your in-app purchases are only of the consumable variety). This might mean that an add-on won't be available to a user for a while if they paid and you failed to verify the transaction because of networking problems, but they won't be re-charged for it and it will eventually be unlocked correctly.

In case you're interested, here are some examples of fake base64 encoded receipts:

Y29tLnVydXMuaWFwLjcxNDA4NTE0
Y29tLnVydXMuaWFwLjMwODEwMjk5
Y29tLnVydXMuaWFwLjk2NTQyNjQ3

The response from Apple, when these types of receipts are sent for verification, looks like:

{"status":21002, "exception":"java.lang.ClassCastException"}

FWIW, Apple's documentation states that any non-0 response is considered a failure for purchases other than auto-renewing subscriptions. For those, they have a set of well-defined error codes. In a future post, I'll try to cover my experiences implementing auto-renewing subscriptions (and the non-technical, app approval overhead that goes with it).

If you're curious, here's the distribution of where fraudulent purchase attempts came from:

United States - 38.46%
Israel - 14.11%
Canada - 5.45%
United Kingdom - 4.49%
Saudi Arabia - 3.81%
Netherlands - 3.53%
Germany - 1.60%
Mexico - 1.60%
United Arab Emirates - 1.28%
Japan - 1.28%
Others - rest

Don't Let Apple's Latency Mess With Your iOS App

Last week, I released an update for the iOS Audiogalaxy app which featured its first in-app purchase that allows users to 'pin' songs to their devices. Users can try the feature by pinning a limited number of songs for free, and are then prompted to purchase if they wished to continue pinning more songs. As suggested in Apple's SDK documentation, I built and tested the in-app purchase against the sandbox. After everything checked out as I expected, I submitted both the app update and the in-app purchase for approval by Apple.

The application and the in-app purchase were concurrently approved. After verifying that the status of in-app purchase product in iTunes Connect was "Ready for Sale", I released the updated application. It usually takes about an hour for app updates to start propagating to users, so I anxiously waited to get the update on my iPhone and verify that in-app purchasing worked fine. Once I got the update, I tried to purchase the in-app product but it failed. I had written code to handle potential errors, and looking at the error message I realized that the StoreKit API wasn't able to identify the in-app product I was requesting.

I went back and verified that the product identifier was correct, and everything still worked fine in the sandbox. Users had started hitting the same problem - not good! It's definitely not the best first impression for users to upgrade, try new features and immediately see an error. I scoured the web for clues and found a StackOverflow question which suggested that it can take up to 24-36 hours for in-app products to 'go live'. Uh-oh!

There is no mention of such a delay anywhere in Apple's documentation, and neither is it reflected in the status of the in-app purchase product. If I had known, I obviously would have waited to release the update. Apple doesn't provide the ability to roll back to previous versions, so basically I was SOL. If indeed it took ~24 hours for the in-app product to show up, it could be disastrous. The fallout could include:

• Disappointed users. There is nothing more disappointing that looking at the "What's New" section of an app before upgrading, getting excited and then watching the feature just not work as expected.
• Negative reviews/ratings. App Store ratings/reviews provide a medium for users to vent their frustrations or to shower praise. For developers, it's really a matter of making sure that the people with exceedingly pleasurable experiences with your application exceed the number of people with even remotely dissatisfying experiences. Audiogalaxy was a 5-star rated app, but this snafu could potentially affect that negatively, really fast.
• Personal disappointment. I take a lot of pride in making sure that I deliver delightful experiences to users, and this would be the exact opposite of delightful. The hurt would be tempered by the fact that this was beyond my control, but it would hurt nonetheless.
• Lost revenue. From my experience with in-app purchases on Android (and now also on iOS) it was abundantly clear to me that purchase volume is very high in the first few days of a new in-app purchase being available. Users upgrade, try what's new and buy it if they like it. If they're unable to, they might never come back to buy it.

I was really hoping that the problem was the delay-to-propagate issue and not anything more serious. Damage would be limited if the in-app purchase went live before too many users upgraded and started using the new feature, and I was secretly hoping that this was a matter of a few hours instead of 24-36 as suggested in the SO question.

I started monitoring how many users were pulling down the update and trying this new feature (yay for building simple monitoring tools that are fully in your control) and also dumping these userIDs into a table so I could keep track - a subset of these users were (hopefully) trying to purchase the add-on, but were failing to do so. The number started growing rapidly, and I began to panic. 5-6 hours later, the in-app product still wasn't live, negative reviews had started showing up and folks had started emailing us to let us know.

Watching user behavior, I knew something needed to be done. We decided that the best option was to make it impossible to even attempt to purchase the add-on. I updated the server to make it believe that all iOS users had already purchased the add-on - the app would then automatically hide all purchase related options. The plan was to just let everyone use the feature for free until the in-app purchase showed up for real. I wrote the code, tested it and pushed it to production. I watched it work for a bit, and then went to sleep.

The next morning, 14 hours after the status was "Ready for Sale", I was finally able to purchase the in-app product from the app. Thankfully, the server tweaks had helped and there were no new negative reviews or support emails. I rolled back the code from the previous night and waited for purchases to start showing up. Within minutes, I could see folks buying so I knew things were working fine now.

The last thing that needed to be done was informing users who might have previously had trouble and might potentially never try buying again. I wrote a script to run through the list of all users who had tried the feature and to send them a personal note thanking them for their support, explaining what had happened and letting them know that the in-app purchase was now available if they wished to purchase it. A surprisingly large percentage of users went ahead and purchased it, and a few also went back and updated their reviews.

I took away three lessons from this experience -

• Design your apps/clients to be as dumb as possible and the server as much in control as possible. We've always tried to do that with Audiogalaxy, and it certainly helped in this situation (as it has on a few previous occasions).
• Spamming all your users isn't a good idea, but it's perfectly OK to email a subset of users directly impacted by specific issues. Users are receptive to personally addressed emails with honest dialogue, and don't consider it spam.
• Wait for 24 hours before releasing iOS app updates if you have added new in-app purchases.

All in all, this didn't turn out as badly for us as it could have - but it shouldn't have happened. I still believe Apple is at fault for not communicating this information to developers, but that's not a rare occurrence for Apple - it's best to anticipate and prepare for the unexpected. I hope this blog post helps other who might be planning to add in-app purchases to their apps. In addition to better documenting such gotchas, it would also be awesome if Apple added the ability for developers to roll back to previous versions (or at least 1 previous version) like Google's Android Market recently did.

On Mediocre Software - Where is the Love?

A recurring theme in some of the feedback we've seen for Audiogalaxy in reviews and direct feedback from users is how surprised they are when everything just works. A lot of the praise that the product has received goes something like this:

"Having tried and failed with several similar products I was shocked that when I used Audiogalaxy, it worked! No long delays, no dropping back to the initial screen, no anything wrong, just the music playing for as many hours as I needed."

"It does exactly what it's supposed to do. How often do you see that?"

"I never write reviews because most apps usually suck, but this is... different."

I won't feign modesty - we've worked pretty hard to make sure that Audiogalaxy works well. But that's not the point of this post. What statements like the ones above imply is that users are accustomed to underwhelming software. Most people, it seems, approach new software products with skepticism and wariness. Having been bitten by bad software several times in the past, they have become conditioned to assume the worst, and to be surprised when software works as it's expected to. While this might be true of other things too, in this case it points to a collective failure of the software development community. Commoditization of software developers/development and easy software distribution channels seem, on the surface, a boon to the software industry but are also a root cause for this mediocrity that people seem to now expect.

A telling sign of how extensive this perception is the following email I received from a friend after he used Audiogalaxy:

"Frankly speaking, my expectations were not very high. I was like, 'it's gonna be yet another process sucking the life out of my machine'. I was expecting to spend at least 30 minutes configuring everything, but was pretty much done in a couple of clicks and listening to music in a couple of minutes. It blew away my expectations in terms of how light weight it was (like 5 MB of memory and close to 0-1% cpu) and also super responsive in the browser to!!!"

On the surface, this comment seems to fall into the same bucket at the others above. However, this friend is actually a really good software developer himself. So this default notion of 'crappy software' isn't just users' perception. It is a dirty insider secret - we know about it, we expect it and yet we just suck it up and deal with it.

It's time to fix this, my fellow software geeks. You may say that people will vote with their money, and that bad software will fall by the wayside, and that the wheat will separate from the chaff, yada yada... All that may be true. But by producing crappy software, we're diluting the value of software as a whole, which eventually trickles down and affects each one of us.

Now this is the part of the post where I really should offer a silver bullet that will 'fix it all' and permanently change users' perceptions. Alas, I'll leave that to folks who are smarter than I. The one thing that I've learnt in my short career, and the only advise I can offer is this:

Build software with love, nurture it with care, take pride in its successes and share blame in its failures.

This mantra won't make you write bug free code or automatically make your software user friendly overnight - that's what experience, skills, iteration and good engineering practices are for. Heck, no software developer is anywhere close to writing 100% bug-free code. However, it will guide you in making decisions that lead to good, solid software which does what it says and says what it does. After all, software isn't just code - it is the culmination of every decision made in its journey from its inception to its current state.

A world where bad software/technology is an anomaly, not the norm, would be a good world - and it starts with us.

Integrating Core Data into your Three20 App

The Three20 library for iPhone app development is pretty cool - it take care of a lot of the 'mundane' code that you might need to write to accomplish basic tasks. Apple really should include some of the functionality of Three20 in the iPhone SDK itself, specially the Network stack.

As I built a more advanced app using Three20, I realized that it would be great to be able to persist to a database to make loading quicker when state on the server is unchanged. I needed some pretty basic Core Data integration in my app, and it was a little annoying that Three20 didn't have a Core Data-backed implementation of TTModel built into it. It seems like a lot of people want this, but there isn't any good post about it I could find. Jeff Verkoey, who currently maintains Three20, started writing a Core Data based tutorial but didn't get around to actually doing much with it. And there are a few threads in the Three20 Google group but nothing too comprehensive.

Having built this for my app, I figured I would post some more concrete information here for others who might find it helpful. I really want to just contribute a clean cut of this code as a TTCoreDataModel with unit tests to the Three20 source, but I don't have the luxury to spend time on it currently.

So here's a quick step-by-step on how to modify your existing Three20 based app to add basic Core Data support. The specific example I use fetches data from a server using TTURLRequestModel and writes it out to SQLite using Core Data entities. When displaying data to the user, it fetches data from Core Data using a NSFetchedResultsController, which efficiently interacts with Core Data.

Step 1:
Add CoreData.framework to your list of dependencies. Then, add the Core Data stack. Usually, you would do this in the AppDelegate. Jeff's AppDelegate files have this implemented, if you want an example. In my case, I wanted different databases for different Models, so I had these in my TTModel implementations.

Step 2:

Create your model. Add a new file of type "Data Model" in XCode (available in the 'Resource' section). Make sure the .xcdatamodel file gets added to your Target so that it's compiled into your binary. Look here for information on using the modeling tool. Once you've created your model, import the code for the Entity objects into your project. Now you're ready to start using Core Data.

Step 3:

Update your TTURLRequestModel implementation. Let's assume you're simply downloading some JSON and want to write it out to Core Data as an Entity you created in Step 2. Update your (void)requestDidFinishLoad:(TTURLRequest*)request implementation of TTURLRequestDelegate as follows:

A few notes about the code above:

• DBResource is my simple entity containing an "id" and a "name".
• The NSAutoreleasePool helps clean up memory allocated in this tight loop here
• -reseting the context makes sure that the data faults in as needed instead of staying in memory when you read it using the NSFetchedResultsController

Step 4:

In your TTTableViewDataSource implementation, create a NSFetchedResultsController and configure it to fetch your Entities as needed. Then, override the (id)tableView:(UITableView*)tableView objectForRowAtIndexPath:(NSIndexPath*)indexPath method to return a TTTableItem built using the Entity you fetched from Core Data

Now, your TTTableViewDataSource is configured to read results from the DB you created and display them in your view. Remember that tableViewDidLoadModel can be called multiple times, so clean up any previously allocated NSFetchedResultsController if you need to.

Step 5:

Implement your logic to delete/update your Core Data entities. Depending on how your app uses data persistence, you want to ensure that you update/delete your entities from Core Data. For instance, if you always want to fetch data from the server and load them into Core Data on first run, then update Step 3 to first delete all entities. If you wish to show local data when available but fetch when not, then update Step 3 to not hit the server before checking locally. When deleting entities, remember that Core Data only lets you delete a single entity at a time (so you must fetch all and loop through each to delete it). If you want DROP TABLE ... semantics you're going to need to maintain different NSManagedObjectContent and NSPersistentStoreCoordinator for each such table, and then simply delete the database file.

So, this isn't too hard really. Assuming you're familiar enough with building your Three20 app, my notes above should help you get started with integrating Core Data into your app. My app uses a TTSectionedDataSource, and with a few tweaks to NSFetchedResultsController it's simple to do that too.

An Apple A Day Keeps the Bugs Away

I've been working from home for the last year or so, and it has tremendously increased my productivity as a developer. Besides the obvious reasons (lack of distractions, no meetings, etc) there is one other advantage that I haven't often heard people mention before.

I have been able to find solutions to some really hard technical problems much quicker, and I attribute it to the fact that I work from home instead of in a more office-like environment.

Most software engineers will tell you that some of their best ideas have come to them when doing the most mundane of things - taking a shower, strolling across lawns, eating a snack and so on. I don't know the psychology behind this, but I know it's true for me. Something about being alone and thinking about your problem subconsciously, and yet being physically distracted with an unrelated task seems to work wonders. I have solved more tough problems while in the shower, when sitting down to eat an apple, when watering the plants or simply doing the dishes than I have sitting at a desk and just thinking. So obviously, being at home makes it much easier for me to find one of these mundane tasks that let me 'detach' and yet perform an activity that is beneficial to me both personally and professionally.

It's become a pretty important weapon in my software development arsenal - if I'm stuck on something for more than 30 minutes, I get out of my workspace and go do something mundane around the house or eat a quick snack. More often than not a solution magically appears in my head.

Back when I worked in an office, if I was stuck trying to solve a hard problem my best option was to find someone familiar enough with the problem space and bounce ideas off of them. This worked pretty well initially but didn't scale too well with problems that became narrower and deeper in scope - I often found myself spending a lot of time trying to catch others up to where it would be valuable to bounce ideas off them. And heading off into the office kitchen to reorganize stuff or watering random plants around the office would just make me look weird.

So, the next time you're having a hard time finding a solution to a problem in the office think about taking the next day off and working from home. Oh, and make sure the dishes aren't done. ;)

Slow onclick Javascript handling in Google Chrome 5.0.375.55

Google's Chrome has been a pretty slick browser, and it's speed was the primary reason it became my default browser. The V8 Javascript engine supposedly blows other engines out of the water, and it did feel noticeably faster than others - until this latest version.

Sometime last week, my version of Chrome was auto updated to 5.0.375.55, and I noticed that the website I've been developing became terribly slow. Because of a previous Chrome bug, I hadn't migrated my site from jQuery 1.3 to 1.4, so one of the first things I did was verify the fix and then update my site to use the newer, faster jQuery. As I began testing, I noticed a significant reduction in performance of some features on my site.

Initially I suspected it had something to do with jQuery upgrade. After a few hours of changing things around, I was pretty convinced the problem wasn't jQuery 1.4. I noticed that Firefox and even IE 8 didn't exhibit this latency, so now I started to suspect the new version of Chrome. I hacked up a couple of test pages, and I think I found the culprit. It looks like the onclick handler's performance degrades with the size of the HTML page, resulting in a very noticeable (1-2 second) lag once you have a table with 3000/4000 rows. Firefox and IE 8 don't seem to have any trouble handling this.

I've logged Issues 45620 in the Chromium project with details and a repro to track this problem. This definitely isn't good news for me - with growing adoption of Chrome and general perception that Chrome is the fastest browser, problems like these make website look faulty even when they might not be.

As an aside, I have noticed that Chrome feels like it has regressions more often than other browsers (perhaps a function of shipping more often?) I've run into several issues developing for Chrome - the silver lining is that bug fixes are quicker than rival browsers, but better QA before release would certainly help developers.

Other Chromium issues that I've had trouble with and have had to work around are Issue 7771, Issue 30693, Issue 33411 and Issue 36559.

Update: An comment on Issue 45620 identifies one of the reasons for the slowdown and a workaround.

Living Every Developer's Wet Dream

Back in June I wrote about new technologies I had started working with and my experiences learning those technologies. Having continued working with most of those since then, I've become incredibly comfortable and proficient with them. It's pretty normal to have a week where I write a bunch of PHP, a few SQL queries, some fancy Javascript, learn new CSS tricks, hack up features in client software using Win32/Cocoa, debug some corner-case issue using logs and a mini-dump and also write some straight up C/C++ server side code with unit tests. Initially, jumping languages and client/server boundaries came with large context-switch penalties but it's become almost second nature now, which feels very empowering and has had a dramatic impact on productivity.

Then, there are the one-offs - new technologies that need to be learnt, integrated into the product/service (in a way that's easy to understand/change in the future) and then 'forgotten' before moving on to the next one. These tend to be client-side technologies usually - server side stuff is much more on-going and stable. Some of the 'bigger' things I've done in the last 6-9 months that would fall into this one-off category are:

• Fiddling around with HTML5 <audio> and <video> before giving up
  For all the promise of new HTML5 multimedia standards, the sad reality is that we are far from any consensus. Different stakeholders have different motivations, which leaves the issue of a single standard codec up in the air. The proprietary nature of Flash, licensing issues surrounding H.264, Apple's opposition to including Ogg Theora in Quicktime and Microsoft's silence have resulted in the current stalemate. And this shows when you experiment with the different browsers. Development for web/mobile-web is poised to become radically easier once/if this is sorted out, but there is a lot of pain to be endured before then - pain that wasn't worth it to me presently.
  
  
• Flash audio support
  Having decided to not want to build a flash media plugin myself, I first played around with this MP3 Player and later switched to the much more powerful SoundManager2. With a few tweaks, I was able to make it do exactly what I wanted.
  
  
• Facebook Connect
  For single sign-on and any sort of social media integration, Facebook Connect is a must have. Having implemented Facebook Connect for my website and for iPhone, I must say that I am very pleased with the facilities offered by the platform and the consistency across client/server applications. XFBML and FQL are nice touches. My only gripe is that, for a mature and widely adopted platform, documentation seems disproportionately poor as compared to the richness of the Platform APIs.
  
  
• Online Payments
  There's a lot you learn about how safe online transactions really are (they're not) when you dive into how payments really work. I developed a quick payment portal using Payflow Pro by PayPal. Can't really vouch for Payflow Pro yet, though. A few weeks after I finished working with Payflow Pro, PayPal X was announced. I'll eventually get around to migrating to it, I guess.
  
  
• Identicons
  Simple, cute, pretty. I use the ones provided by Gravatar.
  
  
• Javascript Craziness
  One of the crazier things I've done, among others, with Javascript is to try and emulate a desktop-software like user experience in a website. Dragging, dropping, keyboard navigation, sorting, multi-select with Ctrl/Shift/Meta... oh boy! It look a while, but with some help from jQuery UI (the different components work great individually, but try to also make a sortable a draggable and a selectable, and you're in a world of hurt) and a lot of long nights, I finally got a lot of this done. For Forefox. Then I spent some time making it work for Chrome. And then a whole bunch of more time making it work for IE8. It sorta works in IE7, and not at all in IE6, but honestly if you're still using those browsers you should be chastised. Someday, I want to wrap all this into my own little plugin that others can use. If you told me a year ago that I would be some kind of JS nut, I would have laughed. Ha!
  
  
• iPhone web-app (before I ditched it for a full blown iPhone app)
  Not wanting to deal with Apple and their strange AppStore practices, I decided to try and be really creative with native Quicktime support built into Safari on the iPhone. I almost got all the way to hacking up really neat functionality, complete with callback from Quicktime into my website, background playback and an rich interactive experience. With Flash coming soon to other mobile platforms, this would have been ideal for providing mobile experiences without needing multiple platform-specific applications. However, the iPhone web app felt clumsy and not polished, so I ditched it and went the iPhone app route. I reserve comment on my experiences with development for the iPhone - my app hasn't been submitted/approved yet. ;)
  I also foresee myself playing with other mobile development platforms in the very near future. I think a blog post comparing my experiences with the different platforms might make for good reading.

In addition to these one-offs, there are the perpetual how-will-it-scale and how-will-it-perform issues that need to be considered for every little feature. One of the harder lessons for me was to change my default mindset about how I approached design. While I wouldn't say I optimized prematurely, my design was heavily influenced by potential for optimization. (segue: This, I now realize, is a result of my years at Microsoft. At companies like Microsoft, you do things once and you do them right - rapid iteration is often not possible or not practical. So, while this approach has its merits it's not ideal for agility in startup environments). The new approach that's imbued into me now is to build it quick and make it simple first. Simplicity is the bedrock of agility. Simpler code is easier to revisit and to scale. Complexity is technical debt that you don't really need or want.

What's the biggest thing I've learnt in the last year? I learnt how to learn efficiently.

You can spend hours studying/learning about something and barely scratch the surface. Or you can study to retain information for very short periods of time. Exceptional developers are able to identify when they need to deep-dive vs when they need to simply skim the surface. They can instinctively answer the "How do you learn enough to make well informed decisions and yet don't spend a disproportionate amount of time learning without producing?" question. Transient learning is only good when you never need to revisit a decision - such cases are rare. Otherwise, learning to execute quickly and leaving a trail behind you is a crucial skill. I believe I'm well on my way down this path.

So yes, in many ways I am living a developer's wet-dream. And fortunately, I am able to do a little more than just development stuff. But I know this probably won't last forever. This series of blog posts is my attempt to capture my memories for when I'm not doing this everyday anymore.

SecureCRT and Windows 7 - The Case of the Magically Disappearing Window

Ever since I upgraded my copy of SecureCRT to 6.5.1 on Windows 7 x64, I started noticing really strange behavior. Occasionally, my SecureCRT window would just 'disappear'. What do I mean by 'disappear'? The SecureCRT icon on the Windows 7 taskbar, which appears like this when a window is open:

would suddenly change to this, which is what an icon looks like when the process has no windows open:

I could still see SecureCRT.exe running, but didn't find any windows for it, even in the Alt+Tab menu. Nothing appeared in my system tray either except for the network connection, volume and system time information.

Once this happened even after a system restart (isn't that the usual fix for everything?! ;)) I got curious. Eventually, I realized that this happened each time I minimized the window. Somehow, minimizing the window is muscle memory for me now and I don't really notice I'm doing it until I consciously try.

Minimizing a windows shouldn't ever cause the window to just 'disappear' so I started looking at the different SecureCRT settings and noticed a curious "Minimize to Activator" setting that was checked, and seemed suspicious. Unchecking it was all it took to restore sanity.

What is this 'Activator'? It's an icon that sits in your system tray. But Windows 7, by default, hides away all my system tray icons so I had no idea this even existed until I started investigation this issue. This is the 'Activator':

SecureCRT removes the window and sticks it into this Activator. Without any sort of visual cue or notification bubble. By default. Ugh! To restore it, you need right click the icon and select the "Restore" option. Umm... isn't that the whole point of Windows' built-in minimize implementation?

This seems like a poorly designed feature, especially given how Windows 7 treats tray icons (and I know SecureCRT v6.5.1 has some Windows 7 specific features so this isn't accidental). Making the process 'appear' like it's not running and sticking into into the 'Activator' (why isn't it called 'system tray icon'?) instead seems like bad design. If anything, this should definitely not be on by default.

Ed: Title inspired by Mark Russinovich's blog posts

Update: Just got an email response back from SecureCRT's support staff to a query about I this I had sent last night. She clarified "... the global option is **not** enabled by default. The session option by default is a "tri-state" option, meaning it uses the setting found in the global option."

XSRF/CSRF Vulnerability Explained in Non-Geek Speak

I recently explained what cross-site request forgery (CSRF/XSRF) is to a friend who isn't a programmer/CS person and I was pretty amazed at how quickly he understood the implications of not protecting against such attacks. With the proliferation of social interactions on the web now, such attacks can be pretty damaging to consumers of new media and e-commerce sites.

The Wikipedia article about CSRF does a pretty good job of explaining the issue - most people familiar with web development should be able to understand it easily. Basically, this attack lets legitimate unsuspecting users submit changes to a site they routinely interact with, but without their knowledge. CSRF attacks leverage your own browser and the trust you associate with it to perform potentially damaging actions on your behalf.

Here's an example of what such an attack might look like. Let's assume that Facebook wasn't protected against such attacks, for the purpose of this illustration. In reality, I'm sure Facebook engineers have done a good job of making sure this can't really ever happen. That's not true of every site out there, though.

When you sign into Facebook, you have the option of remaining signed in. To do this, sites like Facebook set a browser cookie which identifies you to their servers. This is great because it lets you visit the site frequently and navigate the site without needing to enter your password every time. Browsers are good about securing your cookie - they make sure that no other site besides Facebook can see this cookie, so it's not possible for the author of www.hacker.com to steal your cookie and present it to Facebook and then impersonate you. However, each time you go to any URL that contains www.facebook.com, the browser diligently sends over all cookies associated with Facebook, because the receiver of those cookies is www.facebook.com.

Once on Facebook, you do things like update your status or write on a friend's wall. When you do that, the browser sends your new status update to Facebook along with the cookies. By inspecting the cookie, Facebook is able to validate that it is indeed you trying to update your own status, so it proceeds to allow the action. Under the covers, your browser is sending over two pieces of information - some data (your status update text, in this case) and the cookies. The browser was triggered to do this by you clicking the "Submit" button. This sounds pretty straightforward so far.

Now imagine you're browsing the web and land on this page that talks about world hunger and how you can help the cause by simply clicking a "Help the Children" button. With only altruistic intentions, you click on the button. Nothing visible happens, or maybe a "Thank you!" label replaces that button. Then you simply move on to the next thing you were going to do. Suddenly, you start receiving email notifications about friends responding to your latest status update on Facebook, except you never posted any update. You visit Facebook and sure enough your status now says "I am a loser!", updated 2 minutes ago. Huh?! You didn't set that status. You panic - maybe someone hacked your account? Or someone guessed your password? What's going on?

What really happened was that the world hunger website you were on was written by a malicious attacker, and when you clicked the "Help the Children" button that website didn't really contribute 10 cents to UNICEF, but instead told your browser to send a new status update with the text "I am a loser!" to www.facebook.com. This site couldn't access your cookies directly, so the browser didn't do anything bad and stuck to its promise of not letting anyone else see cookies that shouldn't be visible to them. Remember, the attacker never looked at your cookie at all while any of this happened. However, when the attacker asked the browser to send data to www.facebook.com, the browser added the Facebook cookies to the request. So now Facebook looks at the cookies and the data, thinks that it is indeed you and goes on to update your status. Boom!

So what went wrong here? The browser didn't do anything bad - it did exactly what it's supposed to. It hid your Facebook cookies from everyone except Facebook and it sent data that submitting the "Help the Children" button asked it to send. You didn't do anything bad - you simply clicked on a link that was meant to help needy kids. You did have cookies enabled (default for most browsers) and decided to check that "Log in automatically" option, but those seem like reasonable actions.

The flaw here is with the website you trusted - Facebook, in this example. Sites like Facebook that rely on cookies to identify and authorize user actions need to do more things than just validate your cookie when data is presented to the site on your behalf. They need to make sure that place from which the data was sent was 'good', or that a malicious author cannot create data the same way that www.facebook.com can. They need to make sure that when you update your status, the data contain more than just your new status text - there must be some kind of secret information in there that can only be generated by Facebook, and this secret must be different for each Facebook user. They can even control how long cookies and secrets remain valid, and have ways to periodically require users to sign in and generate new unique secrets for users.

Of course, this is just a silly example. I'm sure Facebook is perfectly adept and handling such attacks. However, a vulnerability like this could impact everything from online review sites to banking and e-commerce sites. It's always good to be aware of things that could go wrong online - the Internet is definitely not as safe as you might think it is.

Know What you Inherit

I ran into a couple of bugs recently that took longer than they should have to fix. Like all bugs, it seems obvious now that I know what was wrong but I think I spent a total of 5-6 hours investigating and fixing this across multiple platforms (hey, this is what makes developing software fun!). Here's hoping that those fancy search engines will help someone else find this post and save them time incase they have a similar problem.

I was working on some code that launched another process. This is a fairly mundane task. Interestingly though, the process launched went on to kill its parent and re-launch another copy of the parent. So, process A is running and launches process B, which then kills A and launches A'. Sounds kinda simple, right?

Well, everything seemed to work until I noticed that some resources needed by A' weren't available (ports/files). That seemed odd, given that A had been using those same resources successfully, and those should now be available since A was gone. The title of the post gives it away, but the problem is that a fork causes child processes to inherit the file descriptors/handles of the parent, so the files/ports opened by A were inherited by B and subsequently by A'. This meant that they weren't really cleaned up by the OS when A exited. Hence A' could not re-open those resources (some were locked for exclusive use).

The really interesting aspect of this bug was how it manifest itself on different OSes and the different fixes.

On Windows, TCPView showed the ports in use by a "<non-existent>" process. The PID of this process was that of A, which was already gone by now and didn't show up in the list of running processes.

After some analytical debugging, I guessed that the handles were being inherited by the children resulting in these strange 'orphaned' ports. The solution on Windows was to use the CreateProcess API with the 'bInheritHandles' flag to FALSE. Problem solved. By the way, the not existent process IDs attached to those ports seem to imply that the OS tracks ownership by processID, and hence cannot validate that the new owner of these is the child process now that the parent is dead - the child obviously has a different process ID. This might not be really how it's done, but seems logical.

The Mac was much more interesting. The manifestation of the problem was simply unavailability of the resources. There isn't an equivalent of the bInheritHandles flag for the fork command on UNIX. To cut to the chase, the solution was to fork the process, close all open file descriptors that it inherits and then call exec.

Now, the Mac guys have added a nifty 'open' command to the system which helps launch processes as a child of the launchd process, which doesn't inherit the callers file descriptors (it's parent is the launchd process). However, this open command has a couple of pretty severe limitations. It can only open app bundles, not native binaries. And you can't pass in command line arguments to the application you open. FAIL on both counts! Mac purists will claim that command line arguments aren't the 'right thing' on the Mac, and that Apple Events should be used instead. That's BS, really. For anyone writing platform independent C++ code, it's just not practical. People don't realize that not every app on the Mac is a Cocoa/Carbon app. Anyways, so on OS X I needed to be a little creative and have A launch B, but before doing anything useful in B enumerate all open file descriptors and close them (readdir() /dev/fd on FreeBSD and Darwin enumerates all open file descriptors). Oh and by the way, you might not want to close FDs 0, 1 and 2 - they're stdin, stdout and stderr respectively. This post on StackOverflow has some good related information incase you're curious.

To Every Web/UI Developer I've Scoffed At - Apologies!

I am/was a cocky programmer. Having only worked on system and application level programming thus far, I really didn't appreciate how hard building a good, usable website or application is. I knew that an application or a system is only as good as how usable it is, but I really considered usability to be a 'simple' problem and by inference I assumed that writing the code behind what you see on a website or an application is also 'simple'. As my career matured I suspected I might be wrong, and now I know I was!

Over the last few weeks, I've been messing quite a bit with Html and CSS to make stuff look non-ugly and Javascript/Ajax to make stuff feel non-sluggish. And the combination of these together to make non-ugly things appear non-sluggish. Of course, this also means dealing with everything that goes on in the middle/backend to support non-sluggish, non-ugly things. Oh, and once you have something that's not ugly and not sluggish, you have to worry about the bad guys. Cross-site Scripting, Cross-site Request Forgery, cookie theft and poisoning, etc are things I didn't think much about previously. What cans of worm those are... !

And then a couple of months ago I spent some time making an mac application non-ugly and non-sluggish. Same deal, pretty much.

Working on these new technologies sure feels like being a kid in a candy store, but I have to confess my new found respect for good UI/Web developers (and I am not one, by a long shot). I may have snickered behind your backs previously, but it sure won't happen again!

New Technology (for me) Roundup

Over the last almost 2 months, I've messed with several new technologies - some extensively, some briefly - that I hadn't used previously. This post is primarily a journal entry documenting my impressions of and experience with these, so far:

• PHP - Exposure : Medium
  I think I really like PHP. It seems like it's a good blend of a traditional programming language and a scripting language. I like that it allows for well designed software with OO constructs being first class citizens. I also like the flexibility that the scripting language aspect of PHP provides. I wish there were some kind of PHP compiler which helped catch silly typo-like issues before runtime, but I guess that's the price you pay for having the conveniences of PHP. I also think I would like having a good PHP debugger - currently I use code review and tracing as debugging aids. They've worked so far but I hope I never have to chase down something really nasty or subtle using these techniques.


• Objective-C/Carbon/Cocoa - Exposure: Medium High
  I previously wrote about my experience with programming for the Mac. For someone familiar with C and working with a runtime framework, this shouldn't be hard to grasp. I think I'm pretty comfortable with these technologies, but in many ways these are forced down your throat if you want to write software that runs on a Mac, and looks half decent. I dove deep into a few things related to these technologies (debugging crashes, file system events, application run loops, network support, deployment) and it was kinda fun and fulfilling to accomplish complex tasks with these technologies.


• Bash Scripting - Exposure: Medium Low
  To be productive in a Linux development environment, mastery of scripting is key. I know just enough to get me by, and I'm sure there's so much more I could do to improve. The rule of three seems to work well... if you do the same task three times, script it. And most times, you'll discover it's already been scripted by someone else or simpler still, exists as a built in set of commands. This seems like one of those endless holes of expertise that never ends. My goal is to deal with this as needed instead of being overly enthusiastic about becoming a crazy script monkey.


• Javascript - Exposure: Low
  I'm actually not too bummed about not having had to mess with more Javascript. I know people who just do the craziest things with Javascript and AJAX, but I'm not creative enough (or perhaps not motivated enough) to do anything more than the minimal. Unfortunately, a minimal website only works if you're Google or Craigslist. I haven't messed with Javascript for more than just a few hours, so I don't really have any strong opinions except that for someone who isn't familiar with scripting it takes some getting used to. Something like Script# might be an ideal transitionary tool though (I haven't played with it at all).


• MySQL - Exposure: Medium Low
  MySQL seems really slick. It feels great for simple database functionality (haven't really done anything too fancy with it) and supports some SQL syntax that comes in handy versus Microsoft SQL Server (insert ... on duplicate key update, create table if not exists, etc). PHP support for MySQL seems like a great integration win for both technologies. I haven't really dealt too much with administration and performance issues though, two areas which are super important to a great database solution, so I can't comment on those.


• Python - Exposure: Low
  I wrote a couple of python scripts to get some pretty powerful tasks done. I like that it has a bunch of libraries that enable you to write pretty fancy applications if you're so inclined (I'm not, and likely never will be). Like most scripting languages I'm amazed at how much can be accomplished in a few lines of code. Generally though, I don't like too much magic being done for me when I don't really understand what's happening and my experience with Python certainly made it seem very magical. Which is great for getting small tasks done, I guess.


• Vim - Exposure: Medium High
  This is a big once - even after almost two months of using this daily, I'm not nearly as comfortable with it as I was with the Visual Studio IDE, which makes me feel like I'm not being as productive as I can be. I think I learn one or two new tricks every day that make me wonder if there's anything Vim can't do. It's another one of those 'rabbit hole' technologies where experts can tune it to crazy levels. My approach is more along the lines of 'fix it when it annoys you' which seems to be working (slower than I would like, I admit). It certainly has the potentially to make me super productive once I get better with it, and that's making me stick with it.


• Others - Exposure: Low to High
  Technologies that aren't really 'new' for me but I was out of touch with for the last few years and have reconneced with recently:
  - C
  - Linux
  - Win32
  - Batch file scripting
  - Makefiles
  - gdb

So yea, as you might be able to tell, the last couple months have been insanely fun. I feel like an intern again (in more than one way ;-) )!

f.lux - I Like!

Staring at multiple computer monitors late into the night isn't as pleasant as it should be. I've noticed that the constant bright glare from the monitors causes a lot of fatigue. f.lux is a very simple and yet highly effective tool that has made coding so much more pleasant. I very highly recommend this to anyone who spends long hours working at a computer. Available for Mac, PC and Linux. (PS: MacBooks already have similar functionality built-in that uses the integrated camera to detect ambient light and adjust brightness accordingly).

Developing for the Mac - How does it compare?

I spent most of this week developing some Mac OS X software. Having never before written a single line of Objective-C or any Mac software, I was pretty excited. Here are a few observations of developing for the Mac compared to the Windows platform (primarily .NET) from a week's worth of writing an application using Cocoa. Of course this isn't exhaustive and only includes things I had to deal with (disclaimer - opinions subject to change as time passes):

• Overall, developer reference documentation on Apple's Developer site seems more current and easy to understand than that on MSDN. A nice touch is the reference to examples which use a particular method inline with the documentation of the method <example> (MSDN does it using inline code, so it's not too different). My favorite feature about documentation though is the logical grouping of methods and properties arranged by tasks - that makes it very easy to get stuff done. I think this is the result of being the underdog in this space.


• Mac OS X applications' uniform usability and consistent aesthetic isn't a fluke. Apple's documentation regarding recommended usage is precise down to individual visual controls and yet comprehensive as far as Human Interface Guidelines go.
  


• XCode doesn't have anything on Visual Studio. Visual Studio seems, hands down, a better integrated development environment and debugging application. VS 2008 is really pretty awesome.
  


• Cocoa is nice, but still has some catching up to do with .NET. Cocoa libraries provide just enough abstractions to get stuff done, but they don't seem to be nearly as comprehensive and well thought out as the .NET libraries are. I found myself cursing some abstractions/APIs in Cocoa for their primitiveness compared to Windows pretty often. Generics give .NET an edge that Cocoa still doesn't have. Microsoft has done a great job of handling little things, such as boxing, which make development easier than it is for Cocoa. 
  


• XCode and Interface Builder are unfortunate crutches required for Mac OS X development - I would claim that it's impossible to write any decent application without using both. This sounds great until you realize that IB, which really is a designer, does so much magic for you that you just cannot with code. Linking up IBOutlets, for instance, is something I cannot imagine can be done outside of Interface Builder and has way too much magic associated with it. As a developer I often felt at the mercy of these tools, instead of being the master of it. It's fortunate that this is not true of developing for Windows - there is a lot of helpful magic that Visual Studio builds in but it's certainly not required to develop a decent application.
  


• Overall, the Mac platform (Cocoa) seems more closed than it should be. This sounds weird, considering that it's all basically Unix and standard Unix libraries are provided or can be added relatively easily. My comment here related to how accessible and complete Cocoa/Carbon libraries are. Getting access to weird parts of the system is easier on Windows that on the Mac platform, I believe.
  


• Deploying Mac applications is a snap, better than Windows. However, I think this is related to how developers are accustomed to (ab)using resources in Windows (registry, etc). Not having a lot of Windows' 'legacy' facilities on the Mac makes developers think hard about what their app really should be doing. I claim that good developers should be able to create x-copy deployable apps even for Windows, though, so it's unfair to blame the platform here.
  


• The fact that it's built on Unix makes the mac support a lot more languages, especially scripting languages, out of the box. Windows is catching up fast though - integration of interpreted languages with .NET and PowerShell bring it at par with the Mac (I hear it might be better, but I don't have much experience).
  


• Testing on the Mac seems easier. The fact that there are a very finite set of configurations compared to the Windows platform makes life less complicated for developers. It's almost true that if it works on your Mac it's probably doing to work on most Macs (modulo a few exceptions).  History says this has traditionally been an advantage of the Windows platform, but I'm not so sure it's a good thing anymore. This one's probably the most controversial of my observations above and likely to change as I do more Mac work.
  

So, who wins? I would say Windows for the overall developer friendliness. Apple has some catching up to do in several areas. That said, the Windows platform could learn more than just a few things from what Apple seems to be doing right. 

It's a fun time to be a software developer.

Problems at Facebook?

I was thinking about the whole recession thing and how it impacts Microsoft. As I worked through my train of thought, I remembered the huge 899 million Euro fine that the EU slapped Microsoft with in Feb 2008. I remember thinking then that the EU was almost treating Microsoft like an ATM (probably inspired by this post).

I wondered why in this time of recession the EU hasn't yet again tapped its ATM and thought I'd update my Facebook status to say "Viraj wonders why, with this recession n all, the EU hasn't tapped its ATM (Microsoft) again using the same anti-trust excuse...". To my surprise, I got a little popup that said "Validation Error - A validation error occured" and nothing more informative.

Interesting. What exactly is Facebook validating, and did I have some red-herring text in there? I would expect filters for expletives, racial slurs, geo-political issues... but my status message didn't seem to violate any of those. I tried the following variants, which all failed:

"Viraj wonders why, with this recession n all, the EU hasn't tapped its ATM (MSFT) again using the same anti-trust excuse..."

"Viraj wonders why, with this recession n all, the EU hasn't tapped its ATM again using the same anti-trust excuse..."

"Viraj wonders why, with this recession n all, the EU hasn't tapped its ATM again..."

"Viraj wonders why, with this recession n all, the EU"

"wonders..."

Ok, so it's certainly not the text of my status message that's causing this. I started poking around, and my 'Home' page seemed OK - I had news items, status updates from friends, etc. However, when I jumped to my own profile, I was suprised to find that all data I had published was blank. No wall posts, no posted items, no recent history. However, my photo albums seemed to fine, as did my profile information.

My guess is there's been some kind of data loss-ish issue at FB (a lot of their data is simply held in caches so maybe a cache and replicas blew away - perhaps Wall posts are not persisted?). Anyways, I'll try to update status again and see if stuff comes back... meanwhile, I'm locked out of changing my Facebook status...

Any guesses about what might be going on?

Update: I can't seem to be able to clear my previous status message either... the status is cleared in the webpage (local javascript clearing it, I would assume) but on refresh I see my old status again...

Update2: And we're back... that didn't last too long. Cool!

Doing Something "Just Because We Can!"

Working on a distributed system like Live Mesh is a lot of fun, but finding certain types of bugs can be interesting, to say the least. Going from writing code that executes locally on a workstation to writing code that is executed across several hundred machines is an extremely refreshing experience - you quickly learn to spend a good amount of your time and energy building a good logging infrastructure and leveraging it to diagnose problems. There are several other things you learn too, but that's worth a series of its own posts.

With a large amount of logs comes a large amount of interesting data, and an even larger amount of potentially interesting data. Good logs can go a long way in helping infer information about the scale, health, bottlenecks, usage etc of the system. Much time is spent automating such tasks so that useful reports can be generated from logs. Every so often, someone (usually a non-tecnnical person) asks for additional interesting data that would take time, effort and money (in terms of resource utilization) to extract. However, almost all such requests are missing information about what the requester hopes to conclude based on the retrieved data. My instinct is to always push back on such requests, forcing the requester to justify what questions the data will help answer or what conclusions it'll help lead to. If the data gathered does indeed help answer a crucial question, it's worth spending time and money mining the logs for it.

Very often, the reason for the request is "just because we can", in which case I believe the time/money/effort spent is unwarranted unless you have people/machines sitting around doing nothing, in which case you arguably have other problems.

Moral of the story - try not to ask someone else to do work (or spend your own valuable time doing work) without knowing what you intend to get out of it.