Coversheet of US 7,751,431.

There is one type of junk mail I don’t mind seeing in my postal inbox.  Whenever the US Patent Office grants you a patent, all kinds of opportunists from the awards, personalization, and engraving industry spam you with offers of beautiful plaques engraved with your patent’s front page.

I experienced this junk mail avalanche once again this past summer when US 7,751,431 was granted.  This patent covered work I did with my Motorola colleagues Jim Ferrans and Mike Pearce back in 2004.   Back then we were studying distributed multimodal architectures.  More specifically, we were looking for ways to integrate a speech recognition modality into applications running on mobile handsets, where the speech recognition was resident in the network.  This is one approach that we came up with.

More recently, Jim Ferrans and I published a book chapter in which we survey a wide variety of issues and findings  that resulted from our foray into mobile application architectures with network-based speech recognition modalities.  The book itself is rather pricey, but happily you can download a copy of our chapter for free from Motorola’s technical publications site.

Last month I attended the Google IO 2010 Conference in San Francisco along with my colleague Dr. Hans Dulimarta. The conference is Google’s annual developer’s conference where the company announces their new product initiatives and also offer in-depth sessions on various Google technology platforms. The event attracted a sell-out audience of over 5,000 technologists from all over the world, and was indeed well worth attending. Among the key new announcements Google made was the next version of the Android mobile platform (version 2.2, dubbed “Froyo”) which should start showing up on devices near you this summer.  The company also handed out some rather nifty free gear to attendees.  Weeks before the event attendees were sent a free Android handset (either a Motorola Droid or the Google Nexus One, depending on your country of origin) with a free 30 day service plan.  At the event itself, Google handed out free HTC EVO 4G handsets with a free 30 day service plan to every attendee!

The crowds at Google IO attending the morning keynote session. (Over 5k attendees)

The two areas of primary interest to me relating to my courses and mobile lab at GVSU are Google’s offerings in cloud computing and mobile technology. We use both the Google App Engine platform and Android mobile platform in a variety of CS courses and research projects.

I attended in-depth technical sessions on App Engine. Many of the new improvements to the platform and knowledge I acquired in the sessions pertain to performance optimizations. Two particular topics I learned about and plan to integrate into my future distributed computing lab assignments are Memcache (high performance distributed memory object caching system) and asynchronous URL fetching. Both of these are useful techniques for improving efficiency of applications deployed in the Google Cloud on App Engine. One of the students in our lab has already incorporated these techniques into one of our App Engine projects since we learned of them at Google IO.

Google cupcakes for dessert!

I also attended a number of sessions on the Android platform.  The new 2.2 release of Android contains a whole host of new interesting features.  From an end user perspective, features such as support for native tethering should be very interesting (assuming the operators don’t cripple or charge for it…), and the performance improvements yielded by the new JIT compiler should also bring some welcome improvements.  And of course if Flash is something you can’t live without, you’ll like 2.2 as well!  From a developer’s perspective,  we’re excited about the new Android Cloud to Device Messaging API which allows applications running in the cloud to push messages to Android mobiles.  These messages are “abstract” notifications and can be responded to programmatically by applications running on the mobile phone.  This sort of functionality will make it possible to tightly integrate off-device interactions with code running on the device.  For example, consider a desktop web browsing experience where a user encounters an address.   One could click on an address displayed on a web page, drag it over to a mobile phone icon, and the map pops automatically on the user’s mobile device.

I also attended an interesting session where Android user interface “best practice” patterns were presented by Googlers. The presentation used the recently released Android Twitter client as a case study to present how Android user interfaces can and should be implemented to provide highly usable interfaces.  UI is an area where I believe the Apple iPhone continues to hold a fairly significant advantage over Android, from both a developer and end user perspective.  Nevertheless, I did leave the conference with the impressions that Google knows this very well, and is working hard to make sure the Android UI moving forward is compelling alternative.

Although Google offers a significantly discounted registration rate for faculty, I was rather disappointed to discover there did not seem to be much of a presence with regard to attendees from academia. I think this is unfortunate, given the pervasiveness of Google technologies within CS curriculum these days, as well as in the general university environment (gmail, Google docs, etc).  Perhaps Google should allocate more of the special rate admission seats for academics?  After all, we do have a significant amount of influence on the next generation of developers… That said, I did very much enjoy meeting up with a lot of Motorola friends, and folks from other companies that I’ve crossed paths with prior to becoming an academic.

All in all, Google IO was a very informative conference  – a literal “drinking from the fire hydrant” experience.  It also wins the prize in terms of generous free gear handouts!  I’m looking forward to attending next year.  If you didn’t make it to the conference yourself,  beginning this week, Google is publishing the Google IO presentations on YouTube freely accessible to all. Enjoy.

Not so long ago one of my students (John Spencer) introduced me to Jef Raskin’s book “The Humane Interface“. While I have yet to read the book cover to cover, it’s got some interesting ideas in it including the notion of a “zooming user interface” or “ZUI”. Independent of John’s suggestion, another friend of mine (Zach Moazeni) visited our lab recently and gave a talk on Ruby on Rails. Zach gave his talk using prezi.com – a zooming presentation editor which really piqued my curiosity.

During our recent Spring Break (when my students were all basking in the sun down south) I started playing around with Prezi (they do make if free for educational purposes) and so far I like what I’ve seen. I’m teaching Android in one of my courses, so I started using Prezi for some of my lecture materials. Here’s a sample (click on the fullscreen option for best effect):

There are a couple of things I like about Prezi. First is its ability to focus and shift attention on presentation content in a natural flowing manner. Second, it’s sort of a hybrid between a PowerPoint presentation and a screencast. Its more dynamic than the former and more “macro” than a screencast. For example, when I browse the prezi gallery on prezi.com, I find I get more out of these presentations than a standard PowerPoint presentation (because of the ability to focus and shift attention) but find them not nearly as time consuming and tedious as watching a detailed screencast. Finally, the prezi editor is fairly simple to learn and work with, though I’m sure my initial output probably could use some serious refinement!

Prezi is based on Flash so it will work on any PC, Linux or Mac (but not on your iPhone or iPad!!). You can serve up your Prezi presentation online or you can download a read-only copy to distribute yourself. There is also an offline editor, though I prefer the online editor since I can then get at my presentations from any machine.

There are also some features I’d like to see in a future version. For example, when I’m establishing a path through multiple objects within a frame I’d like to be able to “group” that path as a sub path and refer to it at a macro level. As it stands you really can’t create a path until all the information is out on the table. The equivalent of a “snap to” wouldn’t be bad either when creating content.

I’ve posted my Android Prezis on my Screencast and Prezi Tutorials page. I’ll be adding more in the coming weeks. While you’re at it, give prezi.com a test drive yourself. I think you (and your audiences) will find it a welcome breath of fresh air after decades of PowerPoint and similar presentation tools.

Drill down to a product page on Walmart's iPhone app and they make it easy to consult your friends advice via Facebook.

Every mobile application targeting a retail or hospitality use case seems to have the obvious features: finding business locations, contact information, product/offering information, and in some cases user reviews/ratings of what’s being offered.  These apps amount in many ways to a repackaging of the merchant’s existing website as a native application for the iPhone or Android mobile device.  At the same time, there are also a number of interesting features that go beyond this basic “we gotta have an iPhone App too” mentality, and are beginning to utilize the mobile platform in new and interesting ways.  In what follows we highlight five such features that we’ve spotted recently in one or more free applications in the retail/hospitality space.
1. Integrate the experience with the customer’s social graph. You would think this would be obvious these days, given the rate at which major brands are staking out their outposts on the latest social media frontiers.  Yet, very few apps in this category integrate with the customer’s  existing social media channels.  GAP and  Walmart are two notable exceptions to this.  Once you’ve drilled down to a product detail page, Walmart’s iPhone app (dedicated to their consumer electronic offerings only) suggests you ask your friends on facebook (or via email) for advice.

GAP's StyleMixer App let's you beam a potential new outfit up on your FB wall to get reactions.

The GAP’s StyleMixer iPhone app goes a step futher in that you can dynamically configure an outfit (see next point) and then share it via GAP’s homegrown social community,  Facebook, and/or email addresses.

2. Assist in non-trivial product configuration/selection. Ever try to order pizza over the phone with a room full or car full of hungry adolescents all telling you their topping preferences at the same time?   Not to worry, Pizza Hut’s iPhone application is a well-executed example of using the mobile platform to quickly whip together a precise order with a lot of non-trivial detail.  There are other good examples of this same concept being used in retail to assist customers in product selection.  Both Target’s and BestBuy’s iPhone applications provide novel gift selection wizards that let you quickly select an appropriate gift for that hard to please mother-in-law. As we already mentioned above, GAP’s StyleMixer iPhone app let’s you assemble a new outfit, complete with accessories, and then share it with your friends via Facebook Connect.

Configuring and ordering your pizza is a snap using Pizza Hut's iPhone App.

3. In store positioning. This feature is not exactly widespread at least at the moment, due to the non-trivial nature of doing precise positioning indoors.  However, there are low tech ways that can be used to make the shopper’s life a lot easier.  Though not particularly well executed (latest update seems to have broken the app, at least on my phone) the “Point Inside” iPhone app is a good example of this approach.  The application provides floorplans of most of the larger malls within the USA.

Find your way around the mall with the Point Inside iPhone application.

Users can specify  where they want to go (by selecting the destination store from a list).  A Pin is then overlaid on the mall floor plans to help you gather your bearings and proceed to the store of interest.  The functionality actually goes well beyond the “you are here” directories physically situated throughout the mall in that in addition to finding stores and their contact info, it helps you remember where you parked your car, and/or quickly find that much needed restroom or ATM.  There are however, lots of ways to improve this particular app and make it truly useful for those of us who habitually lose our way in the shopping mall.

4.  Product interest/intent signaling. Making it dead simple for a user to quickly pull up information on a specific product/offer via a mobile phone is advantageous for the merchant as well as the customer.  One of the advantages online retailers like Amazon have is that they have a lot of information about the individual end user.  This information is at a very fine resolution and includes what products you’ve looked up, how long you spent staring at the product information between clicks, etc.  Traditional bricks and mortar retailers have little more than a list of past purchases to work with.  Using the mobile device as a sort of in-store concierge should provide better customer services (e.g. no need to find the guy with a blue shirt in Best Buy – just look up the reviews on your phone) and also help gather a lot of useful customer information beyond the end purchase decision.

The first generation of mobile applications supporting this sort of functionality have taken the obvious route – use the phone’s camera to get an image of the product’s UPC code and then access network services to pull down the relevant product metadata.  Two good examples of this are the Shop Savvy and RedLaser applications.  However, if you have any experience at all using a dedicated barcode scanning device you’ll find that imaging barcodes with a mobile phone is tedious and sometimes not possible in low light situation or when product packaging is such that there is glare on the UPC or its all crinkled up.  These apps have a backup plan of course, in that if all else fails you can type in the UPC code.

You'll need a steady hand, good lighting, and a phone with macro lens to get a good barcode scan with ShopSavvy.

What is shaping up to be an even more interesting  approach than scanning barcodes,  is that taken by Amazon in the “Amazon Remembers” feature in their iPhone app.  Here you simply take a picture of the product of interest and send it off to Amazon.  Amazon first attempts to recognize the product automatically, and if successful a response is sent back to the phone in literally seconds.  If Amazon’s automatic image processing doesn’t yield a it, hit gets outsourced to real human beings via Amazon’s Mechanical Turk and you’ll get product info in short order.

A more recent attempt at this approach is Google Goggles.  Goggles is intended to be a general “search by image” application and supports much more than product lookup.  Though somewhat limited in its current incarnation, it gives you a good glimpse of the future, where you really don’t need a barcode – just point your phone at any object (people included?) and bingo you have everything you want to know about it.

5. Information aggregation. One very common theme is that a lot of applications in this space are not provided by the brand or merchants themselves, but by a third party that is aggregating data from a number of sources.  The net effect is that the prospective customer is essentially armed with a real-time consumer guide in their pocket at all times.   If a merchant doesn’t give the customer the best deal possible, the app will tell them where they can get it, just down the road at a nearby competitor’s establishment or online.  Good examples of these sorts of apps are Shop Savvy (for retail) and UrbanSpoon and Yelp (for hospitality).  If these

"Amazon Remembers" utilizes a combination of automagic and human powered search to match a product to a user generated image.

sorts of apps aren’t already getting the retailers attention they soon will be as more and more people begin to upgrade to application capable mobile phones.  A good strategy for a retailer (beyond offering the lowest price and being as geographically ubiquitous as possible) is perhaps to offer a branded mobile experience of its own that provides a better experience than the third party aggregate apps, and possibly integrates more tightly with in-store experiences (indoor positioning, situated displays, POS, etc).

These are the best exemplars of these retail/hospitatlity mobile features that we’re aware of at the moment.  If you are aware of better examples, please do let us know.  We’re also interested in hearing from you if you can point us to mobile apps in the retail/hospitality space that incorporate features beyond the obvious, and which do not fit in one of the feature descriptions that in our list above.

Can't decide between the multiplicity of Thai choices in Allendale, MI? Give UrbanSpoon a shake and it will make the choice for you.

When I recently returned to my roots here at GVSU, I not only had to an assimilate an amazingly altered campus from what I left behind back in 1988, but I also had to get accustomed to the fact that GVSU now sprawls across the entire W. Michigan area. In addition to the main Allendale campus there is the gorgeous Pew Campus in downtown Grand Rapids, and a second downtown campus on “medical hill”, as well as campuses in Holland, and Muskegon.

Here on the main GVSU campus a project is underway to build a new library facility. This week a fly through of the new facility was published on YouTube. It is going to be a remarkable facility indeed! I was very pleased to see that among with all of the new ultra modern library technologies (including an Apple-like “Genius Bar”), the new library will still have a “Collections Wing” with a “Browsable Books” area where its patrons can wander among shelves of very real physical books and browse for good reads. Being the gadget geek I am, you all might be surprised to learn I don’t yet own a Kindle or equivalent dedicated reader device. I don’t know exactly what it is, but for me there is nothing digital that replaces the authentic experience of reading with a real dog eared tome in hand. Don’t get me wrong, I love Google books, and a Kindle is probably a much more practical form factor if you need to lug the equivalent of a box of books with you on a trip somewhere. However, to truly experience a good read, I still prefer the old fashioned media.

I’ve embedded the youtube video above.  You can read more about the new Mary Idema Pew Library Learning and Information Commons here.

Twitter now let's you organize your favorite tweeters into lists. A very handy way to add some order to the tweet chaos.

The new “list” feature on Twitter is fantastic!  Basically, twitter follow lists lets you the user organize your twitter friends into meaningful lists to share with others.  This morning I started organizing various lists relating to mobile technology that I thought would be of interest to others.  So far, I’ve put together the following lists:

• Mobile Manufacturers:  Every manufacturer of consumer mobile devices I am aware of that has an official and active twitter presence.  This includes their main corporate twitter accounts as well as specialized twitter accounts (e.g. mobile products business units, developer communities, etc.)
• Mobile Operators: Every mobile operator/service provider I am aware of that has an active twitter account.  At the moment this list has a North American bias to it.  Eventually, I’ll get around to creating similar lists for other markets.
• Mobile Techies & Bloggers:  This is perhaps the most interesting list – everybody out there that I’ve encountered so far on twitter whom I think has something interesting to say about mobile.  I tried to be discerning here as there are a lot of people who are tweeting about mobile.  I didn’t not limit myself to folks in the USA, but I did try to limit it to those who tweet primarily in the English language. I’ll keep tweaking this as we go.  I’m sure I’ve overlooked a few obvious folks here… sorry in advance. 

This is still very much work in progress, but if you’re a mobile junkie, eavesdropping on these lists should help you very quickly tune into the tweets that matter.   I’ll be adding some addition mobile-related categories in the future.  If you find this useful, a  retweet would be greatly appreciated!

Google Latitude is available on your iPhone via Safari (e.g. a web app) and not as a native app. The browser requests the user for permission to access the phone's location API.

This morning I came across this interesting and well-written article on the current state of affairs with regard to mobile app stores.   I think the author’s forward looking comments are reasonable if you limit your outlook to a year or two,  but there are a couple of things to consider with regard to app stores in the long term.  First,  the common presupposition made is that app stores will remain a relevant if not primary way users obtain apps for their phone.   From a purely technical perspective  I would suggest we should not ignore the possibility that rich Internet apps eventually begin to displace the need for native app installs, and eventually the need for mobile app stores.  As HTML5 continues to evolve and the underlying mobile platform becomes more accessible to web app developers, what advantage is there in downloading an app, especially when the data seems to suggest that mobile app retention rates are surprising low (25% on average!)?  It seems to me that web-based apps make perfect sense for a very large number of mobile apps that today are typically one-shots from the end user perspective.  Here at MASL our students have already built some very clever iPhone apps entirely in Safari.  Once a shortcut is added to the home screen it is indeed very difficult for the average user to discern whether or not its a native app or a web app. (We intend to do the equivalent of a “Pepsi Challenge” to actually measure this – look for more details in the future.)    Google Latitude on the iPhone is an early example of using the browser as an alternative to iTunes App Store.

That brings us to our second  point.   From a commercial perspective, Google is shaping up to becoming a pivotal stakeholder in the future mobile application ecosystem.   As I pointed out a while back, its raining Androids this Fall.  (For a much more up-to-date and complete listing of confirmed and rumored Android devices see the list that TechCrunch recently compiled.)  Every Android device I’ve used to-date requires you to become a Google citizen (you need to authenticate with a Google account or create one if you don’t have one) before you can use the device, and from that point on your mobile device is very tightly integrated with the Google cloud, and all the data and services you know and love (email, calendar, maps, search, etc.) are accessible from your mobile without the user installing anything.  It makes perfect sense for Google to gradually nudge the mobile application ecosystem towards a model in which the browser is the primary vehicle for third party application providers to access and integrate with Google services and data in the cloud. This also makes a lot of sense for the end user given the current app usage patterns and in the end seems to raise some interesting questions about the long-term viability of the mobile app store model.

I saw a couple of interesting videos this week that are worth taking a look at. The first one is about testing mobile application. So if you have an app that targets a multiplicity of handsets, all with different form factors, technical specs, and service providers, how can you go about testing your apps? Its simply not feasible to acquire all the phones you need to test on, and even if you could you’d still have to worry about setting up service plans, etc. Worse yet, perhaps you’re targeting a non-local market and want to test your app abroad on a particular carrier, etc.

The MOTODEV guys recently interviewed David Marsyla of DeviceAnywhere.com. David’s company offers the “ultimate mobile application testing platform”. Basically they have a network of over 2000 shared devices world-wide that they make accessible to developers via remote access. So in theory you could test your app on any device, on networks anywhere in the world without leaving your desk.

The second video was posted by Engadget and involves stress testing the actual mobile phones themselves. The video is footage they took in Nokia’s product test lab in California.  You’ll see them inflicting all kinds of serious misuse on the devices using an interesting array of robotic technology and other test apparatuses in an attempt to simulate the sorts of misuse you all inflict on your phones over the course of time. Makes for an interesting video.

The full engadget posting is here.

Today its not uncommon to run into people who think of human-like robots when you start talking about Android, rather than Google’s cool new mobile phone platform. This is likely to change dramatically between now and Christmas, at least here in the USA. In the past month a number of manufacturers and carriers have announced new Android-powered phones, and there are more to come in the days ahead.

Available today at a store near you …

At present, if you reside in the USA (we’ll have to write the international version of this post later…) your Android options are far and few between. Today you in fact have only three choices:

• HTC Dream (G1): Actually the first Android phone that became available (October 2008). It has a slide out QWERTY keypad and capacitive touchscreen. Its available on T-Mobile.
• HTC myTouch3G (Magic): This device became available via T-Mobile this past summer. No QWERTY keypad on this phone.
• HTC Hero (G2): This phone looks a bit like the G1, only minus the G1′s controversial “chin”. As of this past weekend, it has become available via Sprint Nextel. (October 11).

The other options you have here in the USA is to visit ebay or opt for the unlocked developer phone made available by Google (a G1 I believe…)

Coming soon …

If you can wait a few weeks more, the number of options available to you here in the USA will increase substantially:

• Motorola CLIQ (aka as DEXT outside the US): The US launch with T-Mobile is supposedly October 19. Its actually the same processor as the original HTC G1, but comes tightly integrated with your favorite social destinations via a layer of software added on top of Android by Motorola (MOTO BLUR).
• Motorola Sholes/Tao: According to latest rumors, this device will launch with Verizon as soon as October 30, but others indicate an early December launch.
• HTC Hero/Desire?? In addition to the Motorola android phone, Verizon is to launch a second Android handset from HTC this fall.
• Samsung Moment (InstinctQ): Originally announced by Samsung last Spring, Sprint Nextel has announced a launch date of November 1.

Coming not so soon …

There are also some more options that are further out that you’ll want to pay attention to if none of the above suit your fancy:

• Dellroid: a Dell manufactured Android smartphone to be available via ATT?
• LG GW620: officially announced by LG, this phone will become available sometime in 2010.

In addition to these two, you can be sure that the usual culprits (HTC, Motorola, Samsung) will be shipping a whole raft of new Android phones in 2010. In fact, last week Gartner forecasted that by 2012 Android will be the #2 smartphone, bested only by Symbian.  We’ll need a lot more devices out there than there are today in order for that to happen!

Do let me know if I’ve missed any notables in the first two categories above.

This semester I’m teaching an undergraduate CS course that has a weekly two hour lab session. Instead of going the traditional route of making up 4-5 programming assignments and assigning them over the course of the semester, I wanted to try and create a set of weekly labs that encourage the students to tinker and explore the concepts more deeply in an experimental sort of way. That is, I wanted to create a CS lab experience that was sort of like what one experiences in a biology course – start with a hypothesis, do some experimentation/observations and come to valid conclusions.

I initially thought about creating a document for each lab session that spelled out in cookbook style the various “coding” experiments, but the tedium tremors set in just thinking about what a massive effort that would be. What I really wanted is something that met the following criteria:

• Minimize “production” time: that is, once the basic ideation for a given lab session was complete (e.g. lab objectives identified, and the code “specimens” written/tested) I wanted to be able to produce the final lab “media” in 2 hours or less.
• Optimize the student experience: I wanted to use something that my students would find interesting and effective, and something they could refer back to after the lab session to review their work.  A thick written “lab manual” with lots of gory details would be dead on arrival.
• Easy to distribute:  just provide a single link and the students find everything from there, no matter where they are, or what kind of computer they like to use (its a real mixed bag – some kids like Linux, others OS X, and  some have Windows).

In the end, I installed ScreenFlow on my MacBook Pro and use it to create screencasts.   I provide the students with a lab assignment in the form of a PDF document that describes the lab exercise. However, instead of step by step directions written out long hand, I simply refer to a YouTube link of the appropriate screencast. The students are then asked to respond to questions about the screencast and are given the opportunity to modify/rerun the code and observe the results.

So far this seems to be working out quite well.  The students bring their earbuds to the lab sessions and most of the responses so far have been positive.  This is a format that they are already very accustomed to, and the 10 minute length limit in YouTube is a convenient goad in helping me keep each segment focused and to the point. We’re starting to see hits on the videos from various parts of the world along with some comments/ratings, so hopefully others can benefit from this as well!

Production time varies, but I’m definitely in the ballpark. Screenflow works very well for creating the screencasts, and the learning curve was fairly flat. Here’s a sample screencast introducing students to network programming in Java:

I’m linking to my complete set screencasts on my Screencast Tutorial page. In addition to the videos links, I’ve also included download links for the source code used in the screencasts.

I’m interested in comparing notes with others who have used this approach in teaching (CS or areas as well). In particular, let me know what tooling you have found useful for creating/editing screencasts and any other lessons learned.