My experience with gestural interfaces
Touch gestures level 2004
The journey with touch gestures for the general public most likely began around 2009, when the iPhone gained popularity and other companies released similar products.
“Nobody wants a stylus”
Yes, I am talking about that time: The time of the stylus. A device, Steve Jobs, CEO of Apple at that time once wanted to put on the graveyard.
Apple however was the same company, who just years later introduced the Apple Pencil.
There was a time, where styluses were the key to control the screen of our devices. And there were companies who mastered this “limitation” in a remarkable way, even back then.
One such company was Palm, which was a well-known brand for pocket PCs at the time. In 2005, I had to make a tough decision: whether to get a Nintendo DS or a Palm Tungsten E2? The Tungsten E2 was an extendable and multipurpose organizer, while the DS was a limited game console with a small touch screen.
I chose the Palm and never regretted it. Why? Because it gave me the capabilities of a smartphone, such as a calendar, contacts, music and video player, before smartphones were widely available.
How to type on such a small screen?
Well nowadays we are blessed with huge screen smartphone so literally every person is made able to use touch input and won’t likely miss the keys.
Palm however back then achieved a reliable text input, without the need of a huge screen.
The key? Gestures!
People were primarily coming from physical notepads back then and while on screen keyboards are hard to achieve for two reasons:
- Small screen size
- Jumping between characters requires advanced eye-hand coordination
However back then it was still too early to implement advanced technology such as hand writing detection. But Palm had its own secret sauce:
“Let’s just create our own alphabet. “
With the gesture based alphabet Palm created, it was easy to write text fast. Why?
- They kept it simple with small chance of variation from user to user. Overall, the detection was very reliable
- You never had to leave the input pad - it was literally possible to “blind type” with gestures
- The input pad followed a clear structure: letters on the left, numbers on the right.
Graffiti never dies
As it turns out, the idea of the PalmOS graffiti still exists today in 2020 and is available on Android.
Nowadays however, I am actively using the swipe-functions on major keyboards (available on iOS / Android), as it allows blind-typing regardless of the phone's screen size.
Going beyond touch gestures
There was a time, where my flat mates and I became obsessed with smart home devices. And overall I must admit, that I am still fascinated by it - but rather not about the input/output part. Why would you replace a physical light switch with a fixed wireless button using a battery that would do the exact same thing - trigger the light?
For me, one of the reasons I would do so, came in the enhanced possibilities in interaction, it could have over the regular light switch. One product that really caught my interest was the Mi cube from Xiaomi.
This cube is basically a motion and level-sensor being able to detect its
- Position
- Movement
- Turns ((semi)-clockwise)
- The current side
With the amount of conditions this device gave me, it allowed me way more interaction opportunities than the average light switch.
bottom line, I defined two main lighting conditions
- Move the cube: Toggle the light
- Flip the cube 90°/180°: Ambient mode / Productive mode
- Turn the cube semi-clockwise / clockwise: Increase/decrease brightness
The reason why I keep on using it in different situations and consider it over voice assistants, which whom I could also control my light: I can control it not only with my hands but even with my feet - for instance when laying on the couch and I casually want to turn off the light or switch to another light setting.
See here a small demo I created to demonstrate the modes the cube features.
Do you consider the interaction “natural”? Explain why.
In the last section, I mentioned two examples. Both of which I would describe them as “natural, once you learned how to use them”.
I would propagate that telling whether a product is natural or not largely depends on what the prerequisites are.
Did the user ever use a computer before? A touch screen? Does the product rely on known patterns?
I think that a real natural product needs to follow law of nature - rules that people are aware of and expect - regardless of their level of knowledge and technological skills.
In both of my examples however, although the interaction follows basic gestural rules , users still need to learn the interaction before they will be able to leverage it for their needs.
Technological side
PalmOS "Graffiti
The hardware side is quite simple - a resistive touch panel, that reacts based on pressure and measures where the pressure occured. These where quite common back then and in contrast to the now more established capacitive touchscreens, there were no such perks as multi-touch support. However, this type of screen is still the preferred choice today in some industrial appliances, for instance for special machines that are usually operated with gloves.
But well, the hardware (the resistive touch panel with positioning matrix) collects data about the strokes being made, while the software constantly matches the input with known patterns.
After conducting some research, i realized that the actual technology originated from the Unistrokes Technology, initially introduced by Xerox and which according to provided performance testing enabled users with a higher "wpm" - "words per minute", which might be an indicator for how memorable and "natural" to learn different technologies had been to users.
Palm learned from the fact, words per minute were subpar in comparison to Unistroke, so they decided to launch a new revision of "Graffiti", that was designed to make letters and numbers appear closer to actual handwriting to make it more natural, so the learning curve would be easier to achieve for new users.
Xiaomi Aqara cube
The device itself is plain simple, the Mi cube consists of:
- Motion sensor (detect motion)
- Battery
- Zigbee chip (Send detected motion to Zigbee hub)
It can connect to every kind of Zigbee hub, which is a common network protocol for smart devices.
By default, this would be a Xiaomi Gateway. This gateway however sends data through Chinese servers, which I consider unnecessary - Just as the Philips Hue Hub would send data through Philips servers in The Netherlands and if you connect Alexa to it, through Ireland as well.
But after all, these hubs are just containing Zigbee chips as well, so thanks to an ambicious community that developed around the Raspberry Pi scene, nowadays you can eaily get a 5$ Zigbee chip, attach it to your Raspberry Pi and connect all the smart devices to one device - without connecting to any third-party servers.
If you are interested on how to integrate this little magic cube in your home, see here:
My Question 1: What seems more natural to you? Entering the room at 10pm, light turns on automatically or Entering the room, you toggle the light switch as usual?
My question 2: What is the most natural input source to you? Hand writing, Keyboard, Swipe keyboard?
My Question 3: Will we ever be able to create true natural interfaces, where people require no onboarding at all? How are they going to look, sound, smell like?
Why this question? I was thinking about the most edgy edge case e.g. how would aliens interact with the CD, that got sent to space in 1977?
Some articles suggest that voice interaction might be the most natural way of interaction - but what about crowded places, where speaking to a voice assistant is considered awkward? Is there a fallback for deaf people?
I think that in any case, the time between action and reaction should be as low as possible to encourage fast learning and thus support the phenomena of instant expertise
Conclusion
In conclusion, touch gestures have come a long way since the time of the stylus, and there are now many other types of gestures available for interacting with devices. As technology continues to advance, we can expect to see even more innovative ways of interacting with our devices in the future.conclusion, touch gestures have come a long way since the time of the stylus, and there are now many other types of gestures available for interacting with devices. As technology continues to advance, we can expect to see even more innovative ways of interacting withconclusion, touch gestures have come a long way since the time of the stylus,
More content? Check out the entire blog 😎