The Value of Magic

Arthur C. Clarke famously stated "any sufficiently advanced technology is indistinguishable from magic." It's easy for me to get caught up in the practicalities of the business of software—aligning development resources with strategic plans, executing against a road map, leveraging relevant frameworks, etc. But even perfectly executed software can fall flat and fail to matter. Perfectly executed or haphazardly hacked, the software that really makes an impact is the one that delivers magic.

I remember the first time I got exposed to chromatography data systems. The thing that impressed me most wasn't the instrument control aspects of the system, but rather the way the data from the detector snaked onto the screen and turned into peaks, and magically the system determined where the peaks started and stopped, drew their baselines, and calculated the area under the curve. Even though the method to do so was relatively straightforward, there was an essence of magic in it. It was non-obvious to me.

There is a science behind cool. There are cognitive underpinnings to what makes us react the way I did the first time I saw a chromatogram being drawn in real time. It goes beyond what is aesthetically pleasing or practically functional—it has something to do with the non-obvious nature of it. I get the same reaction when I see an unexpected conclusion emerge from a mass of information, the aha moment you so desperately seek from the promise of big data.

I see popular software and services criticized for not being novel, useful or monetizeable, and yet these services generate a lot of attention. Some of them are successful because they act on the deeply human need for interaction and personal engagement. They facilitate a conversation, perhaps with complete strangers, but with people reacting to what you have to say or share. The closer to real time, the more personal and authentic this interaction, the more it delights us. Some are successful because they facilitate or automate a workflow and let you perform tasks more efficiently than before. But the examples that fascinate me most are the ones that do the unexpected. Maybe they let you apply cool filters to your photos that transform them in a way you didn't anticipate, or they respond to your voice commands with an uncanny accuracy. Visually, through analytics, or in the way data is combined and correlated to achieve an unexpected conclusion, they deliver a magical experience that didn't occur to you a priori.

Over the years, in the many and varied systems I've worked on, there has always been at least one aspect of the system that seemed magical. I admit now that I haven't always credited those magical elements with the value they deserve. Magic can be hard to achieve, and investing in it can be hard to justify. But it can also represent a barrier of entry to your competitors and a delight for your users and customers—it is your purple cow. Another Arther C. Clarke tenet is "the only way of discovering the limits of the possible is to venture a little way past them into the impossible." As technologists, we have the opportunity to build something that ventures into the (seemingly) impossible. Whether or not you consider it a key component of your value proposition, be cognizant of the magic in the systems you build, and know it can be the one thing that makes all the difference in the world.

Constraints are the mother of invention

I suppose the title of this post is rhetorical, because there is no aspect of our existence that is truly unconstrained. But even in a more narrow application of the term constraint to business and technology, innovation feeds more from a limited environment--often severely limited--than one where resources are abundant, knowledge is pre-existent, and capabilities are easily devised. We often blame our inability to innovate on lack of resources, lack of experience, and unforeseen challenges of execution, but these very conditions produce our most profound inventions. And for me personally, it's these conditions that bring out my best. Granted, I may kick and scream during the process, but if I can muster the courage to rise to the occasion and persevere, in retrospect they are the most fulfilling of times.

You may have read the Inc. article about the best definition of an entrepreneur. Having worked with a number of entrepreneurs, the definition rings true with me: "Entrepreneurship is the pursuit of opportunity without regard to resources currently controlled." Over the course of my career, I frequently find myself acquiring, applying, and "controlling" resources to achieve the goals of the entrepreneur's pursuit. In the most constrained environments, we must gain knowledge, devise capabilities, and figure out a way to succeed while the sands of our investors' money slip through the hourglass.  It's this environment where we learn how to experiment, take chances, and challenge our assumptions.  Sometimes this leads to significant breakthroughs.

Looking beyond the world of technology and business, constraint appears to be a necessary ingredient in human creativity. We impose constraints as a way to foster creativity, a way to express rich emotion and feelings with a subset of the full language we have at our disposal.  Traditional Blues has the AAB lyric constraint and three chords played over a 12 bar scale. Poetic verse is often constrained to a specific meter, form or structure.  Haiku is limited to the 5 - 7 - 5.  Maybe we impose these constraints to challenge our imagination and express something new while fixing certain fundamental variables.

An article in Psychology Today states that you are more creative if you don't allow your mind to roam free. That's counter-intuitive to say the least, but research indicates this to be the case. After all, Science Fiction writers describe alien life forms as similarly humanoid with remarkable consistency. The article mentions work presented in the book Creativity from Constraints: The Psychology of Breakthrough, suggesting that certain types of constraints promote novel thinking and innovation.  Without these constraints, we tend to fall back to past examples, and the variability of our creations are limited.  Constraints can rule out specific solutions that have worked before, and foster novel approaches.  

This ties back to business, technology and innovation.  Without a constrained environment, in a mythical world where all possible solutions were available, you will tend to pick an approach with which you're most comfortable or familiar. While this could result in success (assuming the same conditions for success exist with the new and previous situation), it will not result in a breakthrough. I've seen entrepreneurs fail because they apply the same patterns they used successfully in the past to a new challenge. The presence of constraints is no guarantee for creativity, and creativity is no guarantee for success, but those who support and invest in entrepreneurs often use creativity as an indicator of potential success.  They chastise those whose ideas are overly-constrained without realizing that constraint is at the core of creativity.  Alas, there are different kinds of constraints.

As innovators, we must learn to embrace constraints.  We must be cognizant of the constrained environment in which we operate and explore the full range of possibilities within that field of play. Sometimes you only have 17 syllables to express the wonder of a sunset or the crisp air of a winter morning. Sometimes the limitations you feel are crushing your creativity and suppressing your best work are in fact providing you the opportunity of a lifetime.

Old ideas about interactive, distributed apps reimagined in a mobile context

Over ten years ago, my colleagues and I at Kenamea had some ideas about lightweight desktop apps and event-driven communication models that would allow them to interact with other apps over the internet. Here are some of the considerations that shaped our thinking.

1. HTML and javascript are pretty powerful, and developers who know how to build user interfaces in HTML and manipulate markup using a browser-based scripting language are more plentiful than developers who know how to write server-side code to implement business logic. We wanted to leverage the growing pool of front-end web developers who might be more accessible to the marketing and business functions within organizations funding web development. 
2. To enable truly interactive applications that communicate with other instances of themselves and other applications/services on the internet, the request/response model of the web is insufficient.  We need a reliable messaging infrastructure that provides event-driven communication (i.e. apps that can send and receive messages asynchronously), but do so in a way compatible with the protocols and scale of the web.
3. One reason web apps succeed is because they are more-or-less client-agnostic (browser compatibility not withstanding), and issues about version control and updates are virtually non-existent.
4. Web apps often use a lot of bandwidth re-delivering UI elements that don't change from page to page.  I'm not talking about graphical assets that get cached by the browser, I'm talking about web servers resending page markup containing new data when really only the data has changed.  A more efficient model for interactive apps is to only send new data to the client, and let client-side logic implement how that data is presented in the UI.
5. To be really useful, networked applications should continue to work, albeit in a limited capacity, when there is no network connection.  Web applications typically fail in this regard.
6. Particularly in conjunction with the above, and to enable store-and-forward messaging, applications need to have non-trivial local persistence.  An application-scoped local store of key/value pairs can go a long way to building really interesting client-side apps.

Back in the day, we addressed these considerations by building internet-scale messaging infrastructure (to accomodate issues 2, 4 and 5 above), along with a client-side app environment that allowed developers to build interesting apps in HTML+javascript, with access to services that address issues 1, 3, 5 and 6.  Namely, these services included:

1. A platform-specific framework that runs applications built using HTML and javascript.  Apps have access to javascript objects and events that provide bi-directional messaging services (e.g. send a message to an individual app instance, send a message to a topic to which all app instances listen, receive a message from an app or topic).
2. A facility for installing, registering, and updating apps.  Once installed, the app instance is registered with the messaging platform so it can send and receive both point-to-point and topic-based messages.  App updates are delivered as system messages, and the client-side framework manages the installation and data migration seamlessly.
3. App-scoped local store of key/value pairs, accessible via javascript.
4. Offline functionality—apps within this framework work offline since the HTML, javascript, and other assets associated with the app are stored locally, and the local store is always accessible.  Outbound messages are queued for delivery when connectivity is restored, and inbound messages are queued in the cloud until the app reconnects.

Today, I read a lot of articles that state how HTML5 is revolutionizing mobile app development, because it addresses several of the same concerns we had back in the day.  With the promotion of media objects as first class citizens in the markup, and the corresponding events and methods accessible via javascript, you can build some pretty interesting web apps without using platform-specific plugins.  Also, HTML5 gives you real local storage.  As HTML5 becomes broadly supported in mobile environments, I expect to see more sophisticated mobile apps using these features to implement functionality that was formerly implemented using platform-specific compiled code.  

I've seen some mobile app frameworks that allow you to build your app UI and business logic in HTML5 and javascript, then wrap that in a platform-specific (i.e. iOS and Android) container so it behaves like a native mobile app.  This affords the features associated with native apps, such as presence in the app store, stickiness on the mobile desktop, and offline functionality.  It removes the cross-platform burden from the app developer since the UI and logic are implemented in a cross-platform language.  In many ways, this combination of a platform-specific container combined with platform-independent UI and logic is very similar to the framework we built prior to the advent of mobile apps.

I think there's still plenty of room for innovation in this area.  Many of the concerns we had in 1999 about interactive, distributed applications are even more valid in the world of connected mobile devices. There is still a need for services such as:

• an easily accessible, internet scalable, bandwidth-friendly messaging infrastructure, 
• a robust client-side app framework that provides store-and-forward messaging, app version management, and offline functionality, 
• a programming model that promotes bi-directional data exchange between applications and client-side control of presentation (vs. page-oriented request/response).

I can't help but think about how the lessons we learned building a desktop-oriented app framework back then could be applied to the eerily similar mobile app environments of today.