Some lessons from an open source project that never gained critical mass

I recently announced that I was halting development on QuickUI, and thought it would be a good idea to do a little post-mortem before moving on.

The good side: As a framework for web application UI development, QuickUI measured up to its design goals. It was useful for creating complex application UIs with a good separation of concerns between UI components. The core framework was highly reliable with a fairly tight abstraction and acceptable performance. QuickUI was used in some companies for real production apps with real users. The very small number of developers who actually used the framework said they liked it and were impressed by what it could do.

The bad side: As an open source project, QuickUI never achieved critical mass. QuickUI was my first attempt at kickstarting an open source project to help establish an ecosystem for component-based web user interfaces. I’d initially held the naive view that simply publishing something as open source (which I did in late 2009, two years into QuickUI’s development) would, on its own, generate community interest and participation. But the universe of open source development projects is vast, and simply making something free doesn’t make it popular. Free products still need to appeal to their audience and gain adoption through a good feature set, great distribution, and substantial luck.

Eventually, QuickUI was overtaken by open web component standards (which is a good thing), and it no longer made sense to continue investment in QuickUI. Before moving on, I wanted to write down some of the lessons the QuickUI project held for me.

  1. People care about the stack of technologies beneath your project. I used to think that all that mattered is whether a tool worked and was easy to use, but most developers make demands the construction of the tool itself: programming language, runtime platform, dependent libraries, etc. The most common reason someone will offer for this is that they may need to diagnose a bug in the code and fix it. That may also be true, but I think most people have less glorified reasons for considering the technology behind a tool.

    First, simply understanding the vocabulary associated with a given stack requires investment. Occasionally you come across a tool on GitHub whose Readme says the tool can be configured via a .foo file, where the location and syntax of a .foo file is completely obvious to current users of that tool’s stack. If you don’t already know what a .foo file is, you’re less likely to adopt that tool.

    Second, a developer toolchain is a creaky, cantankerous beast, and incorporating multiple stacks of tech is a pain. This is less about debugging than what Bruce Sterling refers to as the wrangling required to get things to work together.

    Third, you trust tools on stacks that have worked for you in the past, and are skeptical that tools on unfamiliar stacks will actually work as advertised. For example, I’ve hardly ever used Ruby, and rarely use Ruby gems. When I see a tool published via npm, I feel comfortable installing and using it, even if I never look at its source, because I’ve used many tools that way. If I see a tool that does the exact same thing published as a Ruby gem, I view its efficacy as black magic.

    Last, people implicitly trust other people who have selected the same technology stack. “If someone has made the same technology choices I have, they must be as enlightened as I am!” Conversely, people look askance at those who make different choices. A die-hard plain JavaScript coder may view the Clojure community are pot-smoking hippies, while the Clojure developer may view the original party as a latter-day COBOL dork lacking sufficient brainpower and awareness.

    In the case of QuickUI, when I started on it in late 2007 or so, I was most comfortable writing in C#, so when I needed to write a build-time compiler for UI markup, I wrote it in C#. Bad idea. Despite the fact that there was no run-time need for .NET, and that the compiler ran perfectly well on a Mac (under Mono), no one would take it seriously with the taint of .NET on it. I was eventually able drop the compiler altogether, and rely on a stack of tools which jQuery developers were already familiar, but that cost the project time and effort.

  2. Using anything other than native web technologies — plain HTML, plain JavaScript, and plain CSS — significantly constrains your audience. This is similar to the above point, but pertains to your project’s source code rather than the stack of technologies supporting that source. Again, part of the argument here is that they might theoretically need to dive into the runtime source to diagnose a problem, but I think the reaction is usually more instinctive than that.

    At one point in the development of QuickUI I discovered that working in CoffeeScript was much more productive for me than in plain JavaScript. In short order, I ported the QuickUI runtime to CoffeeScript. Switching to CoffeeScript was a huge productivity boost for me — but represented a huge reduction on the potential audience for QuickUI.

    For one thing, it seemed nearly impossible for me to get potential developers to ignore the presence of CoffeeScript. CoffeeScript compiles to plain JavaScript, but anyone who looked at the QuickUI source repo saw “CoffeeScript” listed as the primary language… and walked away. They might say, “Your project looks interesting, but I don’t know CoffeeScript.” I’d tell them they didn’t need to know CoffeeScript — that was an implementation detail — but they’d already decided they were uninterested.

    Over and over, the general feeling was, “I just don’t want to use something that uses something I don’t already know.” And then, of course, for most people who really did want to be able to grok the source, CoffeeScript was a non-starter.

    To build a community around a project, you probably want as many people as possible to be able to participate. Even if you hate JavaScript, it’s the web’s lingua franca. Using any other language for an open web project may not be an insurmountable obstacle, but at the very least it’s a significant handicap. As much as it pained me, when I started up the subsequent Quetzal web components experiments, I did so in plain JavaScript.

  3. Everyone will insist there is one thing you must do that will make your library more acceptable — and all those things are different, and all those things together are probably still insufficient. Over the course of the past few years, I have given many, many QuickUI demos, and received feedback on all aspects of the tool, the accompanying site at quickui.org, the learning process, etc. Very little of this feedback was consistent; everyone fixated on something different. I thought, if I just respond to all the feedback, surely at some point the barrier to adoption will be low enough that people will start adopting the tool.

    On the basis of such feedback, I spent a huge amount of time improving things. Early on, someone said I should move the source to GitHub, so I did. Someone said it needed better documentation, so I wrote a lot of documentation. As discussed above, numerous people suggested moving away from .NET, so I did. Someone suggested having live examples of UI components, so I built those. Someone thought a tutorial would be helpful, so I made an interactive tutorial. (Which many people complimented. Thank you!) One person’s key complaint with the entire framework was that the home page didn’t have icons; if the framework was to be successful, the home page needed little icons to indicate which browsers were supported by the framework. I did that. I received many, many other suggestions, and I incorporated almost all of them.

    None of it mattered. The feedback was actionable, and probably much of it was accurate, but even addressing (nearly) all of it wasn’t enough to make the project successful.

    The fact is that most people are unwilling to invest the time to understand, analyze, and articulate what’s really wrong with your project. Of the people who gave me feedback, few of them actually looked hard at it, very few had actually tried it beyond the online tutorial, and very, very few were willing to speak directly to their key concerns. This is all understandable — people are busy, spending time on a framework of unknown value is potentially wasted, and most people want to be nice to you — but feedback on a project should be accepted with these phenomena in mind.

  4. Paradigm shift is prohibitively expensive. Early on, one person told me that building UI in a component-oriented fashion represented a significant paradigm shift — and that represented a potentially insurmountable obstacle for QuickUI.

    The problem with a paradigm shift is that it’s hard to even have a conversation with someone whose conceptualization of the world doesn’t even allow them to recognize the problem they have. I spoke with many developers who viewed the undifferentiated pile of JavaScript generating their UI as the way things had to be done. They were more concerned with getting their UI to work across multiple browsers than to worry about componentization — even though a good component library is exactly the sort of thing that would have made cross-browser work more manageable. (The cross-browser hacks could have been folded into the components, allowing them to work at a higher level of abstraction, etc.) Interestingly, these same developers would carefully factor their code into classes or functions with clear lines of responsibility. They applied good factoring to every other thing they coded except their web UI.

    Over the past year or so, Google and others have been evangelizing a component-oriented paradigm for web development. At some point, we’ll go through some phase-change where that paradigm will suddenly become dominant. I’m betting that change will happen before late 2015. By then, it will be hard to find a good web UI developer who doesn’t think of their UI in terms of components.

  5. People are only interested in something if others are already using it. This is true for both seasoned developers and novice developers — but for different reasons. The novice teaches themself jQuery or Backbone or LESS because experienced people use those things, which means they’re probably interesting and useful. The seasoned developer picks up a new tool that others are using because the alternative — using something with a tiny user base — represents unacceptable risk.

    This last point was made to me by the most abrasive person I ever spoke with about QuickUI. In fact, I think it’s because they were untroubled by politeness that they could speak the truth. They said: “I would love to use this, but I can’t. If something were to happen to you, I would be stuck having to fix your bugs. I made a bet like this in the past, and was stuck supporting someone else’s framework. I won’t do that again. Come back when lots of other people are using this.”

    So this developer had, at some point, found a great piece of technology, developed by someone else, and they staked their own reputation on adopting this technology, only to have it completely fall apart when the other party went away (went bankrupt, was acquired, whatever). It’s probably safe to assume that most seasoned developers have had a similar experience. For every open library, there must exist some critical mass at which the library’s community becomes self-sustaining.

    At that point, if you find a bug in the library, someone else in the community has probably also found the bug, and maybe even fixed it. Enough other people are invested in the library that, even if the original developer disappears, the remaining investors will keep it going for as long as that makes technological sense.

    I’d be very interested if someone could pinpoint the size of that critical mass. I’m guessing the number is pretty small: perhaps 10 active contributors might be sufficient to create the perception the library is well-maintained and not going away. Whatever that size is, I couldn’t grow QuickUI to that size.

  6. Google could publish a JavaScript library for cloud-based ham sandwiches and a thousand people would immediately star the repo on GitHub. For much of my career, I focused my attention almost entirely on the value a product created, and gave very little thought to how a product would be distributed. Distribution is, in fact, at least as important, and maybe more important, than underlying value. That’s certainly true in the short term. And, over the long term, well, maybe technology changes quickly enough that the long term never comes into play.

    I’m not saying individuals can’t launch successful open source projects, but rather that doing so within the context of a company with name recognition and a developer outreach program makes it much, much easier. The people working within such a context may not realize it. I once heard someone describe people who work at big companies as “basketball players on the moon”. Those people can jump very high, but they may not realize the extent to which their performance depends on that context.

    I have deep respect for a person who can launch something entirely on their own, without relying on the backing of their company (or industry name recognition predicated on work they launched previously at some earlier company). If someone can make a disruptive technology successful entirely on its own merits, both that tech and that person are impressive indeed.

Ah, well, live and learn. I’m still looking forward to watching the web’s UI component ecosystem take shape, because we’ll all get to make awesome stuff together.

Ending active development of QuickUI

I’ve decided to end further investment in the development of the QuickUI web user interface framework.

I’ve spent the past half-year experimentally porting various QuickUI components to HTML custom elements under the rubric of a project called Quetzal. That project makes use of Google’s Polymer project, which supports the deployment of web components to mainstream browsers. While the Quetzal element collection doesn’t yet offer the complete set of QuickUI components, and working on top of Polymer has been shaky at times, Polymer is becoming good enough for real use, and the advantages of building with web standards will quickly outweigh any proprietary advantages which QuickUI could offer.

I started QuickUI a number of years ago because it seemed clear to me that a component-oriented approach to UI design and development would let you create better and more maintainable user experiences. I didn’t see anyone else working on that in an open way, and so began my own effort to make progress in that area.

The current wave of web component standards embody many of the ideas I was pursuing in QuickUI. For fun, I just dug up from my notes the earliest source code I could find for the component markup language I thought would be useful for creating web components. Here’s sample QuickUI markup I wrote at the very beginning, probably late 2007:

<control name="taskPage" arguments="name pageTip content">
  <style>
    h1 { font-face: Arial; }
  </style>
  <script>
  function foo() { }
  </script>
  <template>
    <div ...>
      <h1>%name%</h1>
      <p class="pageTip">%pageTip%</p>
      <div id="#content">
        %content%
      </div>
    </div>
  </template>
</control>

Here’s the corresponding source code in late 2013 for the same custom element in Polymer:

<polymer-element name="task-page" attributes="name pageTip">
  <script>
  function foo() { }
  </script>
  <template>
    <style>
      h1 { font-face: Arial; }
    </style>
    <div ...>
      <h1>{{name}}</h1>
      <p class="pageTip">{{pageTip}}</p>
      <div id="#content">
        <content></content>
      </div>
    </div>
  </template>
</polymer-element>

Aside from minor syntactic differences (curly braces are in vogue now instead of percent signs; “attributes” instead of “arguments”; the style tag needs to be inside the template instead of outside; the element name requires a hyphen), the source code is nearly identical. In other words, the source code I wanted to be able to write in 2007 is essentially the source code I can now write today. Hooray for the open web!

Going forward I’ll be using web components for my own work rather than QuickUI. If you have been using QuickUI or were interested in QuickUI, I would encourage you to look at adopting web components instead, either through the Polymer project or through similar projects such as Mozilla’s x-tags.

I’ll continue to keep the quickui.org site live for the indefinite future. In particular, for the time being I’ll continue to use this blog to post thoughts on developing user interfaces, with a focus on using web components.

To those of you that used QuickUI or provided feedback on it, I’d like to offer my warm thanks for all of your support.

Best,
Jan Miksovsky

Filling in base class slots with the awesome new capability of the shadow element

In a post a few months back, I discussed how custom elements needed an easy way to fill in slots defined by their base classes. The capable folks on Google’s Blink team have now fixed this problem with a small but critical Shadow DOM change, and I wanted to take a moment to walk through an example of how the feature works and explain why this is so cool.

What does “filling in a slot” mean, and why does it matter? The crux of the issue, for us to be able to share web components, I need to be able to create a new web component that defines some, but not all, of its appearance. I need be able to define certain places that you can fill in with your content. Those are the slots: points of visible user interface extensibility.

Example: a series of page templates

The post linked above offers a simple button example, but let’s walk through a more realistic example using a series of page templates. (If you have Google Chrome Canary installed, you can follow along with a live demo.) These sample templates will span a range of general-purpose to very specific. For the sake of clarity, the examples are overly simple, and are intended just to convey a sense of how such templates might be used to build a simple online store app. The same principles shown here could easily be extended to arbitrarily complex interfaces.

Page Template 1
A page-with-toolbar

We start with a completely general-purpose <page-with-toolbar> element that defines a page with a toolbar. For the sake of argument, let’s suppose there’s something fancy about the presentation here — perhaps the toolbar starts out at one size, and then, as the user scrolls, the toolbar becomes smaller and permanently docked at the top of the page. Regardless of what exactly makes this interesting, let’s assume that interesting stuff entails web ninja tricks the average designer or developer doesn’t have the time to learn or implement for themselves. Significantly, some aspect of the toolbar behavior requires real code, not just CSS styling.This user interface pattern is completely general-purpose, so perhaps this template is found in an online library of general-purpose web components. The key extensibility feature here is that the element defines two slots which consumers of the <page-with-toolbar> element can fill in: a toolbar slot at the top, and a main content slot.

Page Template 2
A store-page partially fills in a page-with-toolbar

This second image shows a <store-page> element that forms the standard base for all pages in our hypothetical online store. This element subclasses (extends) the general-purpose <page-with-toolbar> element above, so it automatically picks up the structure defined by that element and the fancy toolbar behavior. The designer of the store app can fill in some of that structure with elements that should be visible on every page in the store, such as a Home button and the story name. That is, the designer can partially fill in the slots defined by the base class: after adding the standard elements they want on every page, they can define new slots that can be filled in by instances or subclasses. The designer can also add visual styling and any behavior that should be available on every page in the app. When complete, this page both extends the above element, and is itself extensible. The element is specific to this company, but could be used by any team at the company working on their online store.

Page Template 3
A product-page partially fills a store-page

The third image shows a <product-page> element. This extends the <store-page> element above to create a template for any page specifically in the app’s Products area. It fills in the toolbar slot with buttons that are only available on product pages. It also partially fills in the main content slot with a header for the product name. Finally, it adds any styling and behavior shared only by pages in the Products area. This product page element is still extensible, but an increasing portion of its interface and behavior are constrained.

Page Template 4
A completed instance of product-page

Finally, we have an example of a filled-in <product-page> element. Presumably the app fills in the page by obtaining product data and used that to populate various elements on the page (e.g., using data binding). Now the whole page has been filled in.

By factoring our app’s design this way, we’ve created a good separation of concerns. The <page-with-toolbar> element doesn’t have to know anything about online stores. Conversely, the <store-page> element doesn’t have to know anything fancy toolbar behavior. The <product-page> designers don’t even need to know where the toolbar behavior comes from — all they need to know is that they get it for free. If the team at the company that owns the <store-page> element later decides to change the toolbar behavior, they can do so without the people who own the <product-page> element needing to do anything.

How does this work?

Until recently, it wasn’t possible for a custom element to fill in slots defined by a base class this way. An element’s template could contain a <shadow> element to ask its base class to render its stuff, but there wasn’t an easy way to pass content to the base class. Over the past couple of months, the Blink folks working on Shadow DOM features have just addressed this shortcoming with a new feature in the Shadow DOM spec and its Blink implementation. Technically speaking, the spec now allows a custom element to distribute nodes into a <shadow> insertion point and then reproject those nodes into an older shadow root.

To see an example of how this works, see the live demo. At the time of this post, you’ll need to use Google Canary. You’ll also need to open chrome://flags and check “Enable experimental Web Platform features”. But this feature will likely make its way into production Chrome in the not-so-distant future, and get polyfilled by the Polymer project for other browsers.

You can look through the code for the complete story, but the gist is this: if your element puts stuff inside a <shadow>, then that stuff will be passed to the base class as content. In our <page-with-toolbar> example element, the template contains a bit for the toolbar that looks like this:

<polymer-element name=”page-with-toolbar”>
<template>
  …
  <div id=”toolbar”>
    <content select=”.toolbar”></content>
  </div>
  …
</template>
</polymer-element>

That <content select=”.toolbar”> bit creates a slot that can be filled in by instances — or by subclasses. When our <store-page> element subclasses <page-with-toolbar>, it fills in that slot:

<polymer-element name=”store-page” extends=”page-with-toolbar”>
<template>
  <shadow>
    <span class=”toolbar”>
      <button>Home</button>
      <content select=”.toolbar”></content>
    </span>
    …
  </shadow>
</template>
</polymer-element>

Everything inside the store-page’s <shadow> will look to the page-with-toolbar base class like regular content. The page-with-toolbar’s toolbar slot will grab the span with class=”toolbar”, so the store-page’s Home button ends up in the toolbar. This lets store-page fill in the toolbar slot defined by page-with-toolbar.

So that store-page itself can be extended, it in turn redefines that toolbar slot. Any toolbar buttons defined by subclasses of store-page (like the Add to Cart and Buy Now buttons in our product-page example) will get picked up by store-page’s toolbar slot, which is nestled inside page-with-toolbar’s toolbar slot.

Conclusion

Sometimes tiny differences in fundamental features, like this refinement to the <shadow> behavior, can open up new realms of possibility. While it doesn’t look like much, when combined with subclassing, you’ve just been given a powerful new tool for creating custom web elements. You’ll be able to create a wide range of new general-purpose elements that can be filled in by other people to suit their needs, and you’ll be able to fill in general-purpose elements created by others to suit your own needs. This is a huge step towards a vibrant ecosystem of web user interface components.

Beyond the expanded capabilities enabled here, I’m personally excited to see this change because I’ve been advocating for it for so long. I made a pitch for this on a visit to Google back in May 2012, and have promoted the idea in various forums since then. Other people like Scott Miles at Google also lobbied for the same thing. Special thanks are owed to Scott and to Dominic Cooney for contributing their own weight behind this idea and building the momentum required to make it happen; to Blink developer Hayato Ito for his work updating the spec and implementing the feature in Blink; and to Dimitri Glazkov for his careful oversight of the Shadow DOM spec. This is one spec change that was totally worth the trouble!

Polymer list box and combo box elements

The updated Quetzal home page shows off a handful of new elements ported from QuickUI:

  • A quetzal-list-box element that presents its children as items in a list. This makes it easy to give any collection of DOM elements the semantics of a single-selection list. The actual tracking of the selection is handled with Polymer’s own polymer-selector element. On top of this, the Quetzal list box adds keyboard navigation: the user can navigate the list using the Up/Down arrow keys, as well as Home/End and Page Up/Page Down. This keyboard navigation follows the Microsoft Windows model, in which navigation keys move the selection, which in turn forces the list’s scroll position to update to show the selected item. This makes it faster to select an item with the keyboard. (In Mac OS X, paging up and down does not update the selection.)
  • A quetzal-combo-box element ties together a text input element, a popup, and a button to invoke or dismiss the popup. See this post on the original QuickUI combo box for details on combo box behavior. Again, a key feature here is keyboard support: the Down key invokes the popup (if it’s not already open), and the Escape key dismisses it. Another feature (inherited from the popup-source element) is positioning the popup above or below the text area as room allows. This combo box element doesn’t provide a specific popup UI. Rather, it’s intended to be used as a base class for custom combo box elements.
  • A quetzal-list-combo-box element combines the two above elements to create a typical combo box which presents choices as a dropdown list box. This includes standard auto-complete behavior. In many situations, a combo box like this is often a better UI solution than a plain text box with auto-complete: the user has a clearly visible button which can be used to invoke the complete list of choices. This is more discoverable than requiring the user to begin typing to see what the possible choices are. The auto-complete behavior matches against the textContent of the elements’ children, so the list items can be any type of item that has meaningful textContent. The sample shown uses a custom element that shows a color swatch next to a color name; typing some text will auto-complete against the set of color names.

These elements still need more work. For example, the combo box needs a stock mobile variant in which the element can fill the screen when making a selection to provide bigger hit targets and maximize the number of visible list choices. Also, at the moment simply tapping a list box element causes the Android Chrome browser to crash. This makes it somewhat hard to debug. :( Nevertheless, I think these element demonstrate that a good set of UI base classes will make it much easier to create custom UI designs as web components take hold.

Quetzal, take 2: creating general-purpose UI components as Polymer elements

Just an update to the experimental Quetzal project I posted about last month. I’ve learned quite a bit, and have changed course on two significant points, and thought I’d share a bit about my experiences with others who might be contemplating the creation of a framework or UI component library on top of web components.

A limitation of the current web component architecture

So far, the greatest challenge in replicating the range of expression in QuickUI’s UI components has been the inability to easily define HTML custom element subclasses that can fill in base class insertion points. To their credit, Google’s Blink (rendering engine) and Polymer (web component framework) teams have been responsive on that point. In response to a Polymer discussion board thread related to that post, a bug has been filed to allow one to distribute content nodes into a <shadow> insertion point. Unless you’re already experimenting with Shadow DOM, this entire issue undoubtedly seems quite arcane, but rest assured, fixing this one problem would open up a lot of important new territory.

I’m sincerely hoping this proposal is adopted. In the meantime, I’ve been using a workaround to simulate the effects of that proposal. That unfortunately broke in a recent Polymer update, but such setbacks are to be expected when building on shifting sand.

Trying Polymer elements

I’ve spent some time experimenting with Polymer’s own element layer, and last month decided to try adopting it. My first cut at creating custom elements for Quetzal was based directly on Polymer’s lower levels, which take care of polyfilling (emulating) various web component standards. I’m now trying to use the higher-level Polymer elements layer, which provides a richer framework for creating custom elements, and adds a number of features to simplify common component tasks. These features are not part of any web component standard, but by virtue of being done by Google, will nevertheless probably become a common feature of the web component landscape.

There are still a number of QuickUI features that have no parallel in standard web components or Polymer elements, but most of those missing features appear to be things which I could implement on top of Polymer elements. For example, a <quetzal-element> base class could provide those missing features, making it easy for new element classes to obtain them. Aside from the critical limitation mentioned above, it now appears to me that most (hopefully, all) of QuickUI could likely be implemented as Polymer elements.

With that change, the latest Quetzal iteration recreates a handful of QuickUI Catalog elements as Polymer elements. So far, this approach feels acceptable, and it would obviously be a big advantage to leave most of the heavy lifting to the Polymer team, and focus on actually creating new custom elements.

Some notes on switching to Polymer elements:

  • Using HTML to declare an element template feels quite verbose and cumbersome compared to QuickUI’s use of CoffeeScript’s concise JavaScript object format. If you haven’t tried CoffeeScript, it improves over plain JavaScript object syntax by using indentation instead of requiring tons of curly braces and commas. Using HTML feels comparatively ponderous and noisy, and to me all that noise makes custom element source files somewhat less legible. Still, HTML is the lingua franca of the web, and using a standard confers a huge advantage over a proprietary format, however efficient it might be. At some point, the Polymer team says they’ll support an imperative JavaScript means to define custom elements, but for now I’m guessing the vast majority of custom elements will use HTML, so that’s what I want to try.
  • Speaking of standard formats, one aspect of Polymer that’s recently changed is that the top-level tag used to define a new element is now <polymer-element>, instead of the standard <element>. While Polymer elements are now just as locked into the Polymer framework as before, this change makes it feel like I’m no longer using a web standard — it feels like just another proprietary UI framework that happens to use XML/HTML as its format. It’s surprising what a difference this small change makes. Using <element> simply felt better.

Going back to plain JavaScript

Switching the top-level component container to HTML instead of script has also prompted me to give up CoffeeScript, at least for now. I actually tried using a combination of CoffeeScript and HTML, but it felt like I was working against the grain, and I ended up giving up on that approach.

Going from CoffeeScript back to plain JavaScript is an excruciating experience. Oliver Wendell Holmes said, “Man’s mind, once stretched by a new idea, never regains its original dimensions.” I think the programming language analogue here is: once your brain has been expanded by a language that lets you more clearly express your intensions, trying to cram your brain back into the tiny box of a less expressive language is unbelievably painful.

Every single time I have to write a JavaScript loop instead of a CoffeeScript list comprehension, or type “function() {}.bind(this)” instead of just “=>”, I physically wince a bit. JavaScript just feels gross, it looks gross, it is gross.

That said, JavaScript is the standard. One thing I’ve learned from QuickUI is that if you’re trying to build a community around a common library, creating that library in a programming language with a narrow audience dramatically limits the rate at which you can grow. One commenter named “jokester” offered on my original Quetzal post: “I’ll unfortunately not contribute to a project coded in CoffeeScript.” Regardless of the advantages I believe CoffeeScript offers to developers, I’d rather allow orders of magnitude more people to contribute in the standard JavaScript language they’re already proficient in.

Anyway, that’s about the state of things. This Quetzal project is still just an experiment, and doesn’t do much useful yet, but it’s proving a good way to learn.

Puzzle: Define HTML custom element subclasses that can fill in base class insertion points

This post presents a little web component architectural puzzle which I’ve come across in the early stages of creating Quetzal. The puzzle deals with how Quetzal should best deliver an important component service on an HTML custom element substrate, and relates specifically to subclassing semantics. Any suggestions or comments would be much appreciated.

Background on the puzzle

Quetzal is an attempt to deliver key features of the QuickUI component model in HTML custom elements. One such feature is that an element subclass should be able to easily populate a slot (insertion points, in HTML parlance) defined by a base class. In practice, there are many situations in which you want to be able to say, “This new UI component should be just like that existing UI component, only with some stuff pre-filled in.” For example:

  • The QuickUI documentation presents a simple page template example in which classes in a small page template hierarchy fill in specific bits of their parents classes.
  • A DateComboBox fills in the popup portion of a ComboBox, which in turn is filling the content portion of a PopupSource. This same facility is also used throughout the QuickUI Catalog. Moreover, it is used in many QuickUI apps in which a stock Catalog component is specialized for the app’s context.
  • Along those same lines, this same issue should crop up in any organization that tries to create a library of standard components which implement the organization’s visual design language. Suppose your site’s designer has created a cool button class as an HTML custom element, and you have used that to create an Add to Cart button. You write some script so the button can show inside the button the existing number of items in a customer’s online shopping cart (to the right of the button label, say). You now want to package up the Add to Cart button so that it can be used as a component in its own right. For flexibility, you want the button’s text label to vary in places.

Well-defined subclassing semantics are essential for creating a UI component library with a good separation of concerns. If you look at the class hierarchy depicted for DateComboBox (above), you’ll get a sense of the degree to which it’s possible to portion out specific roles to a small constellation of classes, such that each class can focus on just doing one thing really well.

I’m hoping that it is possible to take advantage of such subclassing semantics in HTML custom elements — but it’s not proving to be particularly easy.

The puzzle

The puzzle is to come up with an architecture for custom element subclasses that meets the following design criteria:

  1. An instance of a subclass is a proper instance of its base class. All the normal JavaScript stuff should work: property/method access should go up the prototype chain, and a subclass instance should report that it is an “instanceof” the base class. By default, the HTML <element> syntax permits an “extends” attribute to identify a base class, but a purely script-based solution that sets up the class hierarchy correctly is equally valid.
  2. A subclass can put stuff into an insertion point defined by the base class. That is, the subclass can fill in a slot (or slots) defined by a base class. In turn, the subclass should be able to redefine such an insertion point so that the subclass itself can be subclassed.
  3. Unless overridden, all base class behavior should function properly in an instance of the subclass. E.g., if the base class wires up an event handler, then this works as expected for subclass instances too.
  4. Base class properties/methods can be overridden by the subclass. A subclass’ property/method implementation should be able to invoke the base class’ implementation by whatever language means are necessary. (CoffeeScript provides sugar for this; plain JavaScript developers have alternate ways of achieving the same result.)
  5. The base class can be any HTML custom element class; the base class author shouldn’t have to do special work a priori to enable this kind of subclassing. This ensures a Quetzal author can always use someone else’s element class as a base class — even if that other person has never heard of Quetzal.

A successful solution needs to meet all five of these criteria. So far, the approaches I’ve tried can satisfy at most four at a time.

Example

Let’s walk through a example from the small set of custom elements currently shown on the Quetzal home page. This set includes a base element class called quetzal-button that shows its content inside a button, and another element class called icon-button which adds an icon to the plain-button content. For clarity, here let’s just call that base class plain-button instead of quetzal-button, since the following source won’t actually involve Quetzal. In any event, we want markup like this:

<plain-button>Plain button</plain-button>
<icon-button icon="document.png">Icon button</icon-button>

… to produce something like this:

Buttons

Where icon-button is reusing all the styling and behavior from plain-button; it’s not duplicating the styling and behavior. The challenge is to create the icon-button element so it both inherits (in the class sense) from plain-button and extends the visual representation of plain-button.

A partial solution to filling in base class insertion points

The first challenge is: how can icon-button add elements to the content shown by plain-button? Some approaches:

  1. We could try to apply the template for both plain-button and icon-button to the same host element. The Shadow DOM spec supports multiple shadow trees attached to the same host. This feature alone is insufficient for the above example. Unless one shadow tree takes care to incorporate the other somehow, the most recently-added shadow subtree wins. If plain-button renders last, we get a button but no icon; if icon-button renders last, we get an icon but no button.
  2. We can include a <shadow> element in the template for icon-button, and ensure plain-button renders its shadow subtree first. The <shadow> element allows icon-button to effectively include the representation for plain-button. Unfortunately, this inclusion effectively wraps the base representation, rather than filling it in. An icon-button does get a button and an icon, but the icon and content render outside an empty button: shadow
  3. We can have icon-button create an instance of its own base class, then have that instance contain the icon and the icon-button’s own content. (This approach is based on a suggestion from Shadow DOM spec author Dimitri Glazkov.)

Approach #3 does what we want from a strictly visual perspective. (Behavior is a separate matter.) The source for a Polymer element version of this approach looks something like:

<element name="plain-button" extends="button">
    <template>
        <button>
            <content></content>
        </button>
    </template>
    …
</element>

<element name="icon-button" extends="plain-button" attributes="icon">
    <template>
        <plain-button>
            <img src="{{icon}}">
            <content></content>
        </plain-button>
    </template>
    …
</element>

See a live example of this. It should work in most browsers, but use Chrome if you want to see it working with real Shadow DOM. (Note: a Chrome bug prevents the buttons from responding correctly to mouse interactions. The buttons do receive mouse events, but for now it just doesn’t look that way.)

If you open the example and inspect it in Chrome dev tools, you’ll see that an icon-button has a shadow subtree containing a plain-button; the plain-button contains its own shadow subtree. Content of an <icon-button> element is therefore distributed twice: once into the <plain-button> element, and then again into the native <button> element. The ability of Shadow DOM to distribute nodes multiple times is called reprojection. (Or, at least, it meets the definition of reprojection as I understand it: “when an insertion point is a child node of another shadow host”.)

Inheritance versus containment

Unfortunately, while this approach looks right, it doesn’t behave quite right. An icon-button here is not only an instance of plain-button, it also contains a plain-button. That’s problematic.

  1. When icon-button instantiates its inner plain-button, the inner button has no way to know its relationship to icon-button. Among other things, this means icon-button can’t easily override behavior defined by plain-button (one of the design criteria above). In the JS Bin example, you can turn on the Console pane to see debug output. The plain-button element class defines a readyCallback (in Polymer, “ready”) that invokes a base class method called log(). The icon-button class overrides that log() method, but because the inner plain-button is just that — a plain-button — its readyCallback will invoke the base plain-button implementation of log() instead of icon-button’s specialized log() implementation. Running the demo invokes log() four times, when: 1) creating an instance of plain-button to use as the prototype for icon-button, 2) creating the visible plain-button with text “Plain button”, 3) creating the inner plain-button used by the visible icon-button, and 4) creating the visible icon-button with text “Icon button”. It’s #3 and #4 together that are the problem: what we really wanted to do is invoke icon-button’s log() implementation once.
  2. Automatic element references (a la Polymer, and also in Quetzal) aren’t inherited by default. If plain-button defines an element with id #foo, then plain-button methods can access that element via the automatic reference this.$.foo. Similarly, we want an icon-button to have access to the same reference this.$.foo defined by the base class. (Or, at least, we can debate whether such automatic references should be treated as “private” or “protected”, but it seems to me that “protected” would be useful.) It’s possible to work around this particular issue for a known set of frameworks — that is, Quetzal could workaround this problem for its own classes, and perhaps for those defined by Polymer — but it wouldn’t work in the general case of an unknown framework.
  3. It’s easy to end up in situations where both icon-button and plain-button are duplicating work. Suppose an icon-button method invokes a super-method of the same name defined by plain-button, and suppose the base implementation of that method performs expensive work or obtains a reference to some resource. When the inner plain-button is instantiated, it might do that work — and then the same work or allocation might be repeated by the outer icon-button when it invokes the super-method. Conventions could be established to avoid this, but it would complicate otherwise simple situations, and again make it hard to use subclass elements from other frameworks.
  4. As a common case of the above point, if the inner plain-button wires up an event handler, then it’s easy to end up in situations where the event is bound by both the inner plain-button and the outer icon-button. If the event bubbles up from something inside plain-button (a click on the button, say), you would end up handling the same event twice.

We could try to simplify things by just containing an plain-button, and not deriving from it. This forces us to give up one of the original design criteria outlined above: an instance of icon-button wouldn’t actually appear to be an “instanceof” plain-button. Moreover, if plain-button defined some attributes (“disabled”, say), icon-button would have to explicitly handle those too and forward their implementation to the inner plain-button.

We could have icon-button create a placeholder element (a <div>, say), create a shadow root for it, and populate that root with a copy of plain-button’s template but without actually instantiating that inner element as a real, live plain-button. This is the approach that Quetzal currently uses. It solves a number of problems, but is dependent on knowing how a given base element class works. Quetzal reaches into the base class’ implementation to obtain its template and then clones it, which might not be possible with other frameworks. This violates one of the design criteria above.

We could create a temporary instance of plain-button elsewhere, then clone just its contents into the icon-button instance. This avoids requiring detailed knowledge of what the base class is doing. But it could also result in subtle problems. E.g., the base class might not be expecting to have to serialize all its state into its shadow subtree, in which case the cloned content might not represent a coherent instance of the base class.

Looking for suggestions

This post is effectively a form of rubber duck debugging. The simple act of writing this up has forced me to better understand the problem, and led to consideration of alternate lines of attack. The puzzle remains unsolved, however. Given my understanding of custom elements, and the design criteria for the puzzle above, I’m not sure whether a solution exists.

It’s theoretically possible I’ve hit some limit in the expressiveness permitted to custom elements in their current state. Perhaps that limitation could be addressed. If not, I’d  have to write off a big chunk of the solutions used by the QuickUI Catalog and QuickUI apps, and find alternate ways of meeting the same needs.

I’m hoping, however, that I’m just missing something. If you have some passing familiarity with HTML custom elements and Shadow DOM, and have ideas about how to approach this problem within the existing technology, I’d love to hear them!

Do web component developers still need jQuery?

I love you, jQuery. But maybe it’s time we started spending some time apart.

Improvements in cross-browser feature set and compatibility are reducing the need for jQuery, but more importantly, it turns out that a component-based app needs very little jQuery to begin with. These insights led to one of the first decisions I hit in starting Quetzal: should it require jQuery? The answer, so far, is no.

Since its inception in 2007, QuickUI has relied on jQuery as a crucial browser abstraction layer. Without jQuery, QuickUI would have never have come to exist. However, the QuickUI runtime itself only uses a fairly small set of core jQuery functions, and modern browsers now deliver standards-based solutions for those situations. Moreover, I’ve noticed that, when building QuickUI component-based apps, component code tends to rarely require sophisticated jQuery.

Meanwhile, the web has moved forward. The modern browsers — the latest Chrome, Safari, Mozilla, and Internet Explorer 10+ — are all much, much more consistent to develop for than browsers were six years ago. Glancing at pages on QuirksMode, one generally sees a see of green compliance, with most of the red markers of non-compliance for older versions of IE. And according to StatCounter, market shares for both IE 8 and 9 have now dropped below 10% (each), so it’s plausible for mainstream organizations to justify ignoring them.

Let’s break down the things jQuery is good at, and consider how necessary are in a web component-based application (specifically) targeting the modern browsers:

  • Handling an array of DOM objects. It’s convenient to be able to apply a jQuery function or plugin to a jQuery object containing a collection of DOM elements. However, modern web languages (CoffeeScript, ES6, etc.) provide syntax for iterating over a collection, substantially reducing the need for a special library for this purpose. And, while jQuery’s array model is easy to consume, it also makes writing components harder. QuickUI control classes inherit from jQuery, which means every QuickUI class method has to consider the possibility it’s being applied to multiple objects at a time, not just one. In practice, that’s been a consistent source of complexity and bugs.
  • Selectors. The standard querySelector and querySelectorAll functions provide a reasonable way to find matching elements. (I believe those functions were at least partially inspired by jQuery.) I’ve heard that jQuery’s Sizzle engine provides more power than querySelector, but again, in a component-based app, you’re rarely doing sophisticated searching of the DOM. In practice, when I’m writing an app with QuickUI components, I do very, very little searching of the DOM. Every QuickUI component already has its own reference to exactly the sub-elements it cares about. Polymer does something similar with automatic node finding. With such facilities in place, there’s no need to grovel around in the DOM to find something you’re looking for. In fact, with Shadow DOM, the things you’re probably looking for aren’t even findable. This is a good thing; encapsulation is preventing you from writing brittle code. Every time you do a $(“#foo”), you’re running the risk of picking up the wrong #foo — maybe not today, but tomorrow, when someone else adds a #foo element somewhere on the page. In Quetzal or Polymer, you never search for something like that; a component already has a direct reference (this.$.foo) to the element you want to manipulate. Unlike a DOM search, dereferencing is instantaneous and 100% reliable.
  • Traversing. Ditto. In practice, component subtrees  — that is, the set of elements managed directly by a component — just aren’t that deep. A deep subtree is, in fact, often an indicator that component refactoring is in order. It’s exactly analogous to the way a deeply-nested set of code blocks (conditionals, loops) within a single function usually indicates the function should be refactored. Nearly all the time a QuickUI component needs to traverse the DOM, it wants to iterate over its own set of children — which, as noted above, can now be easily done in a modern language with decent syntax.
  • DOM manipulation. As browsers have become more consistent in the semantics of DOM operations, jQuery feels less necessary here. And much of the jQuery DOM manipulations one sees are a means to set up all or part of the DOM. in jQuery, one often sees code like: “Find all the divs of class ‘.menuItem’, and wrap them, stuff them, and wire them up so that they turn into menu items.” The existence of web components provides a much better way to accomplish the same result. All the population can be done through custom elements that provide a template for their DOM. That said, jQuery does provide a useful collection of helper functions. For example, to me it feels easier to use jQuery’s css() method, which can take a JavaScript object as a parameter, than use the raw DOM “style” property directly. Generally speaking, the DOM API feels more like an old school C API, while jQuery feels like a JavaScript API.
  • Function chaining. jQuery chaining lets you concisely apply a set of selector, traversal, and manipulation operations. In practice, all three of those types of operations come up less often in a web component-based app. In particular, one often sees long chains of jQuery function calls when populating the DOM, but a <template> is a cleaner way to do that declaratively. Over the years, in QuickUI apps, I’ve noticed that I use jQuery chaining less and less often, to the point where I only rarely take advantage of it today.
  • Events. Microsoft IE 9 finally added support for addEventListener, so that it’s possible to wire up event handlers in a consistent way. I expect there are still many discrepancies lurking in the details — when each browser decides to fire an event, for example — that might prove tricky to work around without an abstraction layer like jQuery that can normalize behavior.
  • Effects. CSS transitions and transition events now provide an easy, cross-browser way to do many of the same effects jQuery was first noted for. For years, the jQuery home page used to have a simple demo which, when you clicked a button, made a new DOM element appear with a transition effect. Such effects are now easily achievable without jQuery.
  • Data. jQuery provided a useful $.data service to associate data with DOM elements, because a browser’s garbage collector can get confused when DOM elements and JavaScript objects reference each other. The various browsers also had myriad bugs and inconsistencies with regard to extending DOM elements. So UI framework developers like those on Prototype gave up on extending the DOM. However, as far as I can tell from the way Polymer is tackling things, extending the DOM now appears to work (generally) as expected. So perhaps $.data is no longer necessary.
  • Ajax. I don’t write a ton of Ajax code myself, so for argument’s sake, let’s stipulate that jQuery’s Ajax wrappers are handier than directly working with XMLHttpRequest. In particular, jQuery’s use of promises as a data type simplify the task of writing async code. Perhaps for the time being, this particular aspect of jQuery is worth using on its own. A proposed browser standard for futures may reduce that benefit, however.
  • Plugins. Many, many of the jQuery plugins that exist today essentially create component-like things. These plugins effectively constitute a DOM template with some packaged behavior. I’d argue that a web component is a clearer, more maintainable way to achieve the same result. Encapsulation, in particular, is a huge advance to providing robust components. Moreover, many other uses for plugins could now be achieved by extending DOM elements directly.

Generally speaking, in a component-based app, you want to give each component the responsibility for managing its own appearance and behavior. You don’t want code walking all over the DOM tree and mucking around with things that aren’t its direct responsibility. Instead, you talk to the component managing the part of the DOM you care about, and ask that component to manipulate the elements it directly owns. In classic object-oriented programming terms, this is an application of the Law of Demeter.

In practice, compartmentalizing things that way leaves each component doing very simple manipulations on a comparatively small set of elements: instantiating a new element; iterating over its own children; applying or removing a style; etc. If a component wants to do something more sophisticated to its internals, more likely than not the component should delegate that operation to one of its own sub-components. The component’s own need to search is limited, reducing the need for a powerful selector/traversal engine. And, given a reasonably good programming language, simple DOM manipulations can be performed effectively — and with better performance — by directly accessing the DOM API.

I’m no cross-browser DOM API expert, and I’ve only just started to try to do things without jQuery. I could easily hit a landmine tomorrow, tripping upon some cross-browser nastiness I’ve been blithely unaware of, which jQuery for years has been invisibly and reliably been protecting me from. That said, work on Quetzal is progressing fairly smoothly without jQuery, and a week or two into this project, I’m not missing most of jQuery. It would be nice to have a much smaller library of helper functions which present a more JavaScript-flavored approach to the DOM API, as in jQuery’s css(), mentioned above.

From one standpoint, you could say I’ve just traded one browser abstraction layer (jQuery) for another (Polymer’s platform.js). However, platform.js feels like a pretty different animal than jQuery:

  • The beauty of Polymer’s approach of polyfilling forthcoming web standards in older browsers is that you can write code against the abstraction layer today that should invisibly start working against native implementations where and when those exist. It’s still early, and that promise remains to be proven — but that’s a very compelling promise.
  • Those future standards mean you’re working with facilities that someday every other developer will have (whether they want them or not). You’re writing code on top of a library that could get smaller over time, not larger.
  • The other difference with platform.js, as far as today’s web component developers are concerned, is that it’s really the only game in town. Unless you have the luxury of targeting the latest release of Chrome, you’ll need to use platform.js (or the complete polymer.js) to run on other browsers.
  • If one’s goal is to write components that many people will use, it makes sense to reduce the number of additional dependencies. If jQuery’s not required, then the lack of that dependency to some degree facilitates sharing.

We’ll see how the Quetzal experiment evolves, but so far, writing directly to the DOM API is working out okay.

So, jQuery, maybe we should spend some time apart. Maybe we should see some other people. It’s not you — it’s me! Don’t worry; I might miss you terribly and come running back. Or maybe we’ll just be friends.

It’s okay. We’ll always have IE6.