fsb v. web apps, Greenspun, RAD, and EC2

[Thomas Lord]

The other day, I was reading an interesting blog thread
about Microsoft's position relative to the open source market:

http://dlweinreb.wordpress.com/2007/11/17/microsoft-and-open-source/

It discusses various ways in which Microsoft might try to
protect the value of their proprietary software.   As I read it,
I realized that I couldn't get very excited about it for the specific
reason that I think the OS wars aren't going to be "won" at all:
the war itself will soon start rapidly going away, unresolved.

PC Operating Systems are, I think, at the peak of what Gartner
calls a "hype cycle" -- they are at the "Peak of Inflated Expectations":
we now know about as much as we ever will about their potential
value, and much of what we "know" is just hype (so current revenues
aren't sustainable), as we'll soon begin finding out.  The question of
"Linux v. MacOS v. Microsoft" is only going to become less important.

Here's what I think is happening:




* Abstract:

I'll briefly report a technical perspective of some coming
evolutions in systems software, and then I'll opine on some
implications: I'll talk about what kinds of open source projects
and businesses are likely to be commercially important in the
coming years.


* The Near Future: Web Applications,
                   Greenspun's Gadgets and RAD Clusters,
                   Personal Clouds

** Web Applications

  "Web applications" are those which present their user
  interface via a standard web browser.  A relatively new class
  of products is web applications that compete with traditional
  desktop applications: web based spreadsheets, word processors,
  email accounts, etc.

  Web applications have advantages for users and developers:
  They are accessible from -- portable to -- any
  standards-conforming browser that can connect to the
  appropriate remote services.  So they reduce development
  costs, are accessible from a very wide range of devices, and
  are built on a comparatively stable platform.  Because a web
  application is provided by a remote service, and that service
  typically serves many users at once, developers have a direct
  and immediate path to update the software for many users at
  once, instantly.  In some situations, the aggregation of
  user's data at the service affords the opportunity to offer
  new features.

  The capabilities of today's generation of web applications are
  limited by factors that are changing rapidly.  First, the
  applications often have poor interactive performance and poor
  graphical presentation.  Currently ongoing improvements to
  Javascript implementations and implementations of "Dynamic
  HTML" (such as scalable vector graphics), along side currently
  ongoing improvements to the programmer's toolkits used to
  build these applications, are likely to quickly close the user
  interface gap.  Second, the applications are often limited
  because a user's data becomes "siloed" at the service-provider
  site.  So isolated, it is awkward, at best, to apply third
  party software to the data unless the third party tool has
  specifically been ported to connect with the service where the
  data is stored.  Currently ongoing development of new
  "commonly supported Web APIs," including APIs for
  user-controlled storage, are likely to close rest of the
  capability gap between web applications and traditional
  applications.  (Examples: OpenID, Amazon SC3)


** Greenspun's Gadgets[1] and RAD Clusters[2]

    [1] http://philip.greenspun.com/business/mobile-phone-as-home-computer
    [2] http://philip.greenspun.com/business/mobile-phone-as-home-computer


  The marketplace for hardware is changing rapidly, partly in
  response to improvements in and the promise of web
  applications.

  On the consumer side, the distinctions between personal
  computer and terminal (or other peripheral) are rapidly
  dissolving as is especially visible in convergences between
  instant messaging, web browsing, cellular telephony, gps,
  digital photography, general purpose storage, and music
  playing.  It can also be seen in products such as "Linux on a
  USB memory stick" intended to exploit public PCs as,
  essentially, terminals.

  On the provider side, increasing demand for web applications
  has created increasing demand for server-side computing,
  making it a much larger share of the market than in past
  years.  With the new scale, freed from the constraints of
  having to ship individual stand-alone servers to remote
  customers, hardware engineers have returned to once again
  designing (quite large) mainframe computing clusters.  Current
  ongoing research in the efficient administration of these
  large clusters is likely to quickly produce working system
  (e.g., RAD).  Some products for on-demand virtualized
  computing already exist (e.g. EC2).

** The Hype Curve and Prospects of Maturation

  As Gartner[3] might put it, if they agree with my assessment,
  web applications in general are past the "Technology Trigger"
  that starts things off, but still not yet at the "Peak of
  Inflated Expectations" which means we still have more to learn
  about their long term value.  Individual web application
  platforms, such as components of a LAMP-like stack, go through
  their own hype cycles with many current technologies somewhere
  in their "Trough of Disillusionment" (the point at which
  inflated expectations are essentially dispelled), or "Slope of
  Enlightenment" (when the advocacy learns to concentrate on
  what is truly good about the technology) and the "Plateau of
  Productivity" where the main activity is exploiting those
  virtues.

    [3] http://www.gartner.com/pages/story.php.id.8795.s.8.jsp

  Against the still-building hype, then:

  It is exaggerated to predict the imminent death of the PC.
  The sunk costs in PC-based infrastructure (whether desktop or
  server) are huge, the switching costs high, the maintenance
  costs low, and the incremental upgrade costs moderate.  Web
  applications with a compelling advantage can erode that market
  by attrition, but a rapid abandonment of the PC, being so
  fantastically expensive, is certainly impossible.

  The current generation of web application products includes
  some compelling winners, but also many examples that clearly
  promise more than they deliver.  At the same time, technology
  well along in the pipeline appears to be able to close these
  gaps, and so we're not yet at the point in which we know all
  we kneed to know about the full capabilities of web
  applications (that is, we're still to the left of the inflated
  expectations peak of the hype curve).

  The advent of personal, fairly powerful browser-based
  terminals with local storage (such as some current cellphones)
  -- an early example of a Greenspun Gadget -- helps to prove
  that a long term "Plateau of Productivity" exists for web
  applications (because, were it not for already existing web
  applications, these devices would have considerably less
  appeal).

  And, against the still-building counter-hype:

  I have heard it suggested that Microsoft will eventually
  dominate the market for web applications by leveraging their
  Office and operating system products.  This seems unlikely for
  two reasons.  First, a web service that "works better" for
  those who use other Microsoft products is, equivalently, a web
  service that works less well for anyone not using Microsoft
  products.  So, for example, a web service that benefits Office
  users is less interesting to an iPhone user than a web service
  that doesn't require the use of Office.  Second, technological
  advantages to PC software are eroding, as new products emerge
  that allow users to connect their browser to private storage,
  and private computation (both operated by a web application
  viewed in the browser).  (E.g., Google's new client-side
  storage and compute service -- but also imagine that in the
  form of a Greenspun gadget with a cell-phone, pda, or close
  form-factor.

  Microsoft can extend the life of its operating system and
  Office products by building in good support for web services,
  but they can't prevent others from building services that
  deliver the same functionality without tying that
  functionality to a Microsoft platform.


** Market Size

  Essentially everyone (in developed regions) already does or is
  soon likely to use multiple web applications, every day.  Much
  of this activity is monetized and its overall value well
  studied.  This is a very large market.

  The market for an individual web application is murkier, of
  course.

  Upstream of individual web applications are markets for
  service hosting, development platforms, and pre-packaged
  applications.  

  Upstream service hosting is usefully aggregated as a wholesale
  product (e.g., in the manner of EC2).  This is because an
  upstream service, by using fine-grain provisioning-on-demand,
  can exploit its scale to operate hardware more efficiently
  than any one customer could.  One can estimate the size of
  this market by estimating the number of cluster nodes demand
  will support and multiplying that by the cost of such nodes
  when purchased only at the scales of individual customers.  At
  any rate, it is quite large and the potential profit is then
  made by delivering equivalent or superior service, at around
  the same price, but with lower marginal costs per node thanks
  to the efficiencies of virtualization and scale.  (This market
  is already in a highly competitive build-out phase in
  anticipation of imminent demand.)

  Upstream development platforms and pre-packaged applications
  are a segregated market.  

  In one segment, which we can dub the "Systems Segment",
  technologies such as virtualization platforms are sold to
  sellers of clustered computing (such as Amazon).  Revenues in
  this market theoretically grow with the number of cluster
  nodes supported (even in open source markets) but profits are
  limited by the profit margins of hosting sellers.  That is,
  the "Systems Segment" market is a *part of* the hosting
  service market with profit no greater than the potential scale
  related efficiencies of virtualized service hosting.
  Moreover, it is a minority share of that market since the
  marginal cost of producing and installing software for the
  cluster is tiny compared to the capital investment in
  hardware, and the costs of energy, maintenance, and bandwidth.

  In another segment, which we can dub the "Platform Segment",
  technologies such as relational databases and web application
  frameworks are sold (or often, in the open source world,
  given) to those who sell web services.   (e.g. MySQL to
  Craigslist.)   This is, for similar reasons, but a part of
  the overall market for service hosting.   For some firms,
  such as Oracle, the platform segment is quite a bit larger
  than the overall systems segment of the market.  For most
  firms, the platform segment is a very small market.

  In the third segment: "Complete Applications Segment,"
  technologies are sold or given either directly to end-users,
  or to web service sellers who intend to add value by hosting
  the app and adding content, not usually by changing the
  program itself.  (e.g., a Facebook page to a customer.)  For
  similar reasons, this is a part of the systems segment.  It is
  the smallest part of those listed here.


* Opinion: The Overlooked Value Proposition

** Where the Money Currently Goes

  Every successful web application business is able to command
  some price for the activities of users where that price
  exceeds the marginal cost of hosting the service for each
  user.  The price might be paid by each user individually, by
  advertisers, or by some other means.

  That price, for hosting each user for a particular service, is
  the one point in the chain of transactions where price is
  *not* constrained by the marginal cost of service hosting.
  The systems segment, platform segment, and complete
  applications segment are all limited markets.  By definition,
  they are but a fraction of the profit margin for efficient
  service hosting.  But the price for hosting users is largely
  independent of the cost of service hosting: it reflects the
  use-value of the service.

  Web application providers are able to lay claim to the
  use-value profit of each customer because they maintain a
  monopoly: They provision computing and cycles for the user,
  but they monopolize the right to install application software
  on the server, and often they monopolize the software the user
  wants to install there.  To use a service, a user must either
  himself become a web application provider and host (perhaps
  even write) the service himself, or pay a web application
  service provider.  In the latter case, the application
  provider can claim all of the use-value profit.


** The Overlooked Value Proposition

  There is overlooked value, therefore, in finding ways to
  sell web application *software* directly to users, if a way
  can be found to do so that produces revenue in proportion
  to the use-value of users using that software.

  This means breaking up some current trades into multiple
  trades:

  A typical trade, today:

          * blog host sells space and data to advertisers
          * blog host trades blog to end-user for data and
            advertising space


  The blog host is a monopolizing middle-man there, brokering
  data and ad-space in exchange for ads, and taking all of the
  profit from the ads.

  A trade that would shift profits to software providers:

          * generic service host sells space/cycles to end-user.

          * user subscribes to and installs an "OpenAdSeller"
            program on the hosting service he is renting -- he
            programs it with personal data he is willing to
            sell.  The OpenAdSeller programs is a software agent
            that shops for ads to place on the user's site.
            The subscription to OpenAdSeller is likely free.
 
          * user subscribes to and installs "MyBlog" -- a
            blog service program.   So, now his generic service
            hosts his blog.   This subscription is also, likely
            free but includes a condition: the user will only
            receive automatic updates to the program if they
            configure OpenAdSeller to place ads on the blog
            and share revenues with the seller of the software
            subscription

  Of course, the user might install more than just a blog.
  Or the user might install add-on features to a blog, such as
  hooks to a aggregating services (e.g., feeds).

  As a convenience, the user should be able to pay for
  some or all of the generic service host by suitably
  configuring OpenAdSeller.   A key factor here, distinguishing
  this approach from that being taken by Google, is that
  OpenAdSeller must (a) Host user's private data on storage
  privately leased to the user and not mined by the hosting
  service.  (b) Create an open market for ad seller's rather
  than relying on just one or two particular ad brokers.

  These changes move some profit around, benefit users, and
  enable new kinds of on-line applications:

  The profit (from the use-value of web applications) which is
  currently monopolized by ad aggregators and web application
  hosts is, in the new system of trades, given directly to the
  users whose activity create that value.  For example:
  advertisers pay ad viewers.

  The "agreeable medium" for that advertising and other trade is
  the web application software chosen by the user.  The user
  obtains use-value that exceeds the deliberately chosen
  exposure of privacy, the ad sellers obtain their use value
  from the viewing opportunities and data, and the technologies:
  a generic, programmable web application -- and OpenAdSeller --
  cut out the middlemen between developers of web application
  software and the customers who, ultimately, pay for its use.
  Users enjoy greater control over their privacy and greater
  creative freedom over their web experience.

  There is no need to limit the new system of trades to merely
  "personal" software.  On-demand provisioning of generic
  service instances can be automated *in response to revenue*.
  That is, if usage for a user-installed service is high, and
  OpenAdSeller revenues are sufficient, the generic host service
  can automatically expand itself to additional nodes.
  (There is at least one firm already using EC2 and programmed
  provisioning in repsonse to demand for a consumer-facing
  web application but its name escapes me.   In any event, it is
  not an example of what I'm talking about because the user is
  not choosing what to install and the user is not the owner of
  storage for the user's data.)


* Technical Challenges and Opportunities

  Some of the needed components, such as the hypothesized
  OpenAdSeller program, are not difficult to imagine.  I imagine
  that there will be little difficulty in attracting support for
  building such parts in open source form.

  If all of the software components were available, working
  examples of the proposed business model could be created, with
  low initial investment, by building services atop Amazon's EC2
  (for example).

  One software component is very difficult: the "generic
  (programmable) web application".  This component is worth
  thinking about in some detail.

  In keeping with the cornerstone focus mentioned earlier, the
  user perspective comes first:

  The package management system -- the thing at allows users to
  interactively install, remove, or modify programs that they
  subscribe to, must be exceptionally simple to use.

  A user's hosted services and data must be trivially portable
  (to other generic application hosts), with or without
  preserving the root URL of the user's services.   (A
  peer-to-peer service would be useful for translating
  host-independent links to a user's services into URLs
  corresponding to where those services are currently hosted:
  a decentralized and multi-federated DNS for service names.)

  Another cornerstone focus for the generic (programmable) web
  application is to give maximum practical advantage to
  service hosting providers and to third party developers of
  applications.   In practical terms: defining the generic
  web service as simply and generically as possible (e.g.,
  so that it does not *require* the presence of a POSIX
  shell program) and defining the web service, to the greatest
  extent possible, in terms of existing and stable standards.

  Standardization, within the W3C framework, of the client-side
  components of web applications has already been largely
  accomplished.   What's next is standardization, within the
  W3C framework, of server-side components.

  I have thought long and hard on -- and coded up -- generic
  web services that I think begin to fit the bill.  (See
  www.dasht-exp-1a.com/xqvm)

-t