Can a Phoenix arise from the ashes of Mumps? (Part 2)

Ask a Mumps devotee to explain what he/she believes is its magic ingredient and you’ll get one or more of the following being highlighted:

  • the easy-going interpreted language and its loosely-typed nature, which allows for rapid development
  • the database which uses a concept and model known as Global Storage.  This storage model and its impressive flexibility is described in detail in the “Universal NoSQL” paper that I co-wrote with George James.
  • the fact that the language and database are tightly-coupled.  Mumps is neither a language nor a database, but a combination of the two.

My analysis in part 1 of this article suggests that the rest of the IT industry have never recognised anything particularly special or noteworthy about these characteristics which I think is our (the Mumps community’s) fault, not their’s.

Let’s examine each of the three key magic ingredients of Mumps in a bit more detail:

The Language

If we look at the Mumps language, take out the parts that relate to the manipulation of the database and, unfortunately, you have a language that shows little other than its age.  Whether the Mumps development community like it or not, the reality is that no modern developer will tolerate a language that is as simplistic, limited and old-fashioned as Mumps.

On the other hand, look amongst the most popular of the modern computer languages and the one that I believe has the most similar feel to the Mumps language and puts a similar smile on your face when you’re developing with it is Javascript.  Indeed, by a variety of measures, Javascript has, in recent years, risen to the number 1 slot.  Why are the reasons for its popularity?  Well, of course, the fact that it’s the automatic language of choice for web developers is due to it being built-in to every browser. However, it’s also now proving hugely successful in the server, courtesy of the phenomenon that is Node.js.

Two features that keep coming up as key reasons why developers love working in Javascript are that it is interpreted and that it has loose typing: the same two key language magic ingredients also beloved by Mumps developers.

The Database

If we look at the Mumps database, I think that there has been a long-term problem in people “getting” Global storage.  It seems a simple-enough concept and very powerful and flexible database model to those of us who have “got it”, but my experience of explaining it to others outside the Mumps development community is you either get back a blank look or a rapid dismissal that anything like that could ever be useful at all.  To those that have never encountered Mumps Global Storage, the database model is a multi-dimensional associative array.  For those of you familiar with PHP’s associative arrays, it’s effectively having the ability to store those arrays on disk rather than simply have them defined in memory.

Global storage is schema-free and completely dynamic.  Nothing requires pre-declaration or pre-assignment.  You just create, change and delete data within any level of the multi-dimensional arrays you want to access.

Unfortunately such a concept seems to not, in itself, be particular inspiring to developers outside the Mumps community, and I think it’s because such a unit of storage doesn’t seem natural, relevant or useful within any other language.

As I noted in Part 1 of this article, I think that timing has always been unfortunate for Mumps Globals as a database model.  Non-relational database storage fell completely out of fashion at the very time Mumps was emerging, and by the time the NoSQL movement came along, the industry’s focus was on creating new, Open Source databases that could work well with the accepted modern languages.  NoSQL databases rapidly differentiated into four main types:

  • Key/Value storage (eg Redis)
  • Columnar storage (eg Cassandra)
  • Document storage (eg MongoDB)
  • Graph storage (eg Neo4J)

The more generic Global storage model of Mumps, whilst flexible enough to be used to abstract all four of these NoSQL genres at once, suffers, I think, from being too non-specific and too generic for the new NoSQL generation to “get”.

However, the chameleon-like multiple personality of Mumps Global Storage is undoubtedly a unique and very powerful feature of Mumps Globals.  Every other NoSQL database is a “one-trick pony”: optimised and good at only one of those four main types.  If only Global Storage could be presented and made available to the new generation of developers in terms and in a way they could “get”, perhaps we’d be onto something.

Integrated Language and Database

The fact that Mumps is both a language and database at the same time has been both its strongest point and its most unfortunate and confusing feature.

A fully signed-up member of the Mumps zealot community know just how easy this integrated database and language makes the development of database applications.  For example, I believe there’s no way that something as large and complex as the VA’s VistA application could have been developed using any other database technology and language using such a small development resource and in such a short time.  It wasn’t because of the language per se, and it wasn’t because of the database per se, but because of some particular magic that stems from the fact that the language and database work together in such a slick and seamless way.

However, this tight integration of language and database has always meant that in order to get the best out of Mumps, you’ve had to use all its parts: ie you’ve had to learn its old out-dated language before you could get the benefits of the database.

In part 1 of this article, I mentioned that there have been lots of attempts to make the Mumps database accessible from other mainstream languages.  I think this has never been fully successful for two reasons:

  • language/database integration has never achieved the level of what I’d describe as intimacy that we see in Mumps
  • Mumps Global Storage hasn’t been exposed to the language in a way that is natural enough for practitioners of that language to “get”

In fact, I now believe that it’s these two issues that are the key to understanding why Mumps has never taken off more widely and also the key to turning around the dangerously critical situation we now face.

I believe that the single most powerful feature of Mumps is a single little character that we Mumps developers have had sitting under our noses for all those years, that all Mumps developers use daily.  Actually it’s not the character, but what it represents and its significance.  I’m talking about the ^ symbol.

Mumps developers all know that the only difference between a so-called local array (think PHP-like associative array) and a Global is a ^ in front of the name.  In other words, unlike pretty much every other modern database I can think of, a Mumps developer has an intimate and seamless relationship with the database.  There’s no disconnect or clear line between whats in memory and what’s on disk.  An in-memory array of data is created and manipulated in exactly the same way as an on-disk array of data: the only difference is a ^ in front of the array name which then denotes it as being an array that resides permanently on disk.  There’s no explicit opening of the database, and nothing in the code that particularly says: at this point I need to fetch some data from the database or save some data back to the database. Disk access is implicit, not explicit.

Let me give you an analogy with Javascript and JSON: it’s as if you could create and manipulate a JSON object using normal Javascript object syntax, but instead of that JSON object being in memory, it was actually stored on disk without you particularly knowing or caring.  Not only that, but you would have direct access to and you would be able to manipulate every name/value pair within the on-disk JSON object, and at any level of its object hierarchy, all using normal Javascript/JSON syntax just like you were accessing or manipulating a normal in-memory JSON object.  A Native JSON database, if you will.

Let’s dig into that analogy a bit deeper.  That sounds like MongoDB, you might say.  Well, yes and no.  Yes, MongoDB is designed to store and retrieve JSON documents and Javascript can be used to interface it.  But note the subtle yet powerful difference: in MongoDB, like all other databases, a developer accessing MongoDB will reach a point in the language they are using where they want to retrieve some data from the database – in this case a JSON document, or perhaps a subset of it or result-set derived from a query of it.  They can then play around with the resulting information in-memory within Javascript.  Then, at some other point they will decide to store some JSON back into the database.

Normally, then, a developer is explicitly moving data between database (ie disk) and language (memory) and back again.

Whilst you can do the same thing with a Mumps database, you also have this extra dimension of direct access to the database.  You can literally poke around and touch the on-disk data in the database directly from the language in the same way you can touch the data that is in memory.

The natural representation of data for a Mumps developer is the so-called array, and the only difference between an in-memory array and an on-disk array is whether or not there is a ^ symbol in front of its name.  It turns out to be an immensely powerful and liberating subtlety and I believe it’s the crowning achievement of Mumps and the key magic ingredient that needs to be introduced into a suitable alternative language.

The Phoenix Rises

So I believe we have the basic ingredients for a Phoenix to rise from the ashes of Mumps.

If the Mumps language itself is irretrievably dead, then clearly we need some other language to replace it: one that is acceptable in the eyes of the modern development community and one that ideally has all the same hallmarks that gave the Mumps language its edge in the eyes of Mumps developers.

From my experience it’s clear we have that language: it’s Javascript.  I’ve played around at some point in my career with pretty much all of the other potential candidates for the role, and nothing comes close to having that certain “je ne said quoi” that I, as a Mumps developer, have always looked for.  Fortunately it’s highly popular – many now say it’s the most popular language in use today.  Additionally, through the endeavours of Douglas Crockford in particular, it’s also now highly regarded as a highly capable, professional language.  So we really couldn’t have a better candidate as a language that can resurrect our Phoenix.

What Mumps has benefitted from but what is lacking in Javascript is an integrated database, ie one where the natural unit of data can be seamlessly accessed either locally (in memory) or on disk.  Whereas the natural unit of data for a Mumps developer is the array, for the Javascript developer it’s the JSON object.

Mumps Globals are highly flexible, schema-free, free-form database structures.  Meanwhile, Javascript objects are completely dynamic and highly-flexible.  It turns out that they are exactly suited for providing an OO abstraction of Mumps global storage.  Unlike Cache’s pre-determined class-based mapping of objects on top of Global Storage which has proved pretty limiting, Javascript’s objects, based on prototypal inheritance, can provide a completely dynamic object model that fully reflects and matches the completely dynamic hierarchical storage model of Mumps Globals.  So, once again, we really couldn’t have a better candidate language than Javascript to provide a dynamic, flexible OO abstraction that can make the most of the power and flexibility of Mumps Globals.

A smart combination of Javascript and Mumps globals is therefore what is going to provide us with the way forward.

Why can this approach succeed whereas so many attempts in the past have failed?  I think that it’s the recognition of three key things:

  • it’s not the Mumps language per se that has been the magic ingredient, but certain features of the Mumps language.  It turns out that similar and equivalent features can also be found in Javascript.  We can and should let the Mumps language die out, and focus on Javascript and its developer community as our future.
  • a key magic ingredient has definitely been the Mumps database, but the mistake that has been made in the past has been to fail to understand that it has to be exposed or abstracted in a way that is meaningful and natural within the language.  Persistent associative arrays are natural in the Mumps language but not to a Javascript developer.  If we’re using Javascript as the new language, then Global Storage must be exposed and abstracted as JSON and accessed as standard Javascript objects.  Our same old faithful database, but re-cast in Javascript terms.
  • - the other vitally important magic ingredient is achieving a similar level of intimate integration between the language and database as we have in Mumps.  If we’re moving to Javascript as the language, then we must achieve a seamless blurring of the distinction between Javascript objects that are in-memory and on-disk.  If we do so, it will be a unique achievement.

If we can pull off such a set of tricks and create such a Phoenix, then we’ll have achieved two things:

  • Javascript developers will be able to make sense of the data within an application such as VistA, because it will be exposed and abstracted in terms that make sense to them: they’ll “get it”.  If we can further abstract legacy Mumps APIs such as Fileman into Javascript objects and methods (which I also know we can do), then we have a means of recruiting a new generation of developers to help build and expand upon the huge legacy of existing code in applications such as VistA.  They won’t need to know Mumps code per se, and we don’t need to rewrite all that legacy of Mumps code into Javascript before they’ll be able to understand and use it.  Over time, however, I envisage a migration of code from the Mumps language to Javascript.
  • Whilst not an essential part of this strategy, I believe there will be sufficient of an intrigue factor that Javascript developers will begin to want to find out more about the intimate database integration we’re providing.  Pretty much everything that MongoDB can do will be possible with our regenerated Phoenix.  I think that pitching it as a Native JSON Database is the correct first step in this regard.  Furthermore, we can follow that up with a re-casting of the other 3 NoSQL database types: this will further demonstrate the power and unique capabilities of our Phoenix.  For once, we might have something that others start actively seeking out, exploring and evaluating because it’s relevant to them and different from what else is out there.

None of this is speculative pie-in-the-sky.  It’s all there, ready and available today.  The Phoenix is ready to arise.

In the third part of this article I shall flesh out the technical details and explain how and where to get your own hands (and ideally the hands of a Javascript developer) on the Phoenix and see it for yourself.

About these ads

4 comments

  1. Excellent post Rob.
    Very well written article on the important and misunderstood power of Mumps.
    Equally reasoned view of a smart way forward to widen its appeal to the coding masses.
    Your exposition of the anatomy of this Phoenix is now awaited..
    Thanks
    Tony

  2. Rob,

    It is great to see such a crisp description of what makes MUMPS so unique. Being a newcomer to the M world, I certainly recognize that it takes a while to understand what is so special about the Language/Database that is M (and I clearly wouldn’t claim to have “gotten” it at all yet).

    Clearly separating the Language concepts from the Database concepts is a great way to introduce new developers to this space. Once those two pillars are solid, re-combining the Language concepts with the Database, would provide a solid structure for developers to become productive.

    Your description turns out to be an good guide on how an educational program can be put together to train a new generation.

    There is yet an implicit topic in here, that I hope you will touch on in a future post, and it is the fact that because M integrated the Language and the Database, it also primed the developers mindset to think of Data in terms of “a Language”, and somehow resulted in the M community having an implicit use of concepts of Linked Data (way ahead of its time), that the larger CS and IT community considered to have been rediscovered in the form of RDF and Semantic Web.

    That is, the M way of thinking about data, put developers in a situation where they are prone to think about “meaning” much more than about simple “storage”. Relationships and semantics are put the forefront in the case of M arrays, which is quite different from the simple data collection mindset that one would have when storing data into tables.

  3. Hi Rob, great article!

    I agree that mumps is different in how it integrates its database directly into its language but I disagree with the ‘^’ part. The ‘^’ in mumps is part of its syntax and could be implemented in most languages using (fake) operator overloading. For javascript this would look like this:

    x = ‘some string'; // local variable
    x ^= ['path', 'to', 'a', 'node']; // global, e.g. database access

    Some important questions that need to be answered when choosing between mumps and any other programming language: Is database separation necessary? What if I decide to change database or programming language in the future? How to shard/scale/monitor/test/document/measure etc.
    I am a seasoned javascript/python/java programmer, to me it feels like being locked-in when I pick a programming language that has an integrated database and it seems this is the reason why all the behemoth mumps software projects have never been rewritten in a more modern language. Also, this tight integration between data and logic might be the reason why the language itself has not improved much?

    Ivan

    1. Ivan

      Glad you liked the article. Two comments on your thoughts:

      1) There’s more to it than just replicating the ^ concept. In my opinion the raw hierarchical database structure of Mumps doesn’t fit naturally with the modern object-orientated view of a database – hence the implicit mapping between JavaScript objects and a physical Global store that underpins them and makes them persistent. Making Mumps acceptable to a JavaScript developer means presenting the database in terms that are natural and seamless to him/her.

      2) I’m not actually suggesting or creating a tightly-integrated lock in – the integration is done through a combination of an interface between Node.js and a Mumps database together with a syntactic layer that provides an object mapping to Globals that appears intimate but is in fact technically decoupled from the Mumps database (unlike the Mumps language). I’m not suggesting, for example, that a Mumps technology vendor should implement JavaScript as a tightly-integrated native language – I agree with you that this would be a bad idea.

      So I think we’re in closer agreement than it might at first appear.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 57 other followers

%d bloggers like this: