Using Mumps Globals To Model a System (Part 2)

In Part 1 I introduced the ewdDOM module for Node.js which implements a persistent XML DOM using Mumps Global Storage.  By the end of the article, we saw how a new, but empty, DOM was modelled using a number of Global Nodes that worked together, and how it was the Javascript logic with the ewdDOM module’s functions that controlled how and why those Global Nodes worked together.

In Part 2, we’re going to build out a simple XML Document using the DOM APIs that are implemented in the ewdDOM module, and examine in detail how Mumps Global Storage is used to bring the DOM to life.

The first thing we’re going to do is create our XML document’s outer tag, which we’re going to call ‘testElement’.  In other words, we want to create a minimal XML document that just contains a single tag:

   <testElement />

The first step is to edit the file named domTest.js that we created in Part 1.  Find the lines that we last edited:

var document = dom.createDocument('myDocument');
console.log("\r\nHow myDocument is physically represented in the Mumps Global:\r\n");
console.log(JSON.stringify(xmldom._getDocument(), null, 2));

and edit this section to appear as follows (additions shown in italics):

var document = dom.createDocument('myDocument');
var node1 = document.createElement('testElement');
console.log("\r\nHow myDocument is physically represented in the Mumps Global:\r\n");
console.log(JSON.stringify(xmldom._getDocument(), null, 2));
console.log("\r\nXML Document Listing:\r\n");
document.output();

So we’re going to use two Document-specific methods:

  • createElement(): this creates a new XML Element Node (ie a Tag) within the Document
  • output(): this “walks” the XML DOM tree of Nodes, recursively following its firstChild and nextSibling pointers, and outputs to the console a listing of the Document in XML format.

Save and run the file as before:

  node domTest

The output from xmldom._getDocument() (ie the contents of the xmldom Global in which our DOM has been modelled) should appear as follows.  The differences since the last time we ran domTest.js are shown in italics and are what were created by the createElement() function.:

{
  "docNameIndex": {
    "myDocument": 1
  },
  "documentCounter": 1,
  "dom": {
    "1": {
      "creationDate": 1360568914877,
      "docName": "myDocument",
      "node": {
        "1": {
          "nodeType": 9
        },
        "2": {
          "nodeName": "testElement",
          "nodeType": 1
        }
      },
      "nodeCounter": 2,
      "nodeNameIndex": {
        "testElement": {
          "2": ""
        }
      },
      "nodeTypeIndex": {
        "1": {
          "2": ""
        }
      }
    }
  }
}

So, in summary, the createElement() function did the following:

1) incremented the nodeCounter GlobalNode for document #1:

{
  "dom": {
    "1": {
      "nodeCounter": 2
    }
  }
}

2) Added a new Node (#2) to document #1, with a nodeType property of 1 (=Element) and a nodeName property (ie TagName) of “testElement“.

{
  "dom": {
    "1": {
      "node": {
        "2": {
          "nodeName": "testElement",
          "nodeType": 1
        }
      }
    }
  }
}

3) Added an index Global Node, allowing Nodes within Document #1 to be searched by NodeName. Note the way this index node provides a pointer back to Node #2:

{
  "dom": {
    "1": {
      "nodeNameIndex": {
        "testElement": {
          "2": ""
        }
      }
    }
  }
}

4) Added an index Global Node, allowing Nodes within Document #1 to be searched by NodeType. Note the way this index node provides a pointer back to Node #2:

{
  "dom": {
    "1": {
      "nodeTypeIndex": {
        "1": {
          "2": ""
        }
      }
    }
  }
}

I won’t explain it in detail, but here’s the relevant code in ewdDOM that creates these Global Nodes:

  createNode: function(nodeName, nodeType, docId) {
    var node = '';
    if (nodeName !== '') {
      if (nodeType !== '') {
        if (ewdDOM.documentExists(docId)) {
          var nodeNo = ewdDOM.incrementNodeNo(docId);
          node = {docId: docId, id: nodeNo};
          ewdDOM.setNodeType(node, nodeType);
          ewdDOM.setNodeName(node, nodeName);
        }
      }
    }
    return node;
  },
  documentExists: function(docId) {
    if (docId === '') return false;
    return ewdDOM.dom.$(docId)._exists;
  },
  setNodeName: function(node, nodeName) {
    var docNo = node.docId;
    var nodeNo = node.id;
    var domNode = ewdDOM.dom.$(docNo);
    domNode.$('node').$(nodeNo).$('nodeName')._value = nodeName;
    if (ewdDOM.getNodeType(node) === 1) {
      domNode.$('nodeNameIndex').$(nodeName).$(nodeNo)._value = '';
    }
  },
  setNodeType: function(node, nodeType) {
    var docNo = node.docId;
    var nodeNo = node.id;
    var domNode = ewdDOM.dom.$(docNo);
    domNode.$('node').$(nodeNo).$('nodeType')._value = nodeType;
    domNode.$('nodeTypeIndex').$(nodeType).$(nodeNo)._value = '';
  }

Look again at the output you got back when you ran domTest.js and you’ll notice something that may initially be surprising: although we’ve clearly created the testElement Node, nothing was listed by document.output().  That’s because, although we created the new Node and added it into the Document, we haven’t yet attached it to anything, so it’s just floating free within the Document.  So what we must do is attach our new Node to the top-level Document_Node that was originally created by the createDocument() function.

To do this, edit the relevant lines within domTest.js to appear as follows:

 
var document = dom.createDocument('myDocument');
var node1 = document.createElement('testElement');
var documentNode = document.getDocumentNode();
documentNode.appendChild(node1);
console.log("\r\nHow myDocument is physically represented in the Mumps Global:\r\n");
console.log(JSON.stringify(xmldom._getDocument(), null, 2));
console.log("\r\nXML Document Listing:\r\n");
document.output();

So what we’ve done is to obtain a pointer to the top-level Document_Node, and then invoked its appendChild() method to attach the ‘testElement’ Tag to it.

Now, if you run domTest.js again, you’ll get the listing we expected from document.output():

  <testElement />

Let’s take a closer look at what’s happened within the xmldom Global that we’re using to model our DOM (I’ve shown the changes/additions in italics):

{
  "docNameIndex": {
    "myDocument": 1
  },
  "documentCounter": 1,
  "dom": {
    "1": {
      "creationDate": 1360572320268,
      "docName": "myDocument",
      "documentElement": 2,
      "node": {
        "1": {
          "firstChild": 2,
          "lastChild": 2,
          "nodeType": 9
        },
        "2": {
          "nodeName": "testElement",
          "nodeType": 1,
          "parent": 1
        }
      },
      "nodeCounter": 2,
      "nodeNameIndex": {
        "testElement": {
          "2": ""
        }
      },
      "nodeTypeIndex": {
        "1": {
          "2": ""
        }
      }
    }
  }
}

So, the appendChild() method created firstChild and lastChild pointers from the Document_Node (Node #1) to our new testElement Node (#2), and created a parent pointer from the testElement Node back to the Document_Node.

I won’t go into the detail of how appendChild() works, but you can take a look in the ewdDOM.js module’s source code, search for the appendChild() function and see if you can figure it out.  You’ll see that it’s more complex than you might think, because it has to determine what, if any, pointers are already in place for the object and target Nodes, and adjust them appropriately to create the effect of adding the new Node as the last child Node of the target parent Node.

The other function worth examining is the document.output() method.  You’ll find that it’s mainly a fairly simple recursive loop that takes place between two functions: outputNode() and outputChildren().  It starts with the Document_Node and recursively follows the firstChild and nextSibling pointer Global Nodes.  It then uses the Node properties such as nodeName to display the Tags in standard XML format.

Finally, let’s fill out our example to add an attribute to the testElement tag, and then add a child tag to it that has a number of attributes and a text value:

 
var document = dom.createDocument('myDocument');
var node1 = document.createElement('testElement');
var documentNode = document.getDocumentNode();
documentNode.appendChild(node1);

node1.setAttribute("name","rob");
var node2 = node1.addElement({
  tagName: "div",
  attributes: {id: "myNewNode", abc: 123, def: "this is cool!"},
  text: "This is a new div"
});

console.log("\r\nHow myDocument is physically represented in the Mumps Global:\r\n");
console.log(JSON.stringify(xmldom._getDocument(), null, 2));
console.log("\r\nXML Document Listing:\r\n");
document.output();

Note that I’m using a useful “macro” function called addElement()  which, in one single function, creates the new tag, adds some attributes to it, adds a text node to it and attaches it as a child tag of our testElement tag.  Under the covers, it’s using all the basic DOM API methods, but it just saves us a lot of time and function calls to use this one higher-level function.

Save and run this version of domTest.js and you’ll immediately see that the contents of the xmldom Global is now getting pretty big and complex, with lots of Nodes of various types and pointers created all over the place.  See if you can figure them out!

You should see that the document.output() listing, ie the result of walking all those DOM Nodes that are stored in the xmldom Global now looks like this:

<testElement name=’rob’>
  <div abc=’123′ def=’this is cool!’ id=’myNewNode’>
    This is a new div
  </div>
</testElement>

So, our XML document is starting to come to life.  By using more instances of the addElement() method, you can quickly build a large and complex XML document, entirely programmatically.  Try some experiments and see how you get on.  All the DOM APIs are documented in the Readme file for ewdDOM (or just view it in the GitHub repository) along with some more examples to get you started.

Remember that the difference between this implementation of the XML DOM and others you’ll perhaps be familiar with, is that this version is persistent.  If you remove that line we added in Part 1:

  xmldom._delete();

Then you’ll find that any XML DOMs you create will exist for as long as you like.  To re-use one later, just establish a pointer to it using an appropriate method.  For example, if you know the name of the Document you want to work with, use the getDocument() method:

var document = dom.getDocument('myDocument');

If you want to discover what documents you have, you can use:

var docArray = dom.getDocuments();

which returns an array of Documents.

One you’ve got the pointer to the document you want, you can pick up where you last left off and continue modifying or analysing it using the DOM API methods in ewdDOM.

Note that both of these methods above make use of the docNameIndex DOM Global Nodes to provide pointers to the relevant Documents.  getDocument() restricts itself to ones that match the name you’ve specified, whilst getDocuments() exhaustively runs through the entire index.

Mumps Global Storage in Action

So let’s recap and summarise.

What we’ve seen in Parts 1 and 2 of this article is the way in which you, the developer, can take complete control over Mumps Global Storage and use it in precisely the way you want to use it to model the system you want to bring to life.

The example I’ve used is the XML DOM, which is just one possible way you could use Global Storage.  Indeed, there are any number of different ways I could have used Global Storage to model the XML DOM.  For example, I could have used different physical Globals for the main properties of the DOM and the indices.  I also could have used different string names to denote the properties (eg ‘fc’ instead of ‘firstChild’).  The bottom line is that in a Mumps database, it’s up to me, the developer, to determine what the database is going to look like, how it’s going to be indexed and how it’s going to be searched and accessed.  Yes, it means I have to do all the work, but on the other hand, I’m never shoe-horned into working a particular way and never painted into a corner where I’m trying to do something the DBMS doesn’t want me to do.

The other thing to note in my XML DOM example is the way I’ve built the Javascript logic.  Essentially I first wrote the primitive APIs that do the basic chores of creating, modifying and removing Nodes, attaching and detaching them, creating and removing attributes etc.  I designed them and tested them in isolation, making sure they worked in all possible scenarios.  Everything was then, as far as possible, bootstrapped on top of those primitive functions, so that I quickly reached a point where my logic was only working with higher-level functions rather than physically dealing with the Global Nodes themselves.

This is a fairly typical scenario in Mumps application development.  Indeed, if you look at an application such as the Dept of Veteran’s Affairs Electronic Healthcare Record called VistA, you discover that almost all development is done using what are known as the FileMan APIs. These are methods and procedures that work at a high level of abstraction and hide from the developer the gory details of how the underlying physical Mumps Globals are used in VistA to represent patient healthcare information.

One final note is that it’s impossible to shoehorn any Mumps database you create into any one classic or NoSQL database category.  Consider our XML DOM example: it could be viewed in many different ways, all at once, depending on how we wanted to view it:

  • when we viewed the xmldom Global from within the GT.M shell, we saw it visualised as a Hierarchical Database
  • when we used the _getDocument() method, we saw it visualised as a Document Database
  • the low-level OO methods that are used within ewdDOM treat it as a Native JSON Database
  • the representation of the XML DOM within ewdDOM in terms of Nodes inter-connected by pointers used the Mumps database as a Graph Database
  • when the ewdDOM application is used, it’s behaving as a Native XML Database.

I think this is a pretty fascinating and powerful aspect of the Mumps database technology: that its behaviour depends on how you want to use it at the time, and that the same physical database can be abstracted in many different ways, all at once.  In this respect, it truly is a Universal NoSQL database technology.  As such, it’s a great learning tool for discovering what makes databases tick, but always remember that it’s also a rock-solid, super-fast, tried and tested, industrial-strength database.

Now that it’s easily accessible from Node.js, I believe that it deserves to stop being overlooked as an extremely powerful and flexible database for use in modern applications.  Hopefully my blog articles will help in this respect.

Oh, and did I already mention, it’s great fun to play with!

Advertisements

3 comments

  1. Many thanks Rob.
    A powerful illustration of the low-level control your approach offers for database development, exposing the “swiss-knife” like quality of Mumps, as per your Universal NoSQL Engine paper.

    Dare I ask how one might leverage this technology to support a relational database? ..( if one so chose.)
    I’m not suggesting that’s the best use of the tool of course, just that it may help some folk understand the power on offer here.
    I understand both GtM and Cache offer SQL projections of their underpinning databases, so assume there is a way to do this..

    Perhaps that’s another days discussion..

    Thanks again

    Tony

    1. Tony

      Good question. Actually a relational view can be layered on top of any set of Global Nodes – if you think about it, you can view any level within a Mumps Global Node hierarchy as a table with n primary keys and columns === properties:

      table(key1, key2, key3, property) = value

      Indices provide rapid ways to reach the appropriate table records – it’s the job of the query optimiser to figure out what is/are the most efficient indices to use to reach the required rows and columns.

      Products such as KBSQL and Cache SQL essentially allow relational table mappings to be defined against Globals – SQL queries are then converted to the appropriate gets, sets and forEachs that traverse, access and manipulate the physical Globals.

      So, in summary – yes – you can put a relational view on top of any set of Global Nodes, along with all the other projections I’ve mentioned in these articles.

      Rob

      1. Great, thanks Rob.

        I assumed as much and glad to have that confirmed.

        Which suggests this technology can be explained as a universal NoSQL/SQL engine?
        A snippet of code to illustrate the SQL capabilities would go down well sometime..

        Naturally some folk might wonder why work so low level, though as a database novice>guru I sense there is added educational value here w.r.t explaining..
        -NoSQL/SQL database technology
        -JSON & XML structures
        ..which seems to be a very good thing

        thanks

        Tony

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: