JSON-interfacing VistA (and other legacy Mumps systems)

EWD.js is now providing the means by which legacy Mumps-based systems, such as VistA, can be enhanced and modernised by a new generation of JavaScript developers.

In this technical article, I’m going to explain how legacy Mumps code, such as core VistA/FileMan APIs, can be wrapped so that they become available to JavaScript developers.  Certainly as far a VistA is concerned, I believe this is a critical endeavour: the more core functions in VistA are exposed as JSON-interfaced JavaScript functions, the more easily and extensively can VistA be modernised.  The good news is that the work of exposing the core Mumps-based functionality of VistA can almost entirely be carried out by the same Mumps developers who understand how that functionality works and should be used, and also understand the nuances that inevitably accompany such extensive legacy code.

Of course, this kind of exercise isn’t restricted to VistA: what I’m going to explain in this article applies to any legacy Mumps-based applications.

Mapping between JavaScript Objects and Arrays and Mumps Globals

The key to understanding how legacy Mumps code can be exposed as JSON-interfaced JavaScript functions is understanding how the two document-handling APIs work in EWD.js: _getDocument() and _setDocument().  These automatically map between a JavaScript object and a corresponding representation of the object held in a sub-tree of nodes within a Mumps Global.  Mumps Globals are, of course, the unit of storage within a Mumps database.

Essentially, any name/value pair within a JavaScript object is represented as a subscript/value pair within a Global, where the subscript name matches the name of the object’s name/value pair.  The Global into which the JavaScript object is mapped is determined by the GlobalNode object whose _getDocument() or _setDocument() methods are invoked.

So, take this simple object:

var myObject = {
  firstName: 'Rob',
  lastName: 'Tweed',
  city: 'Reigate',
  country: 'UK'
};

We’ll set a pointer to a Global named ^demographics, and let’s assume the first subscript of this global represents a person’s unique Id, and let’s assume my Id is 123456. We can map the object above into this Global by doing the following:

var demographics = new ewd.mumps.GlobalNode('demographics', [123456]);
demographics._setDocument(myObject);

Assuming that the ^demographics Global was originally empty, it will now look like this:

 ^demographics(123456,"firstName")="Rob"
 ^demographics(123456,"lastName")="Tweed"
 ^demographics(123456,"city")="Reigate"
 ^demographics(123456,"country")="UK"

Hopefully this is a pretty straightforward and intuitive mapping that requires no further explanation.

Of course it’s a reversible process. Applying the _getDocument() method to the demographics object will return the original JavaScript object:

var demographics = new ewd.mumps.GlobalNode('demographics', [123456]);
var myObject = demographics._getDocument();

If the JavaScript object is a more complex one, for example an object of objects, then the outer sub-object names are mapped to intermediate subscripts within the Global. Only the “leaf” name/value pairs are mapped to “leaf” Global nodes with an associated value.

For example, let’s modify that JavaScript object, putting the address fields inside an address sub-object:

var myObject = {
  firstName: 'Rob',
  lastName: 'Tweed',
  address: {
    city: 'Reigate',
    county: 'Surrey',
    country: 'UK'
  }
};

Applying the _setDocument() method will now create the following Global Nodes:

 ^demographics(123456,"firstName")="Rob"
 ^demographics(123456,"lastName")="Tweed"
 ^demographics(123456,"address","city")="Reigate"
 ^demographics(123456,"address","county")="Surrey"
 ^demographics(123456,"address","country")="UK"

This simple rule can be applied to any depth of sub-object nesting, eg:

var myObject = {
  objects: {
    can: {
      be: {
        nested: {
          to: {
            any: 'depth'
          }
        }
      }
    }
  }
};

will map to:

 ^demographics(123456,"objects","can","be","nested","to","any")="depth"

That’s hopefully pretty straightforward and intuitive for objects, but what about arrays? JavaScript objects can contain arrays, and JavaScript arrays can contain objects, nested, in turn, to whatever depth is required.

When an array is mapped to a Global, subscript values are assigned that match the member number of each element in the array, and the element’s value is assigned to the Global node’s data value. So, if we have a simple array:

var myArray = ['Rob', 'Simon', 'Helen'];

Applying the _setDocument() method will now create the following Global Nodes:

 ^demographics(123456,0)="Rob"
 ^demographics(123456,1)="Simon"
 ^demographics(123456,2)="Helen"

When the _getDocument() method is invoked, it determines that a particular level of subscripting denotes an array rather than an object by checking whether the first subscript value is zero and that each subsequent subscript increments the previous subscript by one. So an unbroken numeric sequence of subscript values, starting at zero, will be mapped to an array; otherwise the subscripts are mapped to an object. So, for example, applying the _getDocument() method to the above Global nodes will recreate the original array, but applying it to the following non-consecutive sequence of subscripts:

 ^demographics(123456,0)="Rob"
 ^demographics(123456,2)="Simon"
 ^demographics(123456,5)="Helen"

will map to an object:

  {
    '0': 'Rob',
    '2': 'Simon',
    '5': 'Helen'
  }

Arrays can be combined inside objects, for example:

var myObject = {
  firstName: 'Rob',
  lastName: 'Tweed',
  address: {
    city: 'Reigate',
    county: 'Surrey',
    country: 'UK'
  },
  children: ['Simon', 'Helen']
};

Applying the _setDocument() method to this object would create the following Global nodes:

 ^demographics(123456,"firstName")="Rob"
 ^demographics(123456,"lastName")="Tweed"
 ^demographics(123456,"address","city")="Reigate"
 ^demographics(123456,"address","county")="Surrey"
 ^demographics(123456,"address","country")="UK"
 ^demographics(123456,"children",0)="Simon"
 ^demographics(123456,"children",1)="Helen"

Arrays can, in turn, contain objects, and the mapping simply follows and extends the previous logic. For example:

var myObject = {
  firstName: 'Rob',
  lastName: 'Tweed',
  address: {
    city: 'Reigate',
    county: 'Surrey',
    country: 'UK'
  },
  children: ['Simon', 'Helen'],
  bicycles: [
    {brand: 'Trek', model: 'FX3', type: 'hybrid'},
    {brand: 'Trek', model: 'Madone 4.5', type: 'road'},
    {brand: 'Cannondale', model: 'SuperSix', type: 'road'},
  ]
};

This would map to:

 ^demographics(123456,"firstName")="Rob"
 ^demographics(123456,"lastName")="Tweed"
 ^demographics(123456,"address","city")="Reigate"
 ^demographics(123456,"address","county")="Surrey"
 ^demographics(123456,"address","country")="UK"
 ^demographics(123456,"children",0)="Simon"
 ^demographics(123456,"children",1)="Helen"
 ^demographics(123456,"bicycles",0,"brand")="Trek"
 ^demographics(123456,"bicycles",0,"model")="FX3"
 ^demographics(123456,"bicycles",0,"type")="hybrid"
 ^demographics(123456,"bicycles",1,"brand")="Trek"
 ^demographics(123456,"bicycles",1,"model")="Madone 4.5"
 ^demographics(123456,"bicycles",1,"type")="road"
 ^demographics(123456,"bicycles",2,"brand")="Cannondale"
 ^demographics(123456,"bicycles",2,"model")="SuperSix"
 ^demographics(123456,"bicycles",2,"type")="road"

Any JavaScript object, no matter how complex a mixture of objects and arrays, can therefore be mapped to a corresponding set of Global nodes, and conversely, any Global structure, no matter how complex or deeply nested, can be converted to a corresponding JavaScript object.

Invoking Mumps code from within JavaScript

This bi-directional mapping between JavaScript objects and Mumps Globals provides the first key to interfacing Mumps code for use by JavaScript developers.  The second is the ability to invoke Mumps functions from within JavaScript, eg:

  
  var result = ewd.mumps.function('myMethod^someRoutine', arg1, arg2);

Although this would appear to provide everything needed, there are some very specific limitations to the function() method:

  • arguments can only be simple variables.  Objects and arrays cannot be passed as arguments into a Mumps function
  • in Caché, the maximum length of the result value is limited to 4k: insufficient for large, complex objects to be returned
  • the returned result can only be a string value
  • neither Mumps procedures nor Caché Class Methods can be invoked directly using the function() method.  Only Mumps “extrinsic functions” can be directly invoked.

Despite these apparent limitations, it is actually very simple to wrap any existing Mumps logic in a way that can be invoked from within JavaScript, and so that a JavaScript object can be used to provide the inputs to the Mumps logic and so that a JavaScript object can be obtained as a result of invoking the Mumps logic.  It can be described as a simple “recipe book” that can be applied to any Mumps code.

1) Wrap the Mumps code that you want to invoke inside a simple extrinsic function.  

This Mumps function should have no inputs and should return a string result value.  This value should be a null string if the function completes successfully, or a non-null string value that indicates and describes any error that might have occurred.  For example:

  
  getProblems() ;
   new result
   do some^legacyCode(arg1,arg2)
   set result=""
   QUIT result

For the purposes of this example, let’s assume this function has been added to a Mumps routine named ^vistAAPIs (or GT.M routine file named vistAAPIs.m).

2) Edit the wrapper function to obtain all its input arguments from a temporary Global.

Use any Global name that you want, provided its name doesn’t clash with any other Global used within your system.  Always subscript this Global with the process Id (indicated by the $j variable). How you structure the temporary Global is up to you, but I’d recommend something like the following example:

  
  getProblems() ;
   new arg1,arg2,result
   set arg1=$g(^TMP($j,"inputs","arg1"))
   set arg2=$g(^TMP($j,"inputs","arg2"))
   do some^legacyCode(arg1,arg2)
   set result=""
   QUIT result

3) Merge or map the outputs from the legacy code into a temporary Global.

Exactly how you do this will depend on the legacy code and how it returns its results. Many VistA FileMan APIs return their results in a temporary Global named ^TMP: if this is the case, then you don’t need to do anything further as far as mapping the outputs is concerned, but you will need to understand how the outputs are structured within the temporary Global created by VistA/FileMan.

Other legacy code might return the results as an array, or as a set of variables that are left behind in the symbol table. Modern, well-behaved Mumps code might return the results as a returnValue string or via one or more arrays that are called by reference.

Whatever the mechanism used by the legacy Mumps code, merge it or map it into a temporary Global. Once again the structure of the temporary Global is up to you, but make sure it is subscripted using the $j variable, for example:

  
  getProblems() ;
   new arg1,arg2,outputs,result
   set arg1=$g(^TMP($j,"inputs","arg1"))
   set arg2=$g(^TMP($j,"inputs","arg2"))
   do some^legacyCode(arg1,arg2)
   ; assume this left behind its results in an array named outputs
   kill ^TMP($j,"outputs") ; make sure it's empty
   merge ^TMP($j,"outputs")=outputs
   set result=""
   QUIT result

4) Clean up the symbol table to make sure nothing leaks out of the function wrapper.

Legacy Mumps code is notoriously bad in terms of badly or non-existent variable scoping, so it tends to leak variables into the symbol table. You should identify any such leaking variables by running your function wrapper in isolation from within a Mumps terminal shell and seeing what gets left behind in your process’s symbol table. Use the New command to prevent them leaking out of your function wrapper, eg:

  
getProblems() ;
new arg1,arg2,outputs,result
;
new DT,DUZ,DVARX ; mop up any leftover leaking variables
;
set arg1=$g(^TMP($j,"inputs","arg1"))
set arg2=$g(^TMP($j,"inputs","arg2"))
do some^legacyCode(arg1,arg2)
; assume this left behind its results in an array named outputs
kill ^TMP($j,"outputs") ; make sure it's empty
merge ^TMP($j,"outputs")=outputs
set result=""
QUIT result

What you’ve created is a wrapper around the legacy code that obtains all its inputs from a temporary Global, returns all its outputs via a temporary Global, and leaves nothing behind in the Mumps process’s symbol table.

That’s it: the legacy code is ready for use from within JavaScript.

Invoking the wrapped legacy Mumps Code from JavaScript

The way in which your legacy Mumps code wrapper function is invoked from within JavaScript is now very straightforward:

1) Create a JavaScript object that holds all the input values needed by your Mumps function.  Make sure the structure of this object corresponds to the structure of the temporary Global used by your Mumps function.  For example:

  
var inputs = {
  arg1: 'valueOfArg1',
  arg2: 'valueOfArg2'
};

2) Map this object into the temporary Global using _setDocument(). Note that because the connection to the Mumps database from each Node.js Child Processes used by EWD.js is “in-process”, the processId of the Child Process is the same as the $j value within the Mumps environment. Here’s all you need to do:

  
var inputs = {
  arg1: 'valueOfArg1',
  arg2: 'valueOfArg2'
};
var temp = new ewd.mumps.GlobalNode('^TMP', [process.pid, 'inputs']);
temp._delete(); // ensure it's empty
temp._setDocument(inputs);

From what I explained earlier in this article, it should be clear that this has created the correct Global structure for the inputs, ie:

  
 ^TMP($j,"inputs","arg1")="valueOfArg1"
 ^TMP($j,"inputs","arg2")="valueOfArg2"

3) Now you can invoke the Mumps wrapper function. Note that no input arguments need to be specified in the function() method since all the inputs will be coming from the temporary Global:

  

var inputs = {
  arg1: 'valueOfArg1',
  arg2: 'valueOfArg2'
};
var temp = new ewd.mumps.GlobalNode('^TMP', [process.pid, 'inputs']);
temp._delete(); // ensure it's empty
temp._setDocument(inputs);

var error = ewd.mumps.function('getProblems^vistAAPIs');

Assuming that no errors were indicated (ie if result is a non-null string), we can now extract the results as a JavaScript object from the temporary output Global:

  

var inputs = {
  arg1: 'valueOfArg1',
  arg2: 'valueOfArg2'
};
var temp = new ewd.mumps.GlobalNode('^TMP', [process.pid]);
var inputs = temp.$('inputs');  // point to the "inputs" nodes within the Global
inputs._delete(); // ensure it's empty
inputs._setDocument(inputs);
var error = ewd.mumps.function('getProblems^vistAAPIs');

if (error === '') {
  var outputs = temp.$('outputs');
  var results = outputs._getDocument();
}
else {
  // report back the value of error
}
temp._delete(); // clear down the temporary Global now we're finished with it

Note that it’s a good idea to clear down the temporary Global once we’ve finished with it.

The structure of the JavaScript object named results will, of course, be directly derived from the structure of the temporary Mumps Global that contained the results, based on the rules I explained at the start of this article.

That’s it: we now have a legacy Mumps function being invoked from within JavaScript, and both its inputs and outputs being handled as JavaScript objects.

Any legacy Mumps code can be wrapped in a similar way.  It doesn’t matter how complex the input or output structures, by using the trick of mapping via a temporary Global, we can seamlessly merge the JSON-centric JavaScript world with the Global/Multi-dimensional array-centric Mumps world.

A Request to VistA FileMan Experts

If we’re going to be able to modernise VistA, using EWD.js to move it into a browser-based world, we need to expose the business functionality of VistA through such JavaScript functions.  What I’ve created for you is a means of doing this entirely from within the environment you know, understand and are most comfortable with: the Mumps programming environment.  My request is that those of you who understand the very many FileMan APIs and other APIs and RPCs within VistA, start producing wrapper functions in the way I’ve described.  It’s a very quick and simple process as I’ve demonstrated.

The more of VistA’s functionality that is wrapped in this way, the more VistA becomes accessible to a new generation of JavaScript developers.  Your work will ensure the continuation of the amazing and extensive capabilities of VistA, making all that power and functionality accessible to a new generation.

I realise that the sheer breadth and scope of what VistA does makes this a large task, and it’s going to require the skills and knowledge of many VistA developers to make it happen.  However, I’ve endeavoured to make it as quick and as simple as possible: it’s now over to you VistA experts to make it happen.

 

Advertisements

6 comments

  1. Georgios Kiriazopoulos · · Reply

    Hello Rob,

    Thanks for this fantastic “JavaScript for mumpsters” tutorial. I find your Invitation to write VistA functions for use in EWD especially interesting. I don´t consider myself expert in VistA, but I am confident that I could contribute useful code according your specifications in some areas of Kernel and Fileman. In this context, I think that it would be very usefull to have a thematic guide suggesting some areas of Interest, priorities etc. I am currently resting after a large surgery operation I resently had, but hope to be soon back to my normal Life and make my plans about this reality.

    Thanks

  2. Something I didn’t mention in the main article was the pioneering work of Zach Gonzales from Oroville Hospital. Zach has been building out a suite of VistA/FileMan APIs for Node.js, initially for use at Oroville. He’s been using the techniques I’ve described above, and demonstrated some of the applications that are being built around these APIs at the closing session of this year’s VistA Expo.

    I think that this whole area is ripe for community collaboration. Perhaps a shared Github repository should be set up for VistA/FIleMan APIs for EWD.js, so that people can see what’s already available, which APIs still need to be added, and so that experts can apply peer-review to already-written APIs.

    What I’d be anxious to avoid is constant “reinvention of the wheel”, with multiple versions of APIs being developed, each with slight differences and nuances within their behaviour. A shared Github repository, or similar initiative, would help prevent this. Perhaps something OSEHRA might consider sponsoring and supporting?

    Rob

  3. Roy Gaber · · Reply

    Rob, on hardhats you spoke about the possibility of Zach publishing his API’s; has there been any movement on that?

  4. Roy Gaber · · Reply

    Steve and I are looking into developing some services and would not want to re-invent the proverbial wheel.

  5. […] but it is described by Rob Tweed on his blog, The EWD Files, in JSON – Interfacing VistA (and Other Legacy MUMPS Systems). The idea is to use the fact that MUMPS allows strings as subscripts, and represent a JSON object […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: