A 21st Century Architecture for the VA: Just Do It!

I’ve been helping customers implement high-end, highly-scalable web application architectures since the mid-1990s.  If there’s one thing I’ve learned, it’s that the key to a successful, cost-effective and maintainable architecture is to keep it simple and keep the number of moving parts to a minimum.  Oddly, that seems to not be a lesson learnt by much of the rest of the IT industry, who, in my opinion, appear to go out of their way to make web architectures as complex as possible and with a bewildering accumulation of layers and technologies.  Thankfully, many of these architectures fail to make it beyond the Powerpoint stage, but just dreaming up those Powerpoint slides seems to be good business for many.

People who know me know that I prefer to get on with stuff and do it, rather than just pontificating and talking about it, and that’s very much been my motivation for coming up with my EWD.js technology.  EWD.js, by the way, is the new, official name for what I’d previously called EWD Lite in earlier postings.

What I’d like to do in this posting is to outline the architecture that can be created at the US Department of Veterans Affairs (VA) by using EWD.js.  This is an architecture that can be literally created today, not in several years time, with almost no effort,  Rather than costing eye-watering sums of money, it can be created at almost no cost using existing Open Source products.  The architecture is simple, with almost no moving parts, yet that simplicity belies leading-edge capabilities, taking the VA into the 21st century and allowing the creation of their vision of mobile and web-based access and SOA-based services, whilst still supporting, maintaining compatibility with and making use of their existing VistA system.  The architecture will be fast, scalable, secure and extensible and will allow secure inter-operation across all 130 VA sites, “straight out of the box”.

Those of you who attended the recent VistA Expo in Seattle and stayed until the end will have seen this architecture and technology being demonstrated and put through its paces by me and Zach Gonzales from Oroville Hospital.  What I’m going to describe isn’t some theoretical grand-plan: this exists today, and is already in production use at Oroville where they use VistA as their Electronic Health Record (EHR).

Node.js

Let’s start with the key piece of technology that makes it possible: the Node.js interface that InterSystems added to their Caché database technology.  Caché, of course, is the back-end database technology that supports and runs the Mumps-based VistA application at the VA.  The Node.js interface is an officially-supported, built-in part of Caché and has been available since version 2012.2.  In fact, the interface file (cache.node) can be transferred to any earlier version of Caché and will just work.  The interface is rather unusual for a Node.js database interface in that it provides an in-process interface: ie both Node.js and Caché run in the same physical process.  It is also unusual in that it provides synchronous APIs which turn out to run significantly faster than their asynchronous, non-blocking counterparts.

The key thing is that this interface is sitting waiting to be used in all the Caché systems used by the VA at no additional cost.

So what’s the significance of Node.js?  Node.js is taking the IT world by storm: it’s a server-side implementation of JavaScript that delivers blisteringly fast performance, massive scalability and allows just one language – JavaScript – to be used to describe both the front- and back-end of applications.  If, for some reason, you’ve not heard of Node.js, then you need to do yourself a favour and find out about it today.  I’ve written a number of articles about it already in this blog.

Key to the success and viability of this architecture in the post-OSEHRA world is the fact that an API-compatible, Open Source alternative to the InterSystems Node.js interface has been developed for GT.M.  This ensures that architectures built using EWD.js are compatible and applications written using EWD.js can run identically at both the VA and at sites using the Open Source version of VistA outside the VA,.  Critically, it allows two-way collaboration between the two communities: in other words this architecture will support and embody the spirit of OSEHRA, and supports the needs of the Open Source VistA community.  None of the other proposed architectures for the VA that I’ve seen appear to allow this.

The diagram below summarises how these two interfaces connect Node.js to Caché and GT.M.  The key thing is that to Node.js, the two database technologies appear to be identical.

Nodejs-Cache-GTM-Interface

JSON interfacing of VistA

The combination of EWD.js, Node.js and these interfaces to Caché and GT.M allows something very powerful to be created: a completely JSON-based interface to VistA.  This can be done simultaneously at two levels:

  • Caché and GT.M appear to be JSON persistence engines, in effect turning them into a class of NoSQL database known as a Document Database, providing behaviour similar to databases such as MongoDB.  Both new and existing data, physically stored in these databases as what are known as Globals, are automatically mapped to and from an equivalent JSON representation within Node.js.
  • Existing Mumps programming logic can be invoked from within Node.js, with I/O represented in terms of JSON objects.  In other words, legacy VistA logic can be repurposed very quickly and easily, usually without any invasive changes being required, so that they appear to be JavaScript functions that return JSON from VistA or that save JSON data into VistA.

Client/Server in the Browser

One of the most exciting developments in the HTML5 stack has been the introduction of a technology known as WebSockets.  WebSockets create a bi-directional persistent TCP socket connection between a browser and the back-end.  WebSockets are fully supported in all modern browsers and have mature support in Node.js.  With JavaScript running in both the browser and the Node.js back-end, we have an event-driven capability at each end: not only can the browser send messages to the back-end at any time in response to events in the browser, the back-end can do the same in response to events occurring at the back-end.  Furthermore, the messages sent over the WebSockets can be JSON payloads.

This changes everything: we can now build and run client/server applications in the browser and we are no longer limited by the HTTP protocol.  The browser no longer needs to poll to check for changes at the back-end: when things change at the back-end, it can send a message to all relevant browsers, and receipt of that message can trigger an appropriate change in the browser’s User Interface (UI).

Anyone who has seen the astonishing things that are now possible in a browser via the new generation of JavaScript and CSS frameworks will realise that, when harnessed to a WebSocket-based architecture, browsers will allow the development of applications that were hitherto the domain of CPRS.  Running such applications in a browser means these applications are no longer restricted to desktop environments, but can be deployed to tablets and phones too.  The new generation of responsive frameworks such as Bootstrap means that such applications can be written once and will automatically adjust themselves to work across all platforms.

100% JSON

Put these technology building-blocks together and you create an exceptionally potent framework where applications are defined in terms of JSON messages flowing between the browser and back-end in response to events at each end, and furthermore the database is accessed by getting and setting JSON objects.  There’s just a single language: JavaScript, and a single data representation: JSON, front to back.

This is what EWD.js is all about: creating this 100% JSON environment.

Creating an EWD.js environment on top of VistA

Perhaps the most surprising thing is how quickly and simply this can be enabled on a VistA system running at a VA center.

  1. The Node.js interface is already in place if the VistA system is running on Caché 2012.2 or later.  If not, it can be enabled simply by copying the interface file from a more recent Caché build and putting into the right directory on the older Caché build.
  2. Node.js needs installing on the server(s) that are running Caché.  This is a quick and simple process.  Note that Node.js was recently approved for use by the VA.
  3. EWD.js is enabled by installing an Apache 2-licensed Node.js module called ewdgateway2.

In the near future, I’m hoping that a growing library of JSON interfaces to VistA will become available as a Node.js module.  Zach Gonzales is already building out an initial set of such interfaces at Oroville Hospital that could potentially form the basis of a community-wide module.

That’s all there is to it!

Now I’d like to explore the architecture that EWD.js creates and how it can be used to support the VA’s operations.

 

EWD.js Architecture

The diagram below summarises the architecture created by EWD.js

ewdjs-cache-gtm-architecture

At the core is a single Node.js process that provides the main web server and WebSocket interface with which browsers communicate.  Incoming JSON-based WebSocket messages from browsers are placed onto a queue which is then immediately processed by a dispatcher.  When the EWD.js module is started, a pool of persistent Node.js child processes is also started: each of these child processes binds to an instance of Caché (or GT.M) via the in-process Node.js interface.  The dispatcher passes a message from the queue to a child process in the pool, and as soon as this happens, the child process is removed from the free pool.  Child processes therefore only handle a single JSON message at a time.  Handling of the message is carried out by a Node.js module written by the application’s developer.  Typically this handler will access the Caché  (or GT.M) database, perhaps saving the JSON message, running a JSON-interfaced VistA function, and/or getting data in JSON format from the VistA system.  Typically the handler will return a JSON message back to the master Node.js process which forwards it to the browser that originally sent in its message.  Having completed its processing of the incoming message, the child process then returns itself to the free pool, ready to handle another message.

There’s a key reason for this queue/child process architecture: by ensuring that a child process handles just one incoming request at a time, the programmer can make use of those high-performance synchronous APIs that are provided by the Node.js interface to Cache and GT.M.  By using the synchronous interfaces, there’s a further and very significant benefit: the application developer doesn’t have to tie himself or herself in knots writing the almost unmaintainable asynchronous logic that is otherwise required when handling databases in Node.js (trust me, I’ve been there, done it and it’s not pretty or straightforward to write anything other than trivially simple logic using asynchronous logic, and none of the pseudo-synchronous syntax alternatives – eg Promises – make life as simple and intuitive as synchronous logic).

The system administrator can configure EWD.js to determine the number of child processes to spawn at startup.  Child processes can be dynamically added or removed if necessary without having to stop and restart EWD.js.

 

Anatomy of an EWD.js Application

As illustrated below, a developer of an EWD.js application creates 3 things:

  • A text file file containing the markup, CSS and/or JavaScript that defines the entire UI that will be used for the application.  Any JavaScript or CSS framework can be used with EWD.js.  At Oroville Hospital we’re finding that Bootstrap is a great framework, but you can use YUI, Sencha Touch, ExtJS, jQuery, Dojo, etc etc
  • A text file containing the JavaScript that will a) handle any in-browser events;  b) send associated JSON WebSocket messages to the EWD.js WebSockets interface;  c) handle any incoming WebSocket messages from the back-end; and d) modify the UI accordingly
  • A text file containing the JavaScript module that will run in a back-end child process, handling incoming messages from the browser, accessing the database (in this case via the JavaScript/JSON VistA interface functions), and/or send messages back to the browser.

 

ewdjs-Dev1

An EWD.js-based VistA application is largely defined as two sets of Event Handlers:

  • one set that responds to in-browser events, creates a JSON payload and sends it to the back-end
  • a second set that runs in the back-end, responds to incoming messages from browsers and integrates with the  VistA system via JavaScript functions that provide a JSON interface to the VistA system.  The back-end Event Handlers will often (though not always) send JSON messages back to the browser.

All interaction between the browser and back-end is via WebSockets, and all messages are JSON-based.

Experience at Oroville Hospital is proving that EWD.js applications are extremely quick and easy to develop, and provided that the JavaScript/JSON interface functions exist for accessing VistA, any JavaScript developer can be productive and developing new VistA browser-based client/server applications in a matter of a couple of days: essentially as long as it takes to learn how to use EWD.js.

Integrating VA systems via EWD.js

In addition to supporting browser-based applications, EWD.js has another trick up its sleeve.  Any back-end module can be automatically exposed as a JSON-based web service (the request is a simple HTTPS request, the response is an application/json mime type).   The WebServer functionality within the master Node.js process provides the HTTPS server interface for incoming Web Service requests.  Furthermore, in order to prevent unauthorised access to an EWD.js system, all Web Service requests must be digitally signed.  EWD.js automatically uses the same HMAC-SHA256-based security that is used by Amazon Web Services for its HTTPS-based services (eg S3, SimpleDB).

Just as in Amazon Web Services, a user of an EWD.js web service must be registered within the EWD.js system with a unique Id and Secret Key.  The Secret Key is known to both the client user (or system) and the EWD.js server but nobody else.  In order to invoke a Web Service, the client user must digitally sign the HTTP request using their Secret Key.  On receipt of a Web Service request, the EWD.js system automatically looks up the user’s Id that is in the incoming request and tries signing the incoming request using the user’s Secret Key.  Only if the incoming siganture matches the calculated one will the request be accepted.

Each Web Service is written as a JavaScript function within a module file that runs in an EWD.js child process.  It therefore has access to any of the JavaScript functions that access VistA via JSON.  The JSON object returned by the function is dispatched to the Web Service client as an HTTPS response.  Note that the Web Service author doesn’t need to concern himself/herself with the digital signature mechanics: EWD.js does it all automatically.

 

EWD.js as a Web Service Client

An EWD.js back-end module that executes in an EWD.js child process can act as a Web Service client, making calls out to any externally-available Web Service.  Node.js provides a built-in HTTP(S) client for this purpose.

Of course, the Web Services accessed by and EWD.js child process can be provided by another EWD.js system.  Simply by installing a pre-built EWD.js/Node.js Web Service Client module, your EWD.js applications on one VistA system can be accessing services on another VistA system: all the digital signature mechanics required by the client application are automatically handled by the Web Service Client Module.  All that is needed for one VistA system to securely communicate with another VistA system in this way is for each VistA system to know the Id and corresponding Secret Key of the other VistA system.  This is summarised diagrammatically below:

ewdgateway2-networked

Remember that this inter-linking of VistA systems uses the same security architecture as used by Amazon Web Services, who run their services over the open Internet and have done so for several years without any apparent problem.  I see no reason why all 130 VA systems couldn’t be interconnected using this extremely lightweight but secure Web Service architecture.

As an aside, we used this same secure Web Service mechanism in the Oroville Hospital Scheduler in order to demonstrate scheduling between VA centres.

 

External Message Generators

EWD.js has one further trick up its sleeve.  The master Node.js process has a built-in socket server interface, listening on a pre-determined port for JSON-structured strings.  Any process on the server (or, security considerations notwithstanding, other networked servers) can open this port and send a JSON-structured string to it.  Each EWD.js master process has a secret password assigned to it by the system administrator, and provided the incoming message includes this password, the master process will forward the message to one of three destinations:

  • all browsers currently connected to the EWD.js server
  • any browsers running a specified EWD.js application
  • any browsers whose back-end session data matches the specified values

This allows messages to be sent to specific users from, for example, monitoring equipment that is gathering patient-specific data.  Imagine, for example, a doctor being able to view a graph of such incoming information that is being updated in near-real time, displayed in the browser running on a tablet computer such as an iPad.

An Event-Driven Future

The Event-driven nature of JavaScript and Node.js makes them perfect technologies for the next generation of clinical and hospital applications, and the capabilities of browsers, driven by the demands of the games, social media and marketing industries, are sitting waiting to be exploited in Health IT.

By harnessing these technologies and making use of the Node.js interface to Caché and GT.M, EWD.js provides a powerful and extensible framework and architecture for the next generation of VistA applications.

The key thing is that everything I’ve described in this posting is available today and ready for use.  All the architectural components are free and Open Source.  Hospitals such as Oroville are using EWD.js already, and I predict that use of this technology and architecture outside the VA is going to take off and snowball.  It’s going to happen with VistA systems whether or not the VA wants to make use of it: this train has already left the station and won’t wait for anyone!  So my message to the VA is: if you want to join this ride, get on board!  Stop deliberating, just do it!

Alternatively, keep reading those Powerpoint slides, but don’t be surprised if you’re left behind!

 

Advertisements

6 comments

  1. Thanks Rob,
    I’m reading this as a guy with clinical/management and technical hats so here’s my take on what your saying here.. Please agree/disagree with my summary..

    1) You’re involved in this space as you have long had an interest in Mumps the database, which happens to power the VistA EHR and as you’ve said elsewhere is the Universal NoSQL Database.. i.e. ideal for healthcare.. Cache or GT.M .. proprietary or open .. you take your pick.

    2) You’ve also been involved in web application development for a long time so your expertise is in that space between NoSQL M the database and the Web that the rest of the world knows and loves. Your efforts are unashamedly aimed at bridging that gap..so you’re now commending javascript, json and node.js as key tools to create that join.. for the reasons you mention in this article and elsewhere..

    3) In blending the power of M with the world of the web, you’ve crafted EWD.js as the node library that brings these M and Web worlds together… ideal for a VistA context.. namely it could/should help with integrating VistA components together and into a web environment.. plus it provides the basis of a new Amazon Web Service like distributed architecture for 21st C VistA..

    4) In addition to that, EWD.js offers advantages in front end UI web development.. in terms of bringing the power of “thick clients” to the browser, plus an event driven approach.

    5) You’re outlining the current architecture of EWD.js, the typical parts of an EWD.js application and a recipe book for how to get it running in a VistA environment, so its open source and out in the wild.

    From my reading of the transatlantic efforts towards open source EHRs on both sides of the ocean, these recommendations fit with those areas of common ground … UI/Service Layer and Persistence that have international appeal for those involved in an OS EHR effort.
    http://frectal.com/2013/09/18/transatlantic_thoughts_onvista_nhs/

    I’m also guessing folks will ask ..does any other single approach offer these opportunities to the VA.. if so what is it called.. if not why aren’t they using it?
    Regardless as you’ve said ..the train has already left the station..

    thanks for the post
    Tony

  2. Thanks Tony

    Agree with your summary.

    As to your final question, I’ve seen many Powerpoint presentations by “experts” explaining alternatives but I haven’t seen any that a) don’t cost an arm and a leg, b) will take forever to implement, c) will require god knows how many different technical skill sets and d) therefore probably won’t happen! And most aren’t Open Source.

    As someone who, as you say, has been doing this stuff with Mumps systems since the mid-90s, I think if there was another truly viable, cross-platform, Open Source-compatible approach, I’d have come across it by now and be promoting it

    Rob

  3. Chris Casey · · Reply

    Rob,

    as you showed in Seattle just last week this is available and works NOW!
    It is simple, secure, reliable and fast.
    Why anyone would dream of trying to reinvent this wheel as a caterpillar track baffles me.

    Chris

  4. Hi Rob, well done !
    I would change the title to “A 21st Century Architecture NOT ONLY for the VA” 😉 I would be very happy to see my MEDILIG open source EHR project to get converted to your infrastructure. Wish you all the best for the future of your EWD technology.

  5. Steve Owen pointed me at this:

    http://www.va.gov/TRM/ToolPage.asp?tid=7046%5e

    So it seems that EWD.js has been quietly approved by the VA. All the more reason for the architecture I’m proposing to be considered properly instead of the usual nonsensical ideas I keep seeing popping up at conferences.

    Meanwhile, we still see stuff like this happening:

    http://www.nextgov.com/health/2013/11/va-awards-asm-research-health-it-contract-triple-price-competitors-bids/73223/?oref=nextgov_today_nl

    I’ve no idea how or why it should require $160m to create a new UI for VistA. Oh to have just a fraction of that kind of money and prove what would be possible! I only hope that AgileX et al take a serious look at EWD.js, particularly in the light of its recent approval.

    Rob

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: