VistA Evolution: What’s Wrong with this Picture?

The US Department of Veteran’s Affairs (VA) has embarked on an initiative known as VistA Evolution, the goal of which is to create a new generation of their Electronic Healthcare Record (EHR) that will be known as VistA4.  

The aims of this initiative are entirely laudable: to modernise the ageing, but much loved and very successful, VistA EHR, improve its functionality in areas such as scheduling and make it more accessible for others to build upon using modern tools.  The VA also recognises the importance of making VistA, or rather key parts of its functionality, accessible from the growing range of mobile devices.

The VA has begun awarding a number of very high-value contracts under the umbrella of the VistA Evolution initiative (eg to ASM Research/Accenture), but in my opinion there are problems looming on the horizon.  From what I understand about the direction that these projects are taking (with encouragement, it seems, from within the VA), there’s a real risk that we’ll see a repeat of previous attempts to modernise VistA, the result of which was very expensive failure with essentially nothing to show for it.  The losers, if this happens, are not only US tax-payers: it’s the Veterans whose future welfare depends on VistA4 being a success.

So, what’s wrong with the VistA Evolution picture that is beginning to emerge, and what do I suggest as an alternative approach that will learn from and avoid past mistakes? In order to answer these question, we first need a bit more background.

VistA, the VA’s EHR

At the heart of the issue is VistA.  VistA is the VA’s EHR that supports and manages every aspect of the healthcare provided to US Veterans.  The history of its development is steeped in controversy because VistA arose through the largely clandestine activities of a group of VA programmers and clinicians who worked together as what became known as the Underground Railroad.  The VA’s strategic approach at the time was to create a centrally-managed, mainframe-based system.  However, whilst the official VA efforts cost a great deal and achieved little, the unofficial efforts of the Underground Railroad, based on a decentralised network of mini-computers, proved highly successful, despite efforts by the VA’s IT management to shut down their work.  Eventually, in 1982, the VA capitulated and legitimised their work, naming the product the Decentralized Hospital Computer Program (DHCP).  In 1996 the name was changed to Veterans information system technology Architecture, abbreviated to VistA.

Mumps, the Technology on which VistA is Based

A critically important factor is the technology on which VistA is based.  The Massachusetts General Hospital Utility Multi-Programming System, otherwise known as Mumps or often abbreviated to M, was first developed around 1966-7 and was specifically designed with the requirements of healthcare in mind.  It is testimony to the design and thinking behind the Mumps technology that even to this day it dominates the healthcare sector, worldwide.  Almost all of the most successful EHRs are based on Mumps or a modern derivative. The problem with the Mumps technology is that is almost unknown in the wider mainstream of the IT industry, and the few that have actually heard of it generally have a very poor understanding of it.  This is due to a number of reasons:

  • it is both a database and a language, the language being integrated and designed for both the definition of business logic and also optimised for access to and management of the database.  By comparison, in the mainstream of IT, computer languages and databases are normally expected to be separate entities, so Mumps is wrongly regarded as something of a weird anachronism;
  • the Mumps language is considered to be old-fashioned to the extent of being obsolete by the IT mainstream.  This is hardly surprising, given that it was designed back in 1966-7.  Computer language design has evolved considerably since that time, and the current generation of programmers are not used to what they regard as inherent limitations and shortcomings within the Mumps language;
  • the hierarchical Mumps database was also regarded as something of an anachronism during the period between the 1970s and 2009 when relational databases (eg Oracle, SQL Server, MySQL etc) dominated the IT industry.  The emergence of the NoSQL movement in 2009 actually brought the unique design and capabilities of the Mumps database back into favour: seen in this context, it turned out to be an extremely powerful, adaptable, high-performance NoSQL database that predated the modern movement by some 3 decades.

In my opinion, the success of the Mumps technology within the healthcare sector is entirely due to these unique NoSQL characteristics of the Mumps database.  Unfortunately, within the IT industry (and, sadly, within the Mumps community itself), Mumps is considered to be first and foremost a language rather than a database.  As a result of the old-fashioned nature  of and perceived deficiencies in its language, it gets dismissed as an obsolete technology.  This is unfortunate, because if it was viewed as first and foremost a database, a very different conclusion would be reached.  The IT mainstream therefore throw a uniquely-capable database baby out with the Mumps language bathwater.

If they viewed Mumps as a database, they would quickly realise that the Mumps language is not mandatory and that it can be replaced with something modern and mainstream: something I’ve been focused on for some time.  For a number of reasons identified in my blog postings, it turns out that JavaScript is the ideal language with which to replace the Mumps language.  Furthermore, by abstracting the Mumps database as a persistent store of JavaScript Objects, Mumps becomes a very potent, modern technology.  These are the underlying strategic principles behind my EWD.js framework, an Open Source project that is available for use by anyone interested in modernising Mumps applications such as VistA.

EWD.js in a nutshell

EWD.js is an Apache 2-licensed, Open Source framework for building modern browser-based applications.  It is primarily designed to integrate with Mumps databases, and is ideal for modernising legacy Mumps applications.   Key factors are:

  • the design and function of EWD.js has been based my own extensive experience gained since the mid-1990s in web technologies and their integration with Cache and Mumps since the mid-90s.  In addition to my extensive knowledge of the Mumps technology, I’ve immersed myself into the web browser and JavaScript world, understanding the  mindset and thinking of that community.  I’ve also made it my business to watch and understand the significance of the many rapid developments in the browser and JavaScript marketplaces.
  • EWD.js is designed around what is known as a stateless architecture.  A stateless architecture is essential for providing the kind of high-end scalability and performance that is required to support browser-based, mobile and web service-based applications.  It is based on the recognition that, as far as a back-end server is concerned, for most of the time, users are not actually doing anything that requires its attention – they are looking at their screen, reading what’s on it and deciding what to do next.  A stateless architecture uses a small pool of processes that service the actual activity of those users as and when they need something to be done on the server.  It’s a highly efficient use of hardware resources, and makes it possible to support potentially huge numbers of users.
  • EWD.js makes use of the official Node.js interface that has been developed by InterSystems and included with their Cache database.  Furthermore, EWD.js includes an abstraction of the basic APIs provided by this Node.js interface’s to expose a Mumps database as if it was a set of persistent JavaScript objects.  EWD.js therefore makes a Mumps system meaningful to a JavaScript developer and avoids any need to understand the Mumps language or the need to think of Mumps as an old-fashioned hierarchical database.
  • EWD.js applications are different from “traditional” web applications that use a technology known as Ajax.  Instead, EWD.js makes use of a newer technology known as WebSockets and allows the creation of what, in effect, are client/server applications that run in a web browser.  For those familiar with CPRS, the current Graphical User Interface (GUI) for VistA, EWD.js allows the creation of application that have all the capabilities and power of the CPRS “thick client”, but with all the benefits of using a web browser instead of a thick client.  In particular that means no more issues related to installation and maintenance of those thick clients on each user’s computer, and the potential ability to provide secure, fast access over the internet instead being limited to access within the enterprise network.

The diagram below summarises how EWD.js integrates with VistA.  The three boxes coloured red indicate the areas where developers are required.  Everything else either just works or exists already:

EWDjs-VistA-Architecture

The Need for VistA Modernisation

VistA is highly regarded by its users – its functionality, in terms of both breadth and depth, equals and in many areas exceeds that of even the most successful and well-known commercial EHRs.  However, that VistA needs to be modernised is beyond question, and in two key ways:

  • Its user interface (UI) dates from the late 1970s and is designed around the technology of that time known variously as dumb terminals, green screen devices or roll and scroll.  A very successful and well-regarded graphical user interface (GUI) known as the Computerised Patient Record System (CPRS) was released in 1997.  However, CPRS uses a technology known as Delphi which is now effectively obsolete.  Additionally the style of technical architecture used by CPRS, known as client/server or thick client, has now been largely replaced within the IT industry by web-browser-based thin client approaches;
  • its highly-regarded core functionality needs to made more accessible in industry-standard ways, in particular via web services and as what are known as REST-ful services.  Doing so would open up VistA, allowing a new generation of developers to build new and modern interfaces and extend its functionality still further, in particular allowing VistA to exploit the exciting new opportunities offered by today’s growing range of mobile devices.

These two deficiencies are, in a nutshell, what VistA Evolution aims to address.

A Step Back into Recent History: The HealtheVet Initiative

VistA Evolution isn’t the first time at attempt has been made to modernise VistA.

In 2001, the VA began an initiative known as HealtheVet, the aim of which was broadly similar to today’s VistA Evolution initiative.  Technically, the intention of HealtheVet was to replace the Mumps database of VistA with the Oracle relational database, and to implement an overarching technical architecture that was based around the Java programming language and a set of Java-based services that is known collectively as Java 2 Platform, Enterprise Edition (J2EE), now known as Java EE.

In 2008, the HealtheVet initiative was abandoned after the expenditure of some $600m.  At that time it was estimated that the project would take until 2018 to complete, at a cost of $11b. One of the outcomes of this hugely expensive and highly politically embarrassing failure was the decision by Roger Baker, VA CIO (2009-13) to make a significant strategic change in direction and to move the VistA EHR into Open Source.

His rationale, ratified by Congress, was the recognition that VistA was functionally adequate and it made no sense to replace it outright, and furthermore to recognise that there was a growing community of developers outside the VA who had detailed technical knowledge of VistA and were already embarking on modernisation exercises, from whom the VA potentially stood to benefit.  As part of this new approach, Baker established OSEHRA, the custodial agent responsible for managing and coordinating the Open Sourced version of VistA.

Why Did HealtheVet Fail?

The official reasons behind the failure of the HealtheVet initiative focused on management deficiencies, particularly insufficient project management expertise and adequate milestones.  What tends to be overlooked is the technical side of the project, in particular the architectural complexity of the Java EE platform on which it was based.  This can be clearly seen in one of the official presentation slides that summarises the planned HealtheVet architecture.  This slide was created by one of the project’s technical team at the time:

VA_Java_Architecture

One interesting thing is that the complexity that is plain to see in this diagram is not actually unusual in a Java EE architecture.  The other interesting thing is that the Java EE platform has been very successfully implemented elsewhere and underpins many huge enterprise-scale IT operations around the world.  Perhaps the conclusions were right: with adequate management and that eye-watering sum of $11b, it could have been successful.

However, as clearly illustrated in the diagram above, with so many moving parts, the HealtheVet project was highly ambitious.  Trying to replace the Mumps-based VistA application with a Java/Oracle alternative in one fell swoop made it even more ambitious and risky.

Complexity is Problematic

Even if it had succeeded, there’s another consequence of all those moving parts and complexity: maintenance and ongoing development of the HealtheVet platform would have been extremely difficult, time-consuming and costly.  Like any IT platform, it would be fine whilst things worked successfully, but one thing you can guarantee about IT systems is that at some point, a problem will occur.  The more moving parts you have in the technical architecture, the more places there are for problems to occur and therefore the higher the probability that problems will occur.

Furthermore, it would have been unlikely that any one person would have been able to, hand on heart, say that they fully understood the workings of HealtheVet from end to end.  Much more likely, it would have required a number of specialist teams, each of which would have known about their specific portion of HealtheVet and how it interfaced with its immediately adjacent moving parts.  Whilst most of those teams would have been Java specialists, on the user-facing side of HealtheVet (the team who had designed and maintained the User Interface (UI)), the team(s) would have been specialists in HTML, CSS and JavaScript, and unlikely to be capable of easily conversing with the Java teams at a technical level.

So imagine a critical problem had arisen in some part of the HealtheVet architecture.  How would it have been tracked down and who would have been capable of such troubleshooting?  One look at that diagram above, and mere common sense should suggest that this would have been time-consuming and problematic at the very least.

Another thing you can guarantee about an IT application is that it is never cast in stone.  At some point it will need to be amended and/or extended.  Once again, the HealtheVet complexity would have made any modifications very difficult, time-consuming and expensive.  Multiple teams would have probably have been involved, and the consequences of even small changes would have potentially rippled down through those many Java EE layers.

Java EE: The Benchmark Architecture

All this having been said, there’s nothing intrinsically wrong with the Java EE Platform, and as I’ve indicated earlier, there are many large organisations around the world that run successfully and depend on the Java EE Platform.  In the IT industry it has been regarded as something of a benchmark architecture that offers demonstrable and proven scalability and reliability.  Performance, whilst perhaps not exceptional as a result of all those intermediate layers, is usually more than adequate, particularly when the price/performance of hardware keeps increasing in line with Moore’s Law.  As a result, it’s hardly surprising that Java EE is the IT architecture that tends to be recommended by the large management consultancies, and is the IT architecture for which most of the big contractors have lots of expertise.

In a nutshell, many, if not most, enterprises have come to accept the downsides of Java EE’s complexity against the benefits of its perceived scalability, reliability and ubiquity.  A case of “nobody ever got fired for recommending Java EE”.

The Times They Are A’Changing: Enter Node.js

Over the last three or four years, an interesting change in thinking has begun to take place amongst some of the largest users of the Java EE Platform.  Companies such as EBay, PayPal and Walmart, all of whom have relied on massive Java EE Platforms, have begun to replace them with a relatively new alternative technology called Node.js.  This is all summed up nicely in this Infographic.

Node.js is a server-side implementation of JavaScript, the language that grew up on and was previously was only found in web browsers.  Node.js takes the JavaScript language into the enterprise.  It also provides a significantly simpler alternative to the Java EE architecture, and yet it offers all the capabilities of a Java EE Platform.

The benefits that these large organisations are realising by moving to Node.js are nicely epitomised by PayPal’s experience.  For example:

“Node.js helps us .. by enabling both the browser and server applications to be written in JavaScript. It unifies our engineering specialties into one team which allows us to understand and react to our users’ needs at any level in the technology stack”

“The Node.js app was:

  • Built almost twice as fast with fewer people
  • Written in 33% fewer lines of code
  • Constructed with 40% fewer files”

“.. the Node.js application had:

  • Double the requests per second vs. the Java application. This is even more interesting because our initial performance results were using a single core for the node.js application compared to five cores in Java. We expect to increase this divide further.
  • 35% decrease in the average response time for the same page. This resulted in the pages being served 200msfaster— something users will definitely notice.”

Similar findings are documented by the other large enterprises that have begun evaluating Node.js and have found that it has revealed shortcomings in their established Java EE architectures that are now beginning to make them question its position as the reference standard enterprise IT architecture.

If you want the difference in complexity between a Java EE architecture and a Node.js-based architecture to be thrown into stark relief, you need look no further than the two diagrams above: compare the EWD.js / Node.js / VistA architecture diagram with the HealtheVet diagram.

So What’s Wrong with the VistA Evolution Picture?

Well, let’s first sum up these various threads I’ve outlined above:

  • The VA has already tried and spectacularly failed to implement a Java EE based architecture to underpin VistA modernisation
  • Even if the VA had been successful, it would have left a legacy of expensive, complex, risky and time-consuming maintenance and onward development
  • There’s a growing groundswell of opinion amongst some of the biggest and previously most conservative (in terms of IT) companies that Node.js is proving to be a better alternative to the Java EE Platform
  • It turns out that JavaScript is a perfect modern alternative language to replace the ageing Mumps language, and the exceptionally capable, powerful, performant and scalable Mumps database can be transformed into a Document Database, storing JavaScript Objects.  EWD.js is an Open Source framework, built on top of Node.js, that makes this all possible
  • EWD.js allows a legacy application such a VistA to be handed over to the huge and growing new generation of developers who understand JavaScript, and allows them, over time and in a low-risk, measured way, to replace the logic that is currently written in the Mumps language with equivalent logic written in JavaScript.  This contrasts with the high-risk alternative of a whole-scale replacement of VistA or parts of it with some alternative solution that was attempted in the failed HealtheVet strategy
  • EWD.js has emerged from the Open Source community that has been building around VistA – the very community from which Roger Baker wanted the VA to benefit.

What’s wrong with the VistA Evolution picture is that it looks, to me, very much like a repeat of HealtheVet:

  • a focus around Java and the Java EE Platform
  • a desire to rip and replace entire parts of VistA with Commercial Off the Shelf (COTS) products instead of building on and enhancing VistA’s own built-in capabilities

Why should such an approach to VistA Evolution succeed this time around when HealtheVet failed, no matter how much money and expertise was pumped into it?  Surely that lesson should have been learnt, or is it the case that there has been a whole-scale replacement of senior management within the VA since HealtheVet was canned, and so there’s nobody left who even remembers HealtheVet?

What’s wrong with the VistA Evolution picture is that the knowledge and expertise of the Open Source VistA Community is apparently being ignored, despite the VA’s supposed commitment to the Open Source-based strategic direction that was instituted by Roger Baker as a direct result of the HealtheVet failure, and despite having set up OSEHRA whose very remit includes the creation of a two-way flow of ideas, solutions and best practices between the VA and the Open Source Community.  Instead, it appears that solutions are being prescribed and contracts awarded to companies whose standard solution will be inevitably based around Java and the Java EE Platform, because that’s the skills they have available, not because it’s the most appropriate technology for the VA.

What’s wrong with the VistA Evolution picture is that those large contractors who are being awarded huge and expensive contracts are not being mandated to consult with the Open Source Community, and made to ensure that they don’t drive the VA into a repeat of the HealtheVet attempt to modernise VistA.

Industry leaders are moving away from Java and adopting Node.js.

EWD.js allows the VA to follow suit and modernise VistA with fewer moving parts and much less complexity, and therefore in a much lower-risk way.  EWD.js will allow to VA to realise the same benefits that those industry leaders are reporting.

EWD.js allows the thousands of man-years of effort and skill that has gone into creating VistA to be handed on to the next generation of developers.  With EWD.js, VistA, the EHR that consistently comes out top in surveys of user satisfaction, and that is universally loved by its users at the sharp-end within the VA,  doesn’t need to be confined to the scrap heap.  In the light of the evidence I’ve presented here, surely that would be a serious crime and an act of immense stupidity.

It’s time to fix the VistA Evolution picture.  It deserves to be a masterpiece that can stand the test of time and ever-changing fashion.

Isn’t that what Veterans deserve?

Advertisements

8 comments

  1. Spot on Rob,

    Lest any one wonder how viable is a rework of VistA via EWD.js, see here…

    2 Clinicians: 2 Developers: 20 Days: Agile Development towards NHS VistA
    http://frectal.com/2013/12/30/2-clinicians-2-developers-20-days-agile-development-towards-nhs-vista/

    In 20days work we had;
    a version of VistA up and running,
    accommodating some UK changes
    for reading key clinical data
    + data entry for allergies
    on a browser, tablet and smartphone
    ..via a combination of VistA and EWD.js

    The key mistake being made here is that folk assume you need to spend millions/billions to make progress in this field… hence I’ve not had any offers to progress the next 30/40/50 days of development.

    History will be the judge..

    Tony

  2. I am in the middle of a legacy app conversion and the issues faced by VistaA developers are now part of my life. It is important to understand that these apps embraced client/server approaches because they emulated the original character-based which were the only alternative when the app was created. Approaches like VB and Java made sense because to the development folks. Over time, these technologies got vastly more complex as the demands for expanded feature sets led them to bloat into huge piles of moving parts because the central notion (client/server along with their particular approach) was never examined.

    At the same time the favored UI techs were becoming unmanageable, the web was evolving into a viable development platform. By the millennium, the web (if you used Cache and M/Gateway tools, anyway) had become a viable alternative to fat client/server approaches. But corporate and government inertia along with their huge investment in outmoded technology compelled them to continue down the wrong path despite the warning signs of large lead times and collapsed projects.

    I like to say that Java is now a solution to a series of technical problems (such as VB/Delphi deployment nightmares) that no longer exist. With the advent of node and ewd.js, the final weakness of web apps (real-time intercommunication) is gone and there is no reasonable case to be made for Java or .NET (which took a similar approach…bloat…to solving the client/server issues that they created with their earlier products).

    jb

  3. Something I didn’t specifically cover in the article: so what would VistA Evolution look like if it was based on EWD.js / Node.js instead of a Java EE stack.

    Well I’ve actually covered this in detail before. See:

    https://robtweed.wordpress.com/2013/11/19/json-interfacing-vista-and-other-legacy-mumps-systems/

    If you look at my diagram of the EWD.js / VistA architecture above in the main article, all this work would take place in the red box around FileMan API, named “Function Wrapper (Mumps)”

    If you want to see an example of what the result might be, take a look at Sid Torason’s work at Astute Semantics, eg:

    http://astutesemantics.com/fileman-gui/

    and also Chris Casey’s work on Tony Shannon’s NHS VistA RCP Demonstrator:

    Another aspect of VistA Evolution that is going to be important is the use of REST, particularly driven by HL7 FHIR. The highly secure, scalable EWD.js solution for this is summarised in this set of presentation slides:

    http://gradvs1.mgateway.com/download/VistAREST.pdf

    Once again, a significantly simpler, less complex solution for delivering REST-ful VistA interfaces than the VA’s Java EE / Enterprise Server Bus strategy, with far fewer moving parts, significantly better performance, equivalent scalability and without compromising security. The EWD REST infrastructure is available today, and my proposed REST interfaces to VistA, including the secured initiate/login stage, are already developed and demonstrable.

    I’m happy to provide further detail to anyone who is interested.

    Rob

  4. Good read. Has the VA interviewed organizations that currently make use of a Java stack for their EHR? I still think the real question is how does EWD empower agile over other technologies. There will always be an argument for current enterprise technology, see – https://blog.appfog.com/node-js-is-taking-over-the-enterprise-whether-you-like-it-or-not/ discussion. For instance, “if you do Java the simple and scalable way (e.g., Jetty over Tomcat, Guice over Spring, and no J2EE or JEE stuff), you can get amazing scale with highly productive environment (Language, VM, Library ecosystem). The problem is not Java is the way people use Java.” And on and on…

    Agile can’t handle team A does front-end, team B does backend, team C does db, team D does ops, team X does etc., all in isolation, fielding requests off an interrupt queue. In order to do single-piece flow you need integrated teams that can work across a vertical slice and that are empowered to make changes to shared resources. If you end up waiting two weeks for ops to make a config change or your priority gets bumped in any of the queues, what do you tell your customer at that point? Teams working on stories for integrating data with the use of that data not in sight = rework + bloat.

    How does streamlining your architecture potentially address these problems? How does EWD support dev-ops vertical teams? Compare EWD release to proposed Java Stack release. Who does the work, when. Which one flows better. This is the real issue.

    1. Well, Chris, since you attended the EWD.js course, and as someone with a Java background, perhaps you could answer some of the questions you’ve posed yourself? 🙂

      Regarding devops capabilities/support, see the latest capabilities in the ewdMonitor application that comes with EWD.js. You can monitor, diagnose, reconfigure and maintain child processes from it, without affecting or stop/starting the main process or running applications. You can also now bring up the Node-inspector debugger. See:

      https://groups.google.com/forum/#!topic/enterprise-web-developer-community/AiBrT2xoUnc

      ewdMonitor is explained in the EWD.js Reference Guide:

      http://gradvs1.mgateway.com/download/EWDjs.pdf

      As it happens I’m working on an agile team-based EWD.js project, using git / Github for version control. The whole idea of EWD.js is high-speed development, deployment, updating. There’s a growing list of completed applications which I’ve referred to frequently in this and other posts – the common theme is the exceptional time from start to completion. You’re going to see a lot more – that’s going to be the proof of the pudding and what really matters.

      Rob

      1. I see no reason why you couldn’t use the Strongloop Devops solution, by the way, if you wanted or needed something additional to the facilities provided by ewdMonitor:

        http://strongloop.com/strongblog/strongops-node-js-devops-press-release/

  5. […] Original Article: VISTA EVOLUTION: WHAT’S WRONG WITH THIS PICTURE? […]

  6. FYI: VistA Evolution’s replacement for CPRS is actually written in JavaScript under Node.JS. The prototype was written using Java. It also uses Cache (MUMPS Database) as a database and hitched it to SOLR. Pretty cool stuff. Keep an eye on OSEHRA for it’s release.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: