A New Chapter for EWD.js: now supporting MongoDB

Given the JavaScript and JSON focus of EWD.js, it was somewhat inevitable that I’d want to extend its capabilities to include support for MongoDB: after all, MongoDB is probably the most successful of the Document-oriented databases in the NoSQL marketplace right now.  It’s also a lot better known than the Mumps database technologies that have underpinned EWD.js to date, so I’m hoping we can look forward to more widespread adoption of EWD.js.  Of course, MongoDB is also of interest as it’s an Open Source database.

I’ve put out Build 50 of EWD.js on both GitHub and NPM, and I’ll be enhancing the EWD.js documentation over the next days to cover how to use MongoDB with EWD.js.  In this posting I want to summarise some of the key features and outline why EWD.js deserves consideration by MongoDB developers who are more used to the other Node.js frameworks out there.

Of course, support for the Mumps database technologies will continue in EWD.js.  MongoDB represents a new string to the EWD.js bow, not a replacement, and is meant to compliment, not compete.

The M/Gateway Node.js MongoDB interface

A key feature of EWD.js is its Queue/Child Process architecture.

ewdjs-mongo-architecture

By ensuring that only one request at a time is handled by each child process, EWD.js allows the use of truly synchronous programming: this is very unusual for a Node.js technology, where asynchronous programming is the norm.  Three things encouraged me to take EWD.js down this route:

  • the painful and time-consuming experience of trying to create applications of any size and significance that use asynchronous logic;
  • the maintenance nightmare that such tortuous, unintuitive logic left behind;
  • the fact that the synchronous versions of the APIs that Chris Munt, my colleague at M/Gateway Developments, created for InterSystems’ Node.js interface to GlobalsDB and Cache are significantly faster than their asynchronous counterparts

In order to add MongoDB as a database option for EWD.js it was therefore necessary to have a synchronous interface to MongoDB: the existing options for Node.js were all asynchronous and therefore unusable within EWD.js’s architecture.

Chris did his magic yet again and about a week or so ago he came up with a new Node.js interface module for MongoDB that is synchronous.  Put simply, this amounts to three things for an application developer:

1) “Callback hell”, the bane of the Node.js developer disappears.  So instead of coding like this:

  mongo.retrieve('db.test', {department: 'IT'}, function(result) {
    // do something with the result inside this callback function
  });

the developer can code in the more intuitive, truly synchronous style:

  var result = mongo.retrieve('db.test', {department: 'IT'});
  // now do something with the result immediately after

2) The developer knows the sequence in which everything will happen – the “hidden surprises” you get from asynchronous code, where stuff happens in a way and sequence that is non-intuitive disappear (try writing and handling the consequences of a simple for loop and you’ll see what I mean!).  This makes the code much more maintainable and much quicker to develop.

3) The developer can properly handle errors and faults using standard try/catch constructs.

Of course the developer’s code will have to wait while each database request is completed, theoretically wasting CPU cycles.  However, since the child process that is running the developer’s code is only handling a single user’s request, the only person being held up is that one user.  Requests for other users will be handled in parallel by other child processes in the worker pool.  Meanwhile, the main, core Node.js process of EWD.js is 100% asynchronous, handling, queuing and dispatching incoming requests from all users simultaneously.

Chris’s synchronous Node.js interface for MongoDB can be built for all operating systems that support Node.js, ie both 32-bit and 64-bit versions of Linux and Windows, and also for OS X.  He’s even ported it to the Raspberry Pi’s ARM architecture.  You’ll find pre-built binary builds of the interface for most of these platforms in the EWD.js Github repository: they’ve been specifically built for use with the 0.10 version of Node.js.

Chris will be separately making the sources available for his MongoDB/Node.js interface in due course.

Emulating Global Storage in MongoDB

The runtime environment of EWD.js (or more correctly, the ewdgateway2 module) was written on the assumption that it had access to a Mumps database, and it actually uses such a database for its own internal management purposes.  I didn’t want to have to create a whole new version of EWD.js that required different coding in order to do the equivalent things whilst using MongoDB as its database: I wanted EWD.js to just be a single code-base.

As a result of the experience of emulating a Mumps database on the Raspberry Pi by using a JavaScript object, I realised that it should be possible to do a similar trick, but by using a MongoDB “collection” (conceptually an array of persistent JavaScript objects).  So, EWD.js includes a module named mongoGlobals.js which emulates EWD.js’s JavaScript abstraction of Mumps  Globals and the APIs provided by InterSystems’ Node.js interface to GlobalsDB and Cache.

Since the internal logic of EWD.js makes use of these APIs, by emulating them exactly, no change was needed to EWD.js: when it runs against MongoDB, it still thinks it’s connected to and making use of a Mumps Global database!  Ah, the magic of APIs!

EWD.js & MongoDB: Collections v Mumps Globals

A direct consequence of this emulation of a Mumps database is that a MongoDB developer who uses EWD.js has a choice:

  • using MongoDB as true MongoDB, ie using database storage that is based around the concept of collections of persistent JavaScript Objects.
  • using MongoDB as if it was a Mumps database, and therefore making use of the _getDocument(), _setDocument() and other Global-based APIs that have been created for EWD.js.

The developer can do either or both.

One warning: certainly in its initial early release form, the emulation of Mumps Globals using MongoDB isn’t, as yet, very efficient.  It’s certainly good enough for a good user experience when using EWD.js: the amount of Global database activity within EWD.js itself is pretty small, so you’ll barely notice.  However, if a developer was to perform significant amounts of emulated Global access within his/her application logic, performance would likely suffer.

I’m going to be researching into what’s needed to speed the emulation up: by proper use of indices I believe I should be able to make some significant improvements.  Just how significant those improvements will be, and whether I can get its performance closer to that of a native Mumps database remains to be seen.

That having been said, the MongoDB version of EWD.js hasn’t been designed to be used as a Mumps database: it’s been designed to allow MongoDB developers to benefit from the rapid development framework that it provides for building Web-Socket based client/server browser applications.  On the other hand, if you want to make extensive use of Mumps Globals, you should use EWD.js with one of the Mumps databases.  It’s a case of “horses for courses”.

Conclusions

So that’s the background to the integration of MongoDB and EWD.js.  I’m looking forward to seeing the reaction of the Node.js/MongoDB development community.  EWD.js presents quite a different development paradigm for this community.  It’s also a much bigger and, dare I say it, dynamic and forward thinking community to the one I’ve been working with to date.

Interesting times ahead for EWD.js!

Advertisements

3 comments

  1. Urs Bucher · · Reply

    Hello Rob,
    what should I make of it. Since many documents blogs etc, show what genius concepts behind a Mumps database is, that they are stable, fast and very universal and that all kinds of NoSQL database can be implement with a Mumps databases.
    And EWDlite a clear concept, no unnecessary layers and APIs. Above of all the possibility that a client can access a whole JSON document and individual entries of a large JSON document quickly and efficiently.
    And now these thing is bent to to squeeze in an other database concept.
    I do not understand way all the old arguments are now worthless. E.g. previous the asynchronous concept from node.js was great, now it is the hell.
    We spent a lot of time to find the right database / framework (CouchDB, MongoDB, ArangoDB, Redis ) for our project and finally EWDlite with GlobalDB convinced us.

    Anyway, to be a little more specific. We are working on an application which use quit a lot of “big” JSON documents. Few entries of these JSON documents are accessed a lot. EDW.js is one of the few concept that allows to access a whole JSON document or only one value of it without a huge overhead. A short access time for such single values is very important for us.
    My question:
    Is it possible to use the EWD.js still with GlobalsDB in the future ?
    With the integration of MongoDB are there some restrictions on speed or other things if we still use a Mumps database ?
    What are your plans for the future of EWD.js ? Is the support of Mumps databases still the main line or is EWD.js going to support many different databases over time ?

  2. Urs

    First let me put your concerns to rest: I’ve no intention of removing support for GT.M, Cache or GlobalsDB. MongoDB support is simply an additional capability which, I believe, will be of interest mainly to those who know and like MongoDB but perhaps haven’t experienced what Mumps-based databases are capable of doing (or have been put off them by the poisonous disinformation that surrounds them). Its support broadens the potential interest in EWD.js beyond the relatively small user community who know and use Mumps databases.

    You are correct – a key difference between the JSON storage/retrieval capabilities of a Mumps systems versus MongoDB is the ability in the former to set or get small parts of an object extremely efficiently – in MongoDB you have to save and retrieve entire objects. My emulation of Global storage within MongoDB makes it possible to perform the same trick now, but the performance is currently way below that of a Mumps database. As I say in my article, whether I’ll be able to get this performance anywhere near that of a Mumps systems remains to be seen. Perhaps it will encourage MongoDB enthusiasts to take another look at Mumps databases?

    As to whether I’ve “bent” EWD.js to accommodate MongoDB – actually no, not at all. That was one of the cool parts of the port: by creating the Mumps emulation layer that used the exact same APIs as I’ve used for the Mumps databases, EWD.js was able to run against MongoDB without any change at all (apart from adding the logic to load the MongoDB-specific modules). Your GlobalsDB-based EWD.js system will be completely unaffected by what I’ve done.

    Regarding synchronous v asynchronous, I assume you realise that if you’re using EWD.js with GlobalsDB, you can (and should) be using the synchronous APIs already. Using the asynchronous ones is completely unnecessary, and they’re also significantly slower than their synchronous counterparts. So, once again, this isn’t a change introduced with MongoDB: EWD.js has always been about getting the many benefits that come from being able to program in “traditional” synchronous style, without losing the performance and scalability benefits of Node.js. If you are using the asynchronous APIs for GlobalsDB, then you really should consider switching to the synchronous ones.

    As I was unaware that you had made an evaluation of the databases you mention, and had decided on EWD.js + GlobalsDB, I first want to thank you for making that choice, but I’d also encourage you to provide some of your findings from that evaluation that led you to make the choice you did: I’m sure other readers would find this interesting and useful too. Feel free to do this via comments within this blog. I’d also be very interested to know more about the application(s) you’ve built, and whether or not I and others can see it/them working?

    Anyway, I hope I’ve allayed all of your concerns. In summary:

    – you will be able to use GlobalsDB with EWD.js into the future: I have no intention or desire to remove support for any of the Mumps database technologies that currently work with EWD.js

    – the inclusion of MongoDB has had no impact on the performance of EWD.js with GlobalsDB, GT.M or Cache – no changes were needed to EWD.js at all

    – as to the future, there won’t be any one favoured database for EWD.js. Whether I’ll include more databases will come down to demand. CouchDB and ArangoDB would seem the obvious potential candidates, but currently I’m not even thinking about them.

    Rob

  3. You mentioned that the synchronous Node.js interface for MongoDB was available for OS X, but it appears only windows, linux64, linux32, and Raspberry Pi are in the repository? Is the OS X build available somewhere else?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: