Proposal for Engage services layer technologies

[Colin Clark]

Hi everyone,

Over the past few months, the development team has been doing a lot of  
research and talking about potential technologies for the Engage  
services layer. I've taken this information and distilled it into a  
proposal for the database and Web tools I think we should use on the  
server-side. I've tried to keep this page fairly succinct, focusing on  
the core technologies, our motivations for selecting them, and the  
context in which they will be used:

http://wiki.fluidproject.org/display/fluid/Engage+Server-Side+Technology

In short, the recommendation is to use Apache's CouchDB as our  
database and Lucene for text searches. For Web service development,  
the idea is to use JavaScript on the server, assembling a handful of  
existing technologies into a new lightweight Web container called  
Kettle. This environment will provide us with an extremely  
approachable toolset, allowing us to get started quickly by using  
Infusion to write code on both the client and the server. Infusion  
will remain a completely server-agnostic framework for client-side  
JavaScript development, and will be used unchanged in the Kettle  
environment.

Throughout the process of evaluating and researching these  
technologies, Antranig and I have compiled a pretty large set of notes  
and ideas about the various technologies we looked at:

http://wiki.fluidproject.org/display/fluid/Fluid+Engage+Server-side+Technology+Notes

This page provides a rich source of information, benchmarks, and  
background thinking about the technology landscape for Engage. It  
covers issues like hosting, the schema-less approach, and the other  
technologies we evaluated. It's pretty dense, but there's a lot there  
that you may find useful.

In preparation for last month's Engage All Hands meeting, Antranig put  
together a first draft implementation of the Kettle server-side  
environment using Rhino, Jetty, and Env.js. While this is still very  
much in development, it shows the viability of the approach and  
associated toolset (including debuggers!). This code is available in  
the Incubator source code repository:

https://source.fluidproject.org/svn/incubator/engage-sketches/trunk/

Yura has set up an instance of CouchDB with Lucene on our build server  
and populated it with a small sample of data from the McCord Museum's  
collection. It is available here:

http://build.fluidproject.org:5984/_utils/database.html?mccord

I'm quite enthusiastic to hear your thoughts, comments, and ideas  
about this approach.

Colin

---
Colin Clark
Technical Lead, Fluid Project
Adaptive Technology Resource Centre, University of Toronto
http://fluidproject.org