Engage / ArtifactView / Comments / Storing documents in couchdb and other things

Colin Clark colinbdclark at gmail.com
Mon Dec 21 19:10:05 UTC 2009


Hey all,

[Joan, I know you're currently away from work due to an injury, so  
don't worry about the details of this. My main goal in responding to  
your questions (sorry for the delay!) is so that the whole community  
has clarity on our next steps in terms of data access.]

I like the concept of the database.js implementation. Since it was  
written, we uncovered two JavaScript APIs that ship directly with  
CouchDB:

http://titan.atrc.utoronto.ca:5984/_utils/script/couch.js
http://titan.atrc.utoronto.ca:5984/_utils/script/jquery.couch.js

One is a plain JavaScript API, and the other is designed as a jQuery  
plugin. They offer overlapping functionality, but in general a  
preference is emerging for the jQuery plugin--it's properly  
namespaced, and a bit better structured. Here's a thread on the fluid- 
work list from last week where Antranig and I discuss the issues:

http://old.nabble.com/Fluid-Engage-nightly-build-broken-td26785515.html#a26832718

I think we're always better to contribute back to existing projects  
rather than rolling our own where it's appropriate, so I'd like to  
suggest that instead of working on database.js, we contribute some  
patches back to Couch for their jQuery plugin. One area where we'll  
need to contribute back, in addition to the ones mentioned by  
Antranig, is support for Couch Lucene views, which we use frequently.

Antranig also has some code in his FLUID-213 patch that illustrates  
some slightly higher-level ways of making Couch requests within Kettle:

http://issues.fluidproject.org/secure/attachment/10833/ENGAGE-213-demonstration.patch

This covers the low-level issues of data access. I think our second  
goal is to provide nice abstractions so that every Engage service  
developer doesn't have to write the same data access boiler plate  
goal. In a recent development meeting, we talked a little bit about  
the notion of "data source specs," which would provide all of the  
information needed to retrieve and save data to/from Couch in a  
declarative way.

This data source spec would include the URL in Couch that identifies  
the document or set of documents being used by a particular data feed  
and component. It would also include the requisite model mapping  
information required to correctly reshape the model into a form the  
component can suitably use. You can see a rough sketch of our current  
model mapping code in this file:

http://source.fluidproject.org/svn/fluid/engage/fluid-engage-core/trunk/framework/js/Engage.js

Take a look at the fluid.engage.collections data structure for an  
example of the model maps we're currently using. This code needs quite  
a bit of work to be correctly parameterized and generalized, but the  
idea is to be able to have a document in Couch that represents the  
necessary transformations required on the model for a particular view  
context. The framework will include a number of "canned reshapers" for  
common mapping tasks, and museums could extend this with their own  
specific code if needed.

This second layer of data access will take a bit longer for us to  
implement, so let's focus first on the task--contributing back to  
Couch's jquery.couch.js file based on our own needs.

I hope this helps clarify the issue of data access in Engage a bit.  
This is all still very much in the concept stage, so no doubt it also  
opens up some new questions. Everyone, please don't hesitate to  
respond to this thread with thoughts, comments, and questions.

Colin


On 1-Dec-09, at 9:31 AM, Joan Garcia Vila wrote:

> Hi all.
>
> Recently I've posted an alpha implementation of the Database  
> component (which has to become, I suppose, a subcomponent of  
> ArtifactComments [server side]).
> Needs a conceptual review.
>
> - Talking about storing data on couchdb, this has been my first  
> approach:
>
>             //  Artifact Comment  example.
>             var artifactComment = {};
>             artifactComment.engage_type = "artifact-comment";
>             artifactComment.artifact_id =  
> "00311f0d3f5456c91934e472b880e9c5";
>             artifactComment.insertDate = "2009-11-30";
>             artifactComment.name = "Juan";
>             artifactComment.location = "Barcelona, Catalonia, Spain";
>             artifactComment.abuse = false;
>             artifactComment.info = "Barcelona is a beautiful  
> mediterranian city.";
>
> There are few essential key points to check (understand / validate /  
> talk about):
>
> 1.- This document has a field that identifies its type  
> ("engage_type").
> If we store different object types inside the same database this  
> will help and give higher scores in lucene fulltext search if we add  
> the document type in the query
> (Please see/run the example).
>
> 2 .- This document has a reference to another document. See field  
> "artifact_id". Important to have good views with high selectivity.
> This will help us to create "in-server" high speed paged queries.
> Views for each document type if we need this functionality.
> (Please see/run the example).
>
> 3.- The object also gets its _id and _rev attributes after it has  
> been stored.
> This is coherent with couchdb (object oriented database) and the key  
> is an uuid.
> (Please see/run the example).
>
> 4.- Data attributes is stored as a fixed text with pattern "YYYY-MM- 
> DD".
> This will help us to have date sorted field views.
>
> 5.- One important thing is that having the documents (2 diferent  
> types) inside the same database allows us to create fast views (see  
> Database.js).
> We can also create "pseudo joins" [complex views using map/reduce  
> _design functions]
>
> - Other things that come to my mind, while developing the component,  
> are:
> 1. - How do we know who we are?, I haven't see a login page in the  
> wireframes.
> But there is behaviour that seems to depend on knowing who we are.
> Like report abuse or delete, for an example.
> makes sense?
>
> 2.- How do we know that the user is the moderator?
> makes sense?
>
> Cheers,
> joan
>
> ps: you can find (Database. [js/html] on scratchpad/bug-parade.  
> _______________________________________________________
> fluid-work mailing list - fluid-work at fluidproject.org
> To unsubscribe, change settings or access archives,
> see http://fluidproject.org/mailman/listinfo/fluid-work

---
Colin Clark
Technical Lead, Fluid Project
http://fluidproject.org




More information about the fluid-work mailing list