CouchDB - managing design documents?

Harnum, Alan aharnum at ocadu.ca
Thu Aug 31 18:40:22 UTC 2017


Thanks Tony – looking forward to talking about this.

Gregor & I have been continuing work on an Infusion component for managing design documents (and a few other basic Couch config functions), since we have need of it for a current project: https://issues.fluidproject.org/browse/SJRK-6

While I wouldn’t consider this close to final form, the example hopefully gets the intent across: https://github.com/waharnum/couch-config/blob/master/examples/js/couchConfigBasic.js

I think JSON schema could be additionally powerful if we can find an approach to use them within the context of the validate_doc_update function of a design document. I think this would be possible using the require feature available in Couch: http://docs.couchdb.org/en/2.0.0/query-server/javascript.html#require

I could see a version of an Infusion-based CouchDB config component that did something like the following:


1)       Managed a set of JSON schema

2)       Loaded those schema to Couch

3)       Generated validate_doc_update functions appropriately based on those schema

A common convention in Couch seems to be the use of a “type” field on documents that should be subject to validation, in order to access a particular set of validation rules. The “type” field in this scenario could be the JSON schema reference or similar.

From: Tony Atkins <tony at raisingthefloor.org>
Date: Wednesday, August 30, 2017 at 12:45 PM
To: "Bates, Simon" <sbates at ocadu.ca>
Cc: "Harnum, Alan" <aharnum at ocadu.ca>, Fluid Work <fluid-work at fluidproject.org>
Subject: Re: CouchDB - managing design documents?

Hi, All.

Monday the 11th looks like the only day that works for everyone.  I am waiting to chat with Colin on his return before picking the time slot.  Please hold 15:00-17:30 CET on that date for now, I will get back with the final time on the 4th.

Cheers,


Tony

On Tue, Aug 29, 2017 at 11:14 AM, Tony Atkins <tony at raisingthefloor.org<mailto:tony at raisingthefloor.org>> wrote:
Hi, Simon:

That's a fair point, at a minimum we should be aware of the need and see what if any ground work we can take care of in working on managing design documents.

As far as I know we do not currently have a general solution to managing data migrations.  In the rare cases when I've needed to restructure, I've used a bulk read, transform, and bulk update, but obviously that's only a very temporary solution.

At the risk of entangling this conversation with another fairly big topic, one idea that comes to mind is using versioned JSON schemas and some convention of indicating what schema is used for a particular record (for example, a URI pointing to the schema).  That would let us:

  1.  Detect existing corrupt or incomplete records.
  2.  Limit a migration to a particular type and/or version of a record.
  3.  Verify that the results of the migration are themselves valid.
  4.  Determine which records have already been migrated, and which have not.
Anyway, we should bring up the topic in the architecture meeting and set up an initial meeting in the next week or so.  With that in mind, I have created a Doodle Poll:

https://doodle.com/poll/snpxb4f496g289pk

Cheers,


Tony

On Mon, Aug 28, 2017 at 5:31 PM, Bates, Simon <sbates at ocadu.ca<mailto:sbates at ocadu.ca>> wrote:
To broaden this topic slightly, I'm thinking that some form of data migration automation would also be important to have, in addition to updating design documents. For example, if a document structure is modified, we would want to have an automated way of updating existing data to the new structure.

Do we have anything currently existing to manage data migration and document structure evolution? If not, I'm thinking it would be worth including the topic in discussions on managing design documents.

Simon

________________________________________
From: fluid-work <fluid-work-bounces at lists.idrc.ocad.ca<mailto:fluid-work-bounces at lists.idrc.ocad.ca>> on behalf of Harnum, Alan <aharnum at ocadu.ca<mailto:aharnum at ocadu.ca>>
Sent: August 28, 2017 11:19:59 AM
To: Tony Atkins
Cc: Fluid Work
Subject: Re: CouchDB - managing design documents?

Hi Tony,

Thanks for the replies – we’ve also been looking at the various Couchapp implementations, but have been hesitant both for what you cite below (large additional stack of dependencies) and this part of the current CouchDB docs: http://docs.couchdb.org/en/2.1.0/ddocs/, which discourages the previously-promoted “CouchApps” paradigm that the various CouchApp tools are designed to use. My sense is that the various Couchapp projects are not under terribly active development in light of this change in CouchDB’s latest documentation.

I’d also be an advocate of a Fluid component for capturing views and perhaps other configuration aspects of an Infusion application that intends to make use of CouchDB - I actually did some experimentation in this direction last Friday, which I’ve now cleaned up a bit and added some documentation: https://github.com/waharnum/couch-config

In my ideal world I think I’d be able to write a CouchDB configuration for an application in pure Infusion rather than pushing JSON around with curl or other tools.

From: Tony Atkins <tony at raisingthefloor.org<mailto:tony at raisingthefloor.org>>
Date: Monday, August 28, 2017 at 7:32 AM
To: "Harnum, Alan" <aharnum at ocadu.ca<mailto:aharnum at ocadu.ca>>
Cc: Fluid Work <fluid-work at fluidproject.org<mailto:fluid-work at fluidproject.org>>
Subject: Re: CouchDB - managing design documents?

Hi, Alan:

Thanks again for starting yet another interesting and important discussion.  I've had to do a lot of this in my own work, and have alternated between:

  1.  Storing the design documents as raw JSON (mainly for testing).  This is similar to the approach used in the example you shared.
  2.  Writing design documents using couchapp<https://github.com/couchapp/couchapp>.
The first approach seems easy enough, but is a bit tedious on a few fronts:

  1.  It's difficult to write even moderately complex functions as escaped strings.
  2.  Replacing an existing design document is a bit tedious, as it involves either looking up the current document's _rev or DELETEing and then POSTing the replacement.
  3.  Although it's possible to require libraries from a design document<http://docs.couchdb.org/en/2.0.0/query-server/javascript.html#require>, this is pretty hard to manage using raw JSON files.
Couchapp makes each of these easier:

  1.  You just save your functions to a file, which you can run through our standard linting and more easily check for syntax errors.  You can even write unit tests against the functions you plan to use in your design documents.  This is especially helpful when developing reduce functions, which can be a bear to troubleshoot with CouchDB or PouchDB.
  2.  Although it's trivial, couchapp makes it very easy to update design docs repeatedly.
  3.  Couchapp makes it easier to bundle dependencies as part of your design document (basically by deploying them to a "lib" or similar directory within a design document's directory).
That being said, couchapp is something I have only relied on reluctantly.  I don't use it in tests, for example, as it seems a bit much to me to introduce a whole stack of dependencies (python, required libraries) just to solve what's a fairly simple use case.  Antranig and I have previously talked about building a Fluid component to assist with creating design documents, and it seems like a good time to broaden and refresh that discussion.

Regardless of whomever else is interested, I would be very happy to work together with you (and I'm assuming Antranig) on a ticket outlining the use cases in more detail.  Depending on the scope of what we agree is needed, I might end up simply building it the next time I need to work on a batch of design documents.  I'd also be very happy to help review something someone else builds and test that in my own work.

Cheers,


Tony

On Thu, Aug 24, 2017 at 4:16 PM, Harnum, Alan <aharnum at ocadu.ca<mailto:aharnum at ocadu.ca><mailto:aharnum at ocadu.ca<mailto:aharnum at ocadu.ca>>> wrote:
Hi all,

We (Greg Moss & myself) are curious to know the opinions of others who've worked with it (especially in the context of Infusion / Kettle) about how to best manage design documents when using CouchDB as part of an application.

Specifically, what approaches have worked for externalizing design document code, ensuring the database has the latest design document code, etc?

I know one example of doing this is in gpii-oauth2-datastore, which is a JSON externalization of the views that is imported to the DB as part of application provisioning, described at: https://github.com/GPII/universal/tree/master/gpii/node_modules/gpii-oauth2/gpii-oauth2-datastore/dbViews

I am wondering if people have seen or considered other approaches.

My own feeling is that it would be useful to be able to model an expected CouchDB database design document as an Infusion component, with the component taking responsibility for ensuring a set of view functions are present in the specified DB, but I don't feel a great certainty about this.

Curious as to the thoughts of others!

Alan

ALAN HARNUM
SENIOR INCLUSIVE DEVELOPER
INCLUSIVE DESIGN RESEARCH CENTRE, OCAD UNIVERSITY

E aharnum at ocadu.ca<mailto:aharnum at ocadu.ca><mailto://aharnum@ocadu.ca<mailto:aharnum@ocadu.ca>>

OCAD UNIVERSITY
100 McCaul Street, Toronto, Canada, M5T 1W1
www.ocadu.ca<http://www.ocadu.ca><http://ocadu.ca/>

_______________________________________________________
fluid-work mailing list - fluid-work at lists.idrc.ocad.ca<mailto:fluid-work at lists.idrc.ocad.ca><mailto:fluid-work at lists.idrc.ocad.ca<mailto:fluid-work at lists.idrc.ocad.ca>>
To unsubscribe, change settings or access archives,
see http://lists.idrc.ocad.ca/mailman/listinfo/fluid-work


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idrc.ocad.ca/pipermail/fluid-work/attachments/20170831/4ae0116c/attachment.html>


More information about the fluid-work mailing list