CouchDB - managing design documents?

Harnum, Alan aharnum at
Thu Aug 31 18:40:22 UTC 2017

Thanks Tony – looking forward to talking about this.

Gregor & I have been continuing work on an Infusion component for managing design documents (and a few other basic Couch config functions), since we have need of it for a current project:

While I wouldn’t consider this close to final form, the example hopefully gets the intent across:

I think JSON schema could be additionally powerful if we can find an approach to use them within the context of the validate_doc_update function of a design document. I think this would be possible using the require feature available in Couch:

I could see a version of an Infusion-based CouchDB config component that did something like the following:

1)       Managed a set of JSON schema

2)       Loaded those schema to Couch

3)       Generated validate_doc_update functions appropriately based on those schema

A common convention in Couch seems to be the use of a “type” field on documents that should be subject to validation, in order to access a particular set of validation rules. The “type” field in this scenario could be the JSON schema reference or similar.

From: Tony Atkins <tony at>
Date: Wednesday, August 30, 2017 at 12:45 PM
To: "Bates, Simon" <sbates at>
Cc: "Harnum, Alan" <aharnum at>, Fluid Work <fluid-work at>
Subject: Re: CouchDB - managing design documents?

Hi, All.

Monday the 11th looks like the only day that works for everyone.  I am waiting to chat with Colin on his return before picking the time slot.  Please hold 15:00-17:30 CET on that date for now, I will get back with the final time on the 4th.



On Tue, Aug 29, 2017 at 11:14 AM, Tony Atkins <tony at<mailto:tony at>> wrote:
Hi, Simon:

That's a fair point, at a minimum we should be aware of the need and see what if any ground work we can take care of in working on managing design documents.

As far as I know we do not currently have a general solution to managing data migrations.  In the rare cases when I've needed to restructure, I've used a bulk read, transform, and bulk update, but obviously that's only a very temporary solution.

At the risk of entangling this conversation with another fairly big topic, one idea that comes to mind is using versioned JSON schemas and some convention of indicating what schema is used for a particular record (for example, a URI pointing to the schema).  That would let us:

  1.  Detect existing corrupt or incomplete records.
  2.  Limit a migration to a particular type and/or version of a record.
  3.  Verify that the results of the migration are themselves valid.
  4.  Determine which records have already been migrated, and which have not.
Anyway, we should bring up the topic in the architecture meeting and set up an initial meeting in the next week or so.  With that in mind, I have created a Doodle Poll:



On Mon, Aug 28, 2017 at 5:31 PM, Bates, Simon <sbates at<mailto:sbates at>> wrote:
To broaden this topic slightly, I'm thinking that some form of data migration automation would also be important to have, in addition to updating design documents. For example, if a document structure is modified, we would want to have an automated way of updating existing data to the new structure.

Do we have anything currently existing to manage data migration and document structure evolution? If not, I'm thinking it would be worth including the topic in discussions on managing design documents.


From: fluid-work <fluid-work-bounces at<mailto:fluid-work-bounces at>> on behalf of Harnum, Alan <aharnum at<mailto:aharnum at>>
Sent: August 28, 2017 11:19:59 AM
To: Tony Atkins
Cc: Fluid Work
Subject: Re: CouchDB - managing design documents?

Hi Tony,

Thanks for the replies – we’ve also been looking at the various Couchapp implementations, but have been hesitant both for what you cite below (large additional stack of dependencies) and this part of the current CouchDB docs:, which discourages the previously-promoted “CouchApps” paradigm that the various CouchApp tools are designed to use. My sense is that the various Couchapp projects are not under terribly active development in light of this change in CouchDB’s latest documentation.

I’d also be an advocate of a Fluid component for capturing views and perhaps other configuration aspects of an Infusion application that intends to make use of CouchDB - I actually did some experimentation in this direction last Friday, which I’ve now cleaned up a bit and added some documentation:

In my ideal world I think I’d be able to write a CouchDB configuration for an application in pure Infusion rather than pushing JSON around with curl or other tools.

From: Tony Atkins <tony at<mailto:tony at>>
Date: Monday, August 28, 2017 at 7:32 AM
To: "Harnum, Alan" <aharnum at<mailto:aharnum at>>
Cc: Fluid Work <fluid-work at<mailto:fluid-work at>>
Subject: Re: CouchDB - managing design documents?

Hi, Alan:

Thanks again for starting yet another interesting and important discussion.  I've had to do a lot of this in my own work, and have alternated between:

  1.  Storing the design documents as raw JSON (mainly for testing).  This is similar to the approach used in the example you shared.
  2.  Writing design documents using couchapp<>.
The first approach seems easy enough, but is a bit tedious on a few fronts:

  1.  It's difficult to write even moderately complex functions as escaped strings.
  2.  Replacing an existing design document is a bit tedious, as it involves either looking up the current document's _rev or DELETEing and then POSTing the replacement.
  3.  Although it's possible to require libraries from a design document<>, this is pretty hard to manage using raw JSON files.
Couchapp makes each of these easier:

  1.  You just save your functions to a file, which you can run through our standard linting and more easily check for syntax errors.  You can even write unit tests against the functions you plan to use in your design documents.  This is especially helpful when developing reduce functions, which can be a bear to troubleshoot with CouchDB or PouchDB.
  2.  Although it's trivial, couchapp makes it very easy to update design docs repeatedly.
  3.  Couchapp makes it easier to bundle dependencies as part of your design document (basically by deploying them to a "lib" or similar directory within a design document's directory).
That being said, couchapp is something I have only relied on reluctantly.  I don't use it in tests, for example, as it seems a bit much to me to introduce a whole stack of dependencies (python, required libraries) just to solve what's a fairly simple use case.  Antranig and I have previously talked about building a Fluid component to assist with creating design documents, and it seems like a good time to broaden and refresh that discussion.

Regardless of whomever else is interested, I would be very happy to work together with you (and I'm assuming Antranig) on a ticket outlining the use cases in more detail.  Depending on the scope of what we agree is needed, I might end up simply building it the next time I need to work on a batch of design documents.  I'd also be very happy to help review something someone else builds and test that in my own work.



On Thu, Aug 24, 2017 at 4:16 PM, Harnum, Alan <aharnum at<mailto:aharnum at><mailto:aharnum at<mailto:aharnum at>>> wrote:
Hi all,

We (Greg Moss & myself) are curious to know the opinions of others who've worked with it (especially in the context of Infusion / Kettle) about how to best manage design documents when using CouchDB as part of an application.

Specifically, what approaches have worked for externalizing design document code, ensuring the database has the latest design document code, etc?

I know one example of doing this is in gpii-oauth2-datastore, which is a JSON externalization of the views that is imported to the DB as part of application provisioning, described at:

I am wondering if people have seen or considered other approaches.

My own feeling is that it would be useful to be able to model an expected CouchDB database design document as an Infusion component, with the component taking responsibility for ensuring a set of view functions are present in the specified DB, but I don't feel a great certainty about this.

Curious as to the thoughts of others!



E aharnum at<mailto:aharnum at><mailto://<>>

100 McCaul Street, Toronto, Canada, M5T 1W1<><>

fluid-work mailing list - fluid-work at<mailto:fluid-work at><mailto:fluid-work at<mailto:fluid-work at>>
To unsubscribe, change settings or access archives,

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>

More information about the fluid-work mailing list