CouchDB - managing design documents?

Tony Atkins tony at raisingthefloor.org
Tue Aug 29 09:14:39 UTC 2017


Hi, Simon:

That's a fair point, at a minimum we should be aware of the need and see
what if any ground work we can take care of in working on managing design
documents.

As far as I know we do not currently have a general solution to managing
data migrations.  In the rare cases when I've needed to restructure, I've
used a bulk read, transform, and bulk update, but obviously that's only a
very temporary solution.

At the risk of entangling this conversation with another fairly big topic,
one idea that comes to mind is using versioned JSON schemas and some
convention of indicating what schema is used for a particular record (for
example, a URI pointing to the schema).  That would let us:

   1. Detect existing corrupt or incomplete records.
   2. Limit a migration to a particular type and/or version of a record.
   3. Verify that the results of the migration are themselves valid.
   4. Determine which records have already been migrated, and which have
   not.

Anyway, we should bring up the topic in the architecture meeting and set up
an initial meeting in the next week or so.  With that in mind, I have
created a Doodle Poll:

https://doodle.com/poll/snpxb4f496g289pk

Cheers,


Tony

On Mon, Aug 28, 2017 at 5:31 PM, Bates, Simon <sbates at ocadu.ca> wrote:

> To broaden this topic slightly, I'm thinking that some form of data
> migration automation would also be important to have, in addition to
> updating design documents. For example, if a document structure is
> modified, we would want to have an automated way of updating existing data
> to the new structure.
>
> Do we have anything currently existing to manage data migration and
> document structure evolution? If not, I'm thinking it would be worth
> including the topic in discussions on managing design documents.
>
> Simon
>
> ________________________________________
> From: fluid-work <fluid-work-bounces at lists.idrc.ocad.ca> on behalf of
> Harnum, Alan <aharnum at ocadu.ca>
> Sent: August 28, 2017 11:19:59 AM
> To: Tony Atkins
> Cc: Fluid Work
> Subject: Re: CouchDB - managing design documents?
>
> Hi Tony,
>
> Thanks for the replies – we’ve also been looking at the various Couchapp
> implementations, but have been hesitant both for what you cite below (large
> additional stack of dependencies) and this part of the current CouchDB
> docs: http://docs.couchdb.org/en/2.1.0/ddocs/, which discourages the
> previously-promoted “CouchApps” paradigm that the various CouchApp tools
> are designed to use. My sense is that the various Couchapp projects are not
> under terribly active development in light of this change in CouchDB’s
> latest documentation.
>
> I’d also be an advocate of a Fluid component for capturing views and
> perhaps other configuration aspects of an Infusion application that intends
> to make use of CouchDB - I actually did some experimentation in this
> direction last Friday, which I’ve now cleaned up a bit and added some
> documentation: https://github.com/waharnum/couch-config
>
> In my ideal world I think I’d be able to write a CouchDB configuration for
> an application in pure Infusion rather than pushing JSON around with curl
> or other tools.
>
> From: Tony Atkins <tony at raisingthefloor.org>
> Date: Monday, August 28, 2017 at 7:32 AM
> To: "Harnum, Alan" <aharnum at ocadu.ca>
> Cc: Fluid Work <fluid-work at fluidproject.org>
> Subject: Re: CouchDB - managing design documents?
>
> Hi, Alan:
>
> Thanks again for starting yet another interesting and important
> discussion.  I've had to do a lot of this in my own work, and have
> alternated between:
>
>   1.  Storing the design documents as raw JSON (mainly for testing).  This
> is similar to the approach used in the example you shared.
>   2.  Writing design documents using couchapp<https://github.com/
> couchapp/couchapp>.
> The first approach seems easy enough, but is a bit tedious on a few fronts:
>
>   1.  It's difficult to write even moderately complex functions as escaped
> strings.
>   2.  Replacing an existing design document is a bit tedious, as it
> involves either looking up the current document's _rev or DELETEing and
> then POSTing the replacement.
>   3.  Although it's possible to require libraries from a design document<
> http://docs.couchdb.org/en/2.0.0/query-server/javascript.html#require>,
> this is pretty hard to manage using raw JSON files.
> Couchapp makes each of these easier:
>
>   1.  You just save your functions to a file, which you can run through
> our standard linting and more easily check for syntax errors.  You can even
> write unit tests against the functions you plan to use in your design
> documents.  This is especially helpful when developing reduce functions,
> which can be a bear to troubleshoot with CouchDB or PouchDB.
>   2.  Although it's trivial, couchapp makes it very easy to update design
> docs repeatedly.
>   3.  Couchapp makes it easier to bundle dependencies as part of your
> design document (basically by deploying them to a "lib" or similar
> directory within a design document's directory).
> That being said, couchapp is something I have only relied on reluctantly.
> I don't use it in tests, for example, as it seems a bit much to me to
> introduce a whole stack of dependencies (python, required libraries) just
> to solve what's a fairly simple use case.  Antranig and I have previously
> talked about building a Fluid component to assist with creating design
> documents, and it seems like a good time to broaden and refresh that
> discussion.
>
> Regardless of whomever else is interested, I would be very happy to work
> together with you (and I'm assuming Antranig) on a ticket outlining the use
> cases in more detail.  Depending on the scope of what we agree is needed, I
> might end up simply building it the next time I need to work on a batch of
> design documents.  I'd also be very happy to help review something someone
> else builds and test that in my own work.
>
> Cheers,
>
>
> Tony
>
> On Thu, Aug 24, 2017 at 4:16 PM, Harnum, Alan <aharnum at ocadu.ca<mailto:
> aharnum at ocadu.ca>> wrote:
> Hi all,
>
> We (Greg Moss & myself) are curious to know the opinions of others who've
> worked with it (especially in the context of Infusion / Kettle) about how
> to best manage design documents when using CouchDB as part of an
> application.
>
> Specifically, what approaches have worked for externalizing design
> document code, ensuring the database has the latest design document code,
> etc?
>
> I know one example of doing this is in gpii-oauth2-datastore, which is a
> JSON externalization of the views that is imported to the DB as part of
> application provisioning, described at: https://github.com/GPII/
> universal/tree/master/gpii/node_modules/gpii-oauth2/gpii-
> oauth2-datastore/dbViews
>
> I am wondering if people have seen or considered other approaches.
>
> My own feeling is that it would be useful to be able to model an expected
> CouchDB database design document as an Infusion component, with the
> component taking responsibility for ensuring a set of view functions are
> present in the specified DB, but I don't feel a great certainty about this.
>
> Curious as to the thoughts of others!
>
> Alan
>
> ALAN HARNUM
> SENIOR INCLUSIVE DEVELOPER
> INCLUSIVE DESIGN RESEARCH CENTRE, OCAD UNIVERSITY
>
> E aharnum at ocadu.ca<mailto://aharnum@ocadu.ca>
>
> OCAD UNIVERSITY
> 100 McCaul Street, Toronto, Canada, M5T 1W1
> www.ocadu.ca<http://ocadu.ca/>
>
> _______________________________________________________
> fluid-work mailing list - fluid-work at lists.idrc.ocad.ca<mailto:
> fluid-work at lists.idrc.ocad.ca>
> To unsubscribe, change settings or access archives,
> see http://lists.idrc.ocad.ca/mailman/listinfo/fluid-work
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idrc.ocad.ca/pipermail/fluid-work/attachments/20170829/baf3619b/attachment.html>


More information about the fluid-work mailing list