CouchDB - managing design documents?

Tony Atkins tony at raisingthefloor.org
Wed Aug 30 16:45:29 UTC 2017


Hi, All.

Monday the 11th looks like the only day that works for everyone.  I am
waiting to chat with Colin on his return before picking the time slot.
Please hold 15:00-17:30 CET on that date for now, I will get back with the
final time on the 4th.

Cheers,


Tony

On Tue, Aug 29, 2017 at 11:14 AM, Tony Atkins <tony at raisingthefloor.org>
wrote:

> Hi, Simon:
>
> That's a fair point, at a minimum we should be aware of the need and see
> what if any ground work we can take care of in working on managing design
> documents.
>
> As far as I know we do not currently have a general solution to managing
> data migrations.  In the rare cases when I've needed to restructure, I've
> used a bulk read, transform, and bulk update, but obviously that's only a
> very temporary solution.
>
> At the risk of entangling this conversation with another fairly big topic,
> one idea that comes to mind is using versioned JSON schemas and some
> convention of indicating what schema is used for a particular record (for
> example, a URI pointing to the schema).  That would let us:
>
>    1. Detect existing corrupt or incomplete records.
>    2. Limit a migration to a particular type and/or version of a record.
>    3. Verify that the results of the migration are themselves valid.
>    4. Determine which records have already been migrated, and which have
>    not.
>
> Anyway, we should bring up the topic in the architecture meeting and set
> up an initial meeting in the next week or so.  With that in mind, I have
> created a Doodle Poll:
>
> https://doodle.com/poll/snpxb4f496g289pk
>
> Cheers,
>
>
> Tony
>
> On Mon, Aug 28, 2017 at 5:31 PM, Bates, Simon <sbates at ocadu.ca> wrote:
>
>> To broaden this topic slightly, I'm thinking that some form of data
>> migration automation would also be important to have, in addition to
>> updating design documents. For example, if a document structure is
>> modified, we would want to have an automated way of updating existing data
>> to the new structure.
>>
>> Do we have anything currently existing to manage data migration and
>> document structure evolution? If not, I'm thinking it would be worth
>> including the topic in discussions on managing design documents.
>>
>> Simon
>>
>> ________________________________________
>> From: fluid-work <fluid-work-bounces at lists.idrc.ocad.ca> on behalf of
>> Harnum, Alan <aharnum at ocadu.ca>
>> Sent: August 28, 2017 11:19:59 AM
>> To: Tony Atkins
>> Cc: Fluid Work
>> Subject: Re: CouchDB - managing design documents?
>>
>> Hi Tony,
>>
>> Thanks for the replies – we’ve also been looking at the various Couchapp
>> implementations, but have been hesitant both for what you cite below (large
>> additional stack of dependencies) and this part of the current CouchDB
>> docs: http://docs.couchdb.org/en/2.1.0/ddocs/, which discourages the
>> previously-promoted “CouchApps” paradigm that the various CouchApp tools
>> are designed to use. My sense is that the various Couchapp projects are not
>> under terribly active development in light of this change in CouchDB’s
>> latest documentation.
>>
>> I’d also be an advocate of a Fluid component for capturing views and
>> perhaps other configuration aspects of an Infusion application that intends
>> to make use of CouchDB - I actually did some experimentation in this
>> direction last Friday, which I’ve now cleaned up a bit and added some
>> documentation: https://github.com/waharnum/couch-config
>>
>> In my ideal world I think I’d be able to write a CouchDB configuration
>> for an application in pure Infusion rather than pushing JSON around with
>> curl or other tools.
>>
>> From: Tony Atkins <tony at raisingthefloor.org>
>> Date: Monday, August 28, 2017 at 7:32 AM
>> To: "Harnum, Alan" <aharnum at ocadu.ca>
>> Cc: Fluid Work <fluid-work at fluidproject.org>
>> Subject: Re: CouchDB - managing design documents?
>>
>> Hi, Alan:
>>
>> Thanks again for starting yet another interesting and important
>> discussion.  I've had to do a lot of this in my own work, and have
>> alternated between:
>>
>>   1.  Storing the design documents as raw JSON (mainly for testing).
>> This is similar to the approach used in the example you shared.
>>   2.  Writing design documents using couchapp<https://github.com/co
>> uchapp/couchapp>.
>> The first approach seems easy enough, but is a bit tedious on a few
>> fronts:
>>
>>   1.  It's difficult to write even moderately complex functions as
>> escaped strings.
>>   2.  Replacing an existing design document is a bit tedious, as it
>> involves either looking up the current document's _rev or DELETEing and
>> then POSTing the replacement.
>>   3.  Although it's possible to require libraries from a design document<
>> http://docs.couchdb.org/en/2.0.0/query-server/javascript.html#require>,
>> this is pretty hard to manage using raw JSON files.
>> Couchapp makes each of these easier:
>>
>>   1.  You just save your functions to a file, which you can run through
>> our standard linting and more easily check for syntax errors.  You can even
>> write unit tests against the functions you plan to use in your design
>> documents.  This is especially helpful when developing reduce functions,
>> which can be a bear to troubleshoot with CouchDB or PouchDB.
>>   2.  Although it's trivial, couchapp makes it very easy to update design
>> docs repeatedly.
>>   3.  Couchapp makes it easier to bundle dependencies as part of your
>> design document (basically by deploying them to a "lib" or similar
>> directory within a design document's directory).
>> That being said, couchapp is something I have only relied on
>> reluctantly.  I don't use it in tests, for example, as it seems a bit much
>> to me to introduce a whole stack of dependencies (python, required
>> libraries) just to solve what's a fairly simple use case.  Antranig and I
>> have previously talked about building a Fluid component to assist with
>> creating design documents, and it seems like a good time to broaden and
>> refresh that discussion.
>>
>> Regardless of whomever else is interested, I would be very happy to work
>> together with you (and I'm assuming Antranig) on a ticket outlining the use
>> cases in more detail.  Depending on the scope of what we agree is needed, I
>> might end up simply building it the next time I need to work on a batch of
>> design documents.  I'd also be very happy to help review something someone
>> else builds and test that in my own work.
>>
>> Cheers,
>>
>>
>> Tony
>>
>> On Thu, Aug 24, 2017 at 4:16 PM, Harnum, Alan <aharnum at ocadu.ca<mailto:
>> aharnum at ocadu.ca>> wrote:
>> Hi all,
>>
>> We (Greg Moss & myself) are curious to know the opinions of others who've
>> worked with it (especially in the context of Infusion / Kettle) about how
>> to best manage design documents when using CouchDB as part of an
>> application.
>>
>> Specifically, what approaches have worked for externalizing design
>> document code, ensuring the database has the latest design document code,
>> etc?
>>
>> I know one example of doing this is in gpii-oauth2-datastore, which is a
>> JSON externalization of the views that is imported to the DB as part of
>> application provisioning, described at: https://github.com/GPII/univer
>> sal/tree/master/gpii/node_modules/gpii-oauth2/gpii-oauth
>> 2-datastore/dbViews
>>
>> I am wondering if people have seen or considered other approaches.
>>
>> My own feeling is that it would be useful to be able to model an expected
>> CouchDB database design document as an Infusion component, with the
>> component taking responsibility for ensuring a set of view functions are
>> present in the specified DB, but I don't feel a great certainty about this.
>>
>> Curious as to the thoughts of others!
>>
>> Alan
>>
>> ALAN HARNUM
>> SENIOR INCLUSIVE DEVELOPER
>> INCLUSIVE DESIGN RESEARCH CENTRE, OCAD UNIVERSITY
>>
>> E aharnum at ocadu.ca<mailto://aharnum@ocadu.ca>
>>
>> OCAD UNIVERSITY
>> 100 McCaul Street, Toronto, Canada, M5T 1W1
>> www.ocadu.ca<http://ocadu.ca/>
>>
>> _______________________________________________________
>> fluid-work mailing list - fluid-work at lists.idrc.ocad.ca<mailto:
>> fluid-work at lists.idrc.ocad.ca>
>> To unsubscribe, change settings or access archives,
>> see http://lists.idrc.ocad.ca/mailman/listinfo/fluid-work
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idrc.ocad.ca/pipermail/fluid-work/attachments/20170830/c920026c/attachment.html>


More information about the fluid-work mailing list