Proposal: better intermediary means of managing external front-end Javascript dependencies in Infusion-based projects using nom + grunt

Tony Atkins tony at raisingthefloor.org
Thu Mar 3 16:50:57 UTC 2016 

Hi, Alan:

Thanks a lot for the links.  This is exactly the kind of input I was hoping
to get from the group.

"npm plus something" does seem like an approach people are following, and
it's a reasonable thing to try.  I'd like to see what approach you follow
in distinguishing client-side dependencies from server-side ones in
particular.  The blog you linked to basically suggests solving this using
node-browserify and the "browser" section of a package.json file that it
adds support for.  If the packages we depend on on the client side are
trending towards this approach, then it could be enough to build the router
I have in mind on top of.

I'd also like to see how you manage to cleanly refer to your dependencies.
In other words, a package that depends on yours should be able to follow
the same strategy you do and end up with a combined set of dependencies
that does not contain two copies of any dependency.

If you already have an idea of how you plan to address those issues or some
sample code, please share.

Thanks,


Tony



On Thu, Mar 3, 2016 at 4:02 PM, Harnum, Alan <aharnum at ocadu.ca> wrote:

> Hi Tony,
>
> I believe there have been some specific concerns about Bower in the last
> several months raised by others – I don’t have the full history on that but
> I think it’s part of the general debate in the Javascript community about
> Bower’s future, especially on these two issues:
>
>    - a sense of turn in certain parts of the community against relying on
>    Bower in favour of npm + additional tooling where necessary:
>    https://gofore.com/stop-using-bower/
>    - a sense that Bower as a project may be hitting a wall in terms of
>    contributors and becoming legacy:
>    https://github.com/reactjs/redux/issues/944#issuecomment-154858804
>
> I don’t have enough experience with Bower (or front end dependency
> management in general) to claim a deep understanding – whether this is
> really where the zeitgeist is going, or if this is one of those perpetual
> debates that occur in open source.
>
> Here’s the start of two conversations from the #fluid-work channel that
> may also be relevant to this discussion:
>
> -
> https://botbot.me/freenode/fluid-work/2016-03-01/?msg=61306744&amp=&page=3
> - https://botbot.me/freenode/fluid-work/2016-03-02/?msg=61397389&page=2
>
> My sense of moving forward for my particular use case is that I’ll use
> straight npm + grunt (if necessary); that said, one of the many nice things
> about how we work as a community is being able to try different approaches
> to the same problem when the space is tumultuous or uncertain (as I feel JS
> front-end dependency management is, and perhaps always will be).
>
> *ALAN HARNUM*
> SENIOR INCLUSIVE DEVELOPER
> INCLUSIVE DESIGN RESEARCH CENTRE, OCAD UNIVERSITY
>
> *E *aharnum at ocadu.ca <//aharnum at ocadu.ca>
>
> *OCAD UNIVERSITY*
> 100 McCaul Street, Toronto, Canada, M5T 1W1
> www.ocadu.ca <http://ocadu.ca/>
>
> From: fluid-work <fluid-work-bounces at lists.idrc.ocad.ca> on behalf of
> Tony Atkins <tony at raisingthefloor.org>
> Date: Thursday, March 3, 2016 at 4:15 AM
> To: "fluid-work at lists.idrc.ocad.ca" <fluid-work at lists.idrc.ocad.ca>
> Subject: Re: Proposal: better intermediary means of managing external
> front-end Javascript dependencies in Infusion-based projects using nom +
> grunt
>
> Hi, All:
>
> I did a bit more research this morning.  There are a few libraries that
> will let you use node to tree through bower dependencies and produce a full
> list of full file paths.  Here are the two that seem closest to what we
> need:
>
> https://github.com/ck86/main-bower-files
> https://github.com/ksmithut/bower-files
>
> The second has a lot more features and seems a bit more polished, but
> either would work well as the guts of the "modules" router I'm thinking
> of.  My idea is that the *gpii-express-module* router would use one of
> the above libraries to deliver a content bundle, as follows:
>
>    1. A request for */modules/that* would return the default (javascript)
>    content for the outermost package and its dependencies.
>    2. A request for */modules/that/:extension* would return just the
>    content matching an extension, for example, "css".
>    3. A request for */modules/:package *would return the javascript
>    content for a specific one of our dependencies.
>    4. A request for */modules/:package/:extension* would return just the
>    package content that matches the extension.
>
> This could reduce all client-side dependencies to HTML like:
>
> <script src="/modules/that"></script>
>> <link rel="stylesheet" href="/modules/that/css"></link>
>
>
> If this seems crazy or misses glaring problems, please scream in the next
> few days.    Otherwise, I was thinking of prototyping this for my own next
> round of client-side coding on the UL as early as next week. I would write
> up the full specs for the router, share that with the group, and shortly
> thereafter, build the module out at least enough to start exercising it in
> my tests.
>
> To use this approach, we would need to create and maintain bower.json
> files for our packages.  I would also create a PR that adds a bower.json
> for infusion.
>
> Cheers,
>
>
> Tony
>
>
> On Wed, Mar 2, 2016 at 9:26 PM, Tony Atkins <tony at raisingthefloor.org>
> wrote:
>
>> Hi, All:
>>
>> At the risk of offering to address any particular problem with a hammer I
>> happen to be familiar with, I was thinking about making a gpii-express
>> router for something like this earlier.  I already commonly use a wrapper
>> around the express "static" router to serve up content from node_modules,
>> at least in most of my tests.
>>
>> Instead, we could make a router that handles /modules/:path, where
>> /modules is a static prefix and :path is a string that can be resolved to a
>> filesystem path using fluid.module.resolvePath, as in
>> "%gpii-binder/src/js/binder.js".  That would mirror a lot of the recent
>> server-side work I've done to migrate to using package-relative paths, and
>> does at least simplify things.  This would mean we'd have the same number
>> client-side includes, just with less brittle paths.  However, it's
>> achievable with almost no work.
>>
>> I've also considered making a router that was a bit smarter, that could
>> deliver minified or otherwise compressed bundles with all client-side
>> dependencies in a single go. To make that work, we'd need some way to
>> represent the chain of dependencies.
>>
>> One way would be to add a new fluid.registerBundle top-level function
>> that associates package names with arrays of resolvable paths.  Another
>> would be to tree through the "main" elements in the bower.json files for
>> each package and its dependencies.  Either strategy would allow us to use a
>> single include, something like:
>>
>> <script src="/modules/my-module"></script>
>>
>> With either approach, we would ultimately deliver a single (optionally
>> compressed) bundle with all the relevant content, complete with etags and
>> other trimmings so that it can be cached properly.
>>
>> Anyway, if that sounds somewhat non-crazy, I would be happy to set up a
>> time with a vikingpad and GoToMeeting and sketch things out with anyone
>> who's interested.
>>
>> Cheers,
>>
>>
>> Tony
>>
>>
>> On Wed, Mar 2, 2016 at 3:22 PM, Harnum, Alan <aharnum at ocadu.ca> wrote:
>>
>>> Justin, that would be the case, yes - I’m not sure how I feel about that
>>> myself, looking at the size of the node_modules directory produced for the
>>> chartAuthoring repo by an ‘npm install’. In the simplest case, we’d be
>>> serving a static site with a whole bunch of node_modules directories that
>>> aren’t necessary to its functioning in the clear.
>>>
>>> I’ve been weighing the simplicity of this approach (which I like) vs.
>>> the added additional complexity (which is not huge, but is there) of having
>>> ‘grunt copy:frontEndDependencies’ or similar as a second build step for
>>> deploying the static build site or setting up for local development, and I
>>> think I’m on the side of having grunt in the mix. I worry about the
>>> potential security implications of having so many node_modules served in
>>> the clear, even statically.
>>>
>>> What I mostly want is something to move forward with that doesn’t
>>> require investment in build tooling beyond npm and grunt, and is also
>>> straightforward to automate for deployment.
>>>
>>> Alan
>>>
>>> On Mar 2, 2016, at 8:27 AM, Justin Obara <obara.justin at gmail.com> wrote:
>>>
>>> Does this mean that for something like the build site, we’ll have to
>>> server the node_modules repo and/or run it through kettle?
>>>
>>> Thanks
>>> Justin
>>>
>>>
>>> On March 1, 2016 at 2:11:20 PM, Harnum, Alan (aharnum at ocadu.ca) wrote:
>>>
>>> Antranig, I do like the idea of the basic use case being “use the
>>> front-end packages in the node structure as far as possible” - this would
>>> work just fine for the Chart Authoring case or other cases where the goal
>>> is basically the production of a classic “static” site. My sense is that
>>> the NPM folks are likely to produce some tooling in the short-medium around
>>> this, based on this post:
>>> http://blog.npmjs.org/post/101775448305/npm-and-front-end-packaging -
>>> hence my desire not to wed ourselves too closely to any particular tooling.
>>>
>>> The other use case that comes to mind right now is the production of
>>> “widget” Javascript files (for lack of a better term) where the goal is to
>>> provide a single static Javascript file that can be used by a site to embed
>>> Infusion-based components (that might also include external dependencies
>>> like D3 or Flocking) without knowing anything about Infusion. This is the
>>> situation we’re in approaching the front-end components for the QI, which
>>> we want to be straightforwardly embeddable in the (currently Drupal-based,
>>> to my knowledge) Dev Space site or in other contexts. Issue at
>>> https://issues.gpii.net/browse/GPII-1583 has a bit more detail on this.
>>>
>>> Per that use case, I actually wasn’t aware of the Infusion module
>>> registration scheme (my knowledge on Infusion is pretty much front-end only
>>> at this point) , but this may be relevant to the work that Avtar and I are
>>> doing for the QI. Under that approach (if I’m understanding correctly), the
>>> QI service could handle the inclusion of any necessary external libraries
>>> into an overall JSONP-based payload for the client site after the initial
>>> dependency pull via npm. Make sense?
>>>
>>> *ALAN HARNUM*
>>> SENIOR INCLUSIVE DEVELOPER
>>> INCLUSIVE DESIGN RESEARCH CENTRE, OCAD UNIVERSITY
>>>
>>> *E *aharnum at ocadu.ca <//aharnum at ocadu.ca>
>>>
>>> On Mar 1, 2016, at 1:49 PM, Antranig Basman <
>>> antranig.basman at colorado.edu> wrote:
>>>
>>> Cheers, Alan - this is a really important and strategic issue and I
>>> wanted to thank you for organising the discussion in such a clear way.
>>>
>>> I think the basic plan here is highly desirable - to move towards npm
>>> infrastructure and package.json as our central solution for managing JS
>>> dependencies, and to use browserify when we need to build single-file
>>> distribution bundles.
>>>
>>> I am wondering whether we can simplify the steps in the proposal yet
>>> further - that is, whether we can cut out the grunt tasks and ignore
>>> structure, and as far as possibly simply use the node modules in place
>>> where they are fetched using npm.
>>>
>>> You may be aware that every Infusion-aware npm module now participates
>>> in its "module registration scheme" kicked off from fluid.module.register -
>>>
>>> http://docs.fluidproject.org/infusion/development/NodeAPI.html#fluid-module-register-name-basedir-modulerequire-
>>>  - this makes it easy to resolve paths into foreign modules by simple
>>> expressions like "%flocking/src/stuff" etc.
>>>
>>> Kettle's "static middleware" could host these directories using these
>>> symbolic references directly - or even automatically, if we add extra
>>> directives to package.json or elsewhere -
>>>
>>> https://github.com/amb26/kettle/tree/KETTLE-32#built-in-standard-middleware-bundled-with-kettle
>>>
>>> Other points in our future roadmap might include BRFS or something like
>>> it to bridge the gap between the view of static assets from the point of
>>> view of node and the browser - https://github.com/substack/brfs - or
>>> little things like "staticify" https://github.com/petehunt/staticify -
>>> of which there seem to be a dozen.
>>>
>>> Does this make sense/fit in with the use cases and road map you imagine?
>>> I'm keen to have something that involves us in as little build faff and
>>> boilerplate as possible, based on the npm infrastructure but with as many
>>> of the affordances of bower as possible - what do you think? Are there some
>>> "hard core use cases" that we can't easily meet with the files in their
>>> "native position" inside node_modules directories?
>>>
>>> Cheers,
>>>
>>> Antranig
>>>
>>> On 01/03/2016 18:33, Harnum, Alan wrote:
>>>
>>> *Background*
>>> *
>>> *
>>> The Chart Authoring Tool (
>>> https://github.com/fluid-project/chartAuthoring) is built with
>>> Infusion, the D3
>>> data visualization library, and the Flocking audio synthesis framework
>>> (created by the Fluid community's own
>>> Colin Clark).
>>>
>>> At the moment the non-Infusion dependencies are directly included as
>>> files in the project repository. This
>>> is simple, but bad in various other ways in terms of keeping things up
>>> to date, keeping the repo clean, etc.
>>>
>>> There has been an outstanding issue since last August to improve this
>>> situation
>>> (
>>> https://issues.fluidproject.org/browse/FLOE-405?jql=text%20~%20%22bower%22
>>> <https://issues.fluidproject.org/browse/FLOE-405?jql=text ~ "bower">).
>>>
>>> Originally the thought was to use Bower, but Bower seems to be facing
>>> some challenges as a project
>>> (
>>> https://www.reddit.com/r/javascript/comments/3th4v6/bower_vs_npm_3_frontend_package_management/)
>>> and the
>>> general direction of many other projects seems to be towards using
>>> straight npm (sometimes with a tool like
>>> browserify) to manage front-end dependencies.
>>>
>>> Infusion itself is moving in this direction as NPM becomes a more viable
>>> for front-end management, as
>>> evidenced by this issue:
>>> https://issues.fluidproject.org/browse/FLUID-5745?jql=text%20~%20%22bower%22
>>> <https://issues.fluidproject.org/browse/FLUID-5745?jql=text ~ "bower">
>>>
>>> *My Immediate Goal / Challenge*
>>>
>>> I'm starting work on front-end Infusion-based components for the GPII
>>> Quality Infrastructure where I would
>>> like to be able to reuse some code from the Chart Authoring repo itself,
>>> so rather than replicate the
>>> problem again I’d like to have an approach that does the following:
>>>
>>> - works with the existing toolstack for builds (npm + grunt) without
>>> other tool dependencies such as
>>> browserify, webpack, etc
>>> - produces a suitable static “package" for serving publicly, as is used
>>> currently by
>>> http://build.fluidproject.org/
>>> - has minimal development overhead but improves the current situation
>>> - can be replaced easily enough in the future when we align on a better
>>> means of managing front-end dependencies
>>>
>>> *What I Propose*
>>> *
>>> *
>>> Examples here are in the context of the Chart Authoring Tool project in
>>> my fork at
>>> https://github.com/waharnum/chartAuthoring/blob/FLOE-405/ but I’d like
>>> the general shape of this to be
>>> applicable to other cases:
>>>
>>> - front-end dependencies should be managed using NPM’s /package.json/
>>> format and installed via a standard
>>> ‘npm install’ (
>>> https://github.com/waharnum/chartAuthoring/blob/FLOE-405/package.json)
>>> - one or more grunt tasks should then be used to copy these dependencies
>>> into an appropriate structure for
>>> use by the front end project (
>>> https://github.com/waharnum/chartAuthoring/blob/FLOE-405/Gruntfile.js)
>>> - a .gitignore definition should be used to make sure this external
>>> dependency directory is not versioned
>>> (https://github.com/waharnum/chartAuthoring/blob/FLOE-405/.gitignore)
>>>
>>> Basically, a front-end project will pull in its dependencies via NPM and
>>> then distribute them as needed via
>>> grunt to create the necessary “product"; the specifics of this will
>>> depend on the particular project. We
>>> already use both npm and grunt in other contexts, they’re familiar to
>>> us, and they avoid the need (which we
>>> have, but longer term) to adopt a specific additional piece of tooling
>>> for front-end dependency management,
>>> which is a space that seems in heavy churn right now in the Javascript
>>> community.
>>>
>>> We talked about this approach briefly at the Floe planning meeting
>>> yesterday and people seemed receptive,
>>> but the broader community should weigh in and give their opinion.
>>>
>>> *Questions*
>>> *
>>> *
>>> 1) Do we feel OK with this approach to front-end dependency management
>>> in the short term?
>>> 2) If we don’t feel OK about it, what should we do instead?
>>>
>>> *ALAN HARNUM*
>>> SENIOR INCLUSIVE DEVELOPER
>>> INCLUSIVE DESIGN RESEARCH CENTRE, OCAD UNIVERSITY
>>>
>>> *E *aharnum at ocadu.ca <mailto://aharnum@ocadu.ca>
>>>
>>>
>>>
>>> _______________________________________________________
>>> fluid-work mailing list - fluid-work at lists.idrc.ocad.ca
>>> To unsubscribe, change settings or access archives,
>>> see http://lists.idrc.ocad.ca/mailman/listinfo/fluid-work
>>>
>>>
>>> _______________________________________________________
>>> fluid-work mailing list - fluid-work at lists.idrc.ocad.ca
>>> To unsubscribe, change settings or access archives,
>>> see http://lists.idrc.ocad.ca/mailman/listinfo/fluid-work
>>>
>>>
>>> _______________________________________________________
>>> fluid-work mailing list - fluid-work at lists.idrc.ocad.ca
>>> To unsubscribe, change settings or access archives,
>>> see http://lists.idrc.ocad.ca/mailman/listinfo/fluid-work
>>>
>>> _______________________________________________________
>>> fluid-work mailing list - fluid-work at lists.idrc.ocad.ca
>>> To unsubscribe, change settings or access archives,
>>> see http://lists.idrc.ocad.ca/mailman/listinfo/fluid-work
>>>
>>>
>>>
>>> _______________________________________________________
>>> fluid-work mailing list - fluid-work at lists.idrc.ocad.ca
>>> To unsubscribe, change settings or access archives,
>>> see http://lists.idrc.ocad.ca/mailman/listinfo/fluid-work
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.idrc.ocad.ca/pipermail/fluid-work/attachments/20160303/ffce5b52/attachment.htm>

More information about the fluid-work mailing list