Engage directory structure

[Colin Clark]

Hey all,

Michelle and I talked through all the deployment issues of Engage and  
Kettle. Here's a quick summary, along with a proposed directory  
structure.

Assumptions:
  * Maven is not awesome for build scripts, but it is awesome for  
dependency resolution
  * Our Infusion WAR file is somewhat broken, in that it has all our  
main directories at top level, instead of being nested inside an  
infusion/ directory. This was a regression when we restructured SVN  
for the Infusion 1.0 release.

Our approach for building Engage:
  * Don't mess with being able to easily run and rapidly develop  
inside Eclipse with Jetty
  * Create separate kettleIncludes.json files, one for in Eclipse, one  
for standalone deployment
  * Don't worry about separating Kettle from the Engage server code at  
the moment
  * Build a good, self-contained, production-ready WAR for Engage  
containing the following:
     1. Infusion
     2. Engage client
     3. Engage Server and Kettle
  * Write an Engage Ant build script that takes care of minification  
and preparing the correct WAR directory structure
  * Use the Ant script to swap the Eclipse includes for the stand- 
alone includes
  * Use Maven to build the final WAR and resolve any Java dependencies

Proposed directory structure (this is extremely similar to what we've  
already got:

src/
   webapp/
      infusion/  --> copied/exploded from the product of the Infusion  
Ant build
         components/
         framework/
         lib/
      engage-client/  --> minified and copied into place
         components
      engage-server/
         kettle/
           js/
         artifact/
           js/
         browse/
           js/
   java/
      org/
        fluidproject/
          kettle/
            KettleServlet.java
            .. etc ...
            jetty/
              JettyLauncher.java
              ... etc ...

We'll also want to restructure the engage-client module to resemble  
the structure we use for Infusion.

Seem reasonable?

Colin

On 23-Sep-09, at 9:41 AM, Michelle D'Souza wrote:

> I'm glad this conversation has started - these are exactly the types  
> of things I was hoping would come out.
>
> The structure I was proposing was the svn structure for engage.  
> Since we work of the 'fluid-all' externals project, this structure  
> would be like a sibling to Infusion. So right now, the 'lib' folder  
> may not hold anything but in an Infusion like way, as soon as we add  
> third party dependencies, these would be in 'lib'.  Another note on  
> this structure is that 'kettle' should probably be called 'engage- 
> server'. I'm not sure what it would look like inside the 'engage- 
> server' directory - perhaps someone could take a stab at this?
>
> Does this make sense? Does it still support being able to build a  
> deployable WAR of engage easily?
>
> Michelle
>
>
> On 22-Sep-09, at 5:40 PM, antranig at caret.cam.ac.uk wrote:
>
>> Hi there - I think this looks reasonable, but I think it will be  
>> hard to make
>> any firm decisions "at 10,000 feet" (without a plane!) without  
>> working
>> through some of the fine-grained issues that this brings up. An  
>> important
>> issue is what kind of separation we expect between the parts of  
>> kettle,
>> for example, which are dependent on the Java infrastructure (or  
>> even worse,
>> *are* the Java infrastructure) and those parts which are pure JS.
>> Ominously, of course, this is the kind of issue that depends on us  
>> having
>> stabilised some kind of reasonable IoC infrastructure, though that  
>> shouldn't
>> prevent us making some kind of interim decision now.
>>
>> What would the "lib" directory contain? Infusion?
>> And if we had a "services" webapp, what would go in it?
>> We should bear in mind that if we are planning for this to coexist  
>> in a
>> M2 build structure it needs to work sensibly with respect to "WAR  
>> composition"
>> whereby a composite WAR image is build up, in effect, by simply  
>> unzipping
>> all the WARs into a common directory and then zipping them up again.
>>
>> In "the bad days" this was a problem given that it meant that paths  
>> at
>> development time would differ from those at deploy time - but this  
>> kind
>> of slack can now be taken up by "mounts". The problem we do have  
>> remaining
>> is to make sure that somehow that the component WARs do not contain
>> contents that collide. This can be awkward, especially since M2 often
>> insists that they have some kind of web.xml file ...
>>
>> Quoting Yura Zenevich <yura.zenevich at gmail.com>:
>>
>>> I was just wondering what kettle directory would mean inside this  
>>> structure.
>>> Would it include all kettle server infrastructure and all the  
>>> javascript
>>> apps, or these two would be defined somehow separately?
>>>
>>> Regards,
>>>
>>> Yura
>>>
>>> On Sep 22, 2009 5:22 PM, "Colin Clark" <colinbdclark at gmail.com>  
>>> wrote:
>>>
>>> Hi Michelle and everyone else,
>>>
>>> On 21-Sep-09, at 3:44 PM, Michelle D'Souza wrote:
>>>
>>> Hi everyone,
>>>>
>>>> As we approach the 0.1 release of Engage we need to decide on the  
>>>> directory
>>>> structure of the engage trunk. Here is a suggestion which is  
>>>> based on the
>>>> Infusion structure. Please comment and suggest alternatives. One  
>>>> question I
>>>> have - should there be a services directory? What would go in  
>>>> there?
>>>>
>>>> fluid
>>>>      engage
>>>>              trunk
>>>>                      src
>>>>                              webapp
>>>>                                      components
>>>>                                      demos
>>>>                                      kettle
>>>>                                      lib
>>>>                                      tests
>>>>
>>>
>>> How does this sit with the standard directory structure expected  
>>> by Maven 2?
>>>
>>> I'd love to hear from Antranig and Yura about this proposed  
>>> directory
>>> structure and if it will suit our needs.
>>>
>>> In the end, our goal is to make building Engage as simple as  
>>> possible. Since
>>> it's currently based on Kettle's JVM-based approach, Maven is  
>>> probably the
>>> way this will be built. If we need to add an Ant script for other
>>> build-related tasks, we can look at that, too.
>>>
>>> Colin
>>>
>>> ---
>>> Colin Clark
>>> Technical Lead, Fluid Project
>>> http://fluidproject.org
>>>
>>
>>
>>
>>
>> ----------------------------------------------------------------
>> This message was sent using IMP, the Internet Messaging Program.
>>
>
> ------------------------------------------------------
> Michelle D'Souza
> Software Developer, Fluid Project
> Adaptive Technology Resource Centre
> University of Toronto
>
>
>

---
Colin Clark
Technical Lead, Fluid Project
http://fluidproject.org