Regarding project "Refactor and Automate Infusion Documentation"

Justin Obara obara.justin at gmail.com
Mon Mar 9 13:21:23 UTC 2020


Hi Yash,

Somewhere along the line we lost our CC to the fluid-work mailing list. I’ve added that back in now so that others can follow along and provide input as needed.

Another tool to look at would be https://docusaurus.io <https://docusaurus.io/>. It would be good to also investigate if there are other options/solutions that we haven’t yet mentioned. 

I’ve left some more comments below.

Thanks
Justin

> On Mar 8, 2020, at 1:16 AM, Yash Jipkate <yashjipkate at gmail.com> wrote:
> 
> Hi Justin,
> 
> I just got a reply <https://github.com/11ty/eleventy/issues/1009#issuecomment-596124419> from the 11ty team on the issue that I opened. 11ty has directory specific data files that can be used to define config to be shared to all the lower levels. The major downside of this is that the directory specific data file needs to be placed in the directory itself. Therefore, the data file will be directly in between the content files.

You say that the data file can define configuration that can be shared by lower levels. So you could have two levels of directories, one that contains the data file and a directory of all the docs. In that way you wouldn’t haven’t worry about mixing them together. But I’d have to see how this looks in a more concrete example.

> According to my plan, the content repo was to be cloned in the SSG repo, so if 11ty is to be used then after the cloning we would need to further copy the directory specific data files into respective folders to make them useful.
> 
> I guess this should be counted as a downside for 11ty since this extra copying operation would require an extra script to maintain and would cause a closer coupling. 

You’d still need some sort of script to do the cloning, so not sure how much extra work it would be. Maybe I’m forgetting or missing something with how you were planning to do the clone.

> 
> I am still inclined to HUGO because of our use case. Lesser coupling, faster build times (since there are multiple steps like dispatch and cloning and that would add to the build time) and a file structure convention are some features that outperform 11ty. Some star features of 11ty like multiple templates are not required by us since we use only MD files.
> 
> What do you think?

Addressing some of the comments/questions from earlier in the thread. Not sure what the issue about Hugo being written in Go. Could you explain and expand on that one more.

Regarding flexibility. That’s not necessarily a bad thing. If we maintain consistency in our use and follow best practices, I don’t think it should be a matter for us and could mean that we can tailor it to our own needs. If you know of a specific issue that 11ty’s flexibility is causing, or could cause, it would be good to discuss it more. 

> 
> Thanks
> Yash
> 
> On Sat, Mar 7, 2020 at 11:59 PM Yash Jipkate <yashjipkate at gmail.com <mailto:yashjipkate at gmail.com>> wrote:
> 
> Hi Justin,
> 
> I looked up 11ty more and found the following advantages over HUGO:
> 11ty (2017) is newer than HUGO (2013).
> 11ty is built using JavaScript whereas HUGO is built using Go. JS being a native web language won't ever lose support whereas Go being a third party might lose support.
> 11ty has more flexibility regarding file extensions. It works with formats other than MD as well.
> 11ty has a flexible file system. So we can build it right from the current file structure.
> I also liked the documentation of 11ty compared to the one by HUGO.
> But there is a major downside that since 11ty uses a flexible directory structure, every page needs to have a front matter that specifies the template it needs to use. HUGO being strict about directory structure automates this process and is useful when we want to separate the content from the front matter.
> 
> I found this <https://github.com/11ty/eleventy/issues/55#> issue to fulfill our use case but it was closed with a note stating it was solved in a version but I couldn't find any documentation regarding it. Therefore I opened an 'education' type issue <https://github.com/11ty/eleventy/issues/1009> to get their response.
> 
> If this issue persists, then we might not be able to use 11ty for our purpose.
> 
> Thanks
> Yash
> 
> On Sat, Mar 7, 2020 at 12:28 AM Yash Jipkate <yashjipkate at gmail.com <mailto:yashjipkate at gmail.com>> wrote:
> Hi Justin, 
> 
> Thanks for the reply. I have responded inline. 
> 
> Thanks. 
> Yash
> 
> On Thu, 5 Mar 2020, 7:02 pm Justin Obara, <obara.justin at gmail.com <mailto:obara.justin at gmail.com>> wrote:
> Hello Yash,
> 
> Please see comments inline below.
> 
> Thanks
> Justin
> 
>> On Mar 5, 2020, at 2:43 AM, Yash Jipkate <yashjipkate at gmail.com <mailto:yashjipkate at gmail.com>> wrote:
>> 
>> Hi Justin,
>> 
>> Just confirming whether you got my previous email. Is there something I should add or elaborate on?
>> 
>> Thanks
>> Yash
>> 
>> 
>> On Tue, Mar 3, 2020 at 3:14 AM Yash Jipkate <yashjipkate at gmail.com <mailto:yashjipkate at gmail.com>> wrote:
>> Hi Justin,
>> 
>> I will try to mark the issues on JIRA since it should have only legitimate issues.
>> Also, I tried out the repository communication. I created a dispatcher repo <https://github.com/YashJipkate/sample-repo-to-send-dispatch> and added a workflow to send a dispatch to another consumer <https://github.com/YashJipkate/sample-dispatch-event-consumer> repo created by me. It worked perfectly and the repo was consuming the event.
> How is the consumer repo affected by forks of the dispatcher repo. That is, if I fork your dispatcher repo and push to it, will it send a dispatch event to your consumer repo and trigger an action?
> 
> It’s also worth investigating other methods that do not couple the repos together so closely.
> 
> No, the dispatch event does not work on forks. I forked the dispatcher repo here <https://github.com/yashpart2/sample-repo-to-send-dispatch> and pushed to the main branch. This failed to trigger the workflow in the main consumer repo (you can check here <https://github.com/YashJipkate/sample-dispatch-event-consumer/actions>). Also forking the consumer repo won't have any downside since the dispatch event demands the name of the repo along with the owner so forks won't be affected.
> 
> Another method I initially thought before discovering repo dispatches was to have a special server for the sole purpose of sending and receiving events that would trigger builds. This would allow us to have custom features not limited by those offered by GitHub API. The downside is that we would need to maintain an additional repo and would not be able to access the community help that exists for GitHub's API. Also since out needs are satisfied by the GitHub's API, we won't need any other custom features than the ones already present.
>> I think the design would not undergo much change. I saw the official documentation page of HUGO <https://gohugo.io/documentation/> and they use pretty much similar design - a top nav, a side nav, and the main content.
>> The source code is open-sourced here <https://github.com/gohugoio/hugoDocs>. I looked at the code and saw that the menu bar items were hard-coded in a site-wide config file here <https://github.com/gohugoio/hugoDocs/blob/master/config/_default/menus/menus.en.toml>. The layout for the menu bars are placed in partials/ <https://github.com/gohugoio/hugoDocs/tree/master/_vendor/github.com/gohugoio/gohugoioTheme/layouts/partials> of the theme that HUGO's site uses.
>> To maintain the design of the current site, we would have to make a custom theme of our own (similar to the theme used by HUGO docs <https://github.com/gohugoio/hugoDocs/tree/master/_vendor/github.com/gohugoio/gohugoioTheme>).
>> The structure of the files would also need to be changed accordingly. The md files or the content files would be put under content/ and assets under static/.
>> The repo would be essentially HUGO-oriented, with all the folder structure conforming to the one required by HUGO. In case of switching to another SSG, we would need to follow the folder structure and the conventional best practices followed by that particular SSG. We would again need a restructuring, although not this massive. Also, things like menu bar items which would be in the root config of HUGO would also be needed to migrate since the new SSG might have a different approach.
> Regarding the folder structure, couldn’t we just copy/import the content to whichever location we wanted during that step of the workflow. Do we have to mirror it in the content repo?
> 
> Yes, that's exactly what I am proposing. I plan to re-organize the folder structure and not mirror the current content to the new structure.
>> Splitting the docs and the HUGO code would deprive of the many useful and core features of the tech. The front-matter or the meta-data per page is of great use and can be used to greatly customize and control the pages. Stripping HUGO of such important features could become a problem in the future or even during this project. For example, the GitHub link to edit the page is different on every page and can be stored in front-matter. Also, if we wanted to add an 'author' or 'last updated' information to the page, It would again be a front-matter thing.
> Can this information be automatically generated on import of the content?
> 
> We can have a common front matter template for all files in a directory by the archetype <https://gohugo.io/content-management/archetypes/> feature of HUGO.
>> Also, in case of switching SSG, the main part i.e. the MD doc files would be in the same hierarchy of contents. The only change that mostly happens in the names of the directories and some best practice conventions. Therefore, in my opinion, we should not split the codebase since it would be almost equal work switching in both cases but the one-repo approach gives many more additional features, and this advantage would be in other SSGs too.
>> I took a look at some other SSGs like Jekyll and 11ty. Most of my findings were the same as those by Sachin. One of the main drawbacks of Jekyll is build time increases exponentially with increase in repo size. 11ty has somewhat less community support. 
>> For choosing an SSG, the choice should be between Jekyll and HUGO. Jekyll has a somewhat rigid structure whereas HUGO is flexible. I would recommend HUGO for the task. 
> 
> Could you explain more about your recommendations, in particular against 11ty. Also, did you look at any others besides these three?
> 
> I looked at Gatsby and Gridsome - the two other main SSGs - but they are framework dependent i.e. on React and Vue respectively and can be difficult for those who haven't worked in that area. SSGs like 11ty and HUGO are easy to learn and setup.
> 
> Right now, 11ty has lesser community support than HUGO. One pro of using 11ty is that it can be configured to run in the current directory structure. Although this can be tempting at first, in my personal experience, too much flexibility causes a lack of convention and leads to bad practices. This has been the case of AngularJS which being too flexible gave way to randomness in conventions. One other advantage is that it can handle more than just md files, but since all our documentation is in md format, this isn't of much concern. 
> 
> Therefore I would recommend HUGO. However, I will try to research more on this topic and will update you if I find HUGO outperformed by another SSG.
> 
>> 
>> What are your opinions on this?
>> 
>> On Mon, Mar 2, 2020 at 6:27 PM Justin Obara <obara.justin at gmail.com <mailto:obara.justin at gmail.com>> wrote:
>> Hi Yash,
>> 
>> It isn’t required to work on those issues. However, feel free to triage them and leave comments as necessary (e.g. if they are no longer applicable and should be closed). 
>> 
>> Other things you could look at in preparation for your proposal would be:
>> 
>> The cross repo event communication you mentioned
>> The current structure, design, and etc.. how that might be affected by splitting things up
>> Other static site generators, pros/cons for using which ones
>> 
>> Also, regarding your comments about HUGO, if I’m understanding things correctly, HUGO’s speed benefits are at build/generation time, not necessarily when viewing the site. 
>> 
>> Feel free to ask other questions you may have.
>> 
>> Thanks
>> Justin
>> 
>>> On Mar 1, 2020, at 1:21 AM, Yash Jipkate <yashjipkate at gmail.com <mailto:yashjipkate at gmail.com>> wrote:
>>> 
>>> Also, the issues in the JIRA tracker you shared are very old and some don't seem to apply now. Should I be working on them?
>>> 
>>> 
>>> 
>>> 
>>> On Sat, Feb 29, 2020 at 5:32 PM Yash Jipkate <yashjipkate at gmail.com <mailto:yashjipkate at gmail.com>> wrote:
>>> Hi Justin,
>>> 
>>> Apologies for late reply, was busy with my exams, but they are gone for good now. I have addressed your comments below.
>>> 
>>> 
>>> On Wed, Feb 26, 2020 at 8:31 PM Justin Obara <obara.justin at gmail.com <mailto:obara.justin at gmail.com>> wrote:
>>> 
>>> 
>>>> On Feb 25, 2020, at 3:36 PM, Yash Jipkate <yashjipkate at gmail.com <mailto:yashjipkate at gmail.com>> wrote:
>>>> 
>>>> 
>>>> Hi Justin,
>>>> 
>>>> Thanks for the reply. Apologies for the late response from my side as I am in the middle of my exams. But I'll try to address as much as I can. I have left them inline.
>>>> 
>>>> On Tue, Feb 25, 2020 at 6:58 AM Justin Obara <obara.justin at gmail.com <mailto:obara.justin at gmail.com>> wrote:
>>>> Hi Yash,
>>>> 
>>>> Thanks for your interest in this project. I’ve left some comments inline below.
>>>> 
>>>> Thanks
>>>> Justin
>>>> 
>>>>> On Feb 24, 2020, at 4:29 PM, Yash Jipkate <yashjipkate at gmail.com <mailto:yashjipkate at gmail.com>> wrote:
>>>>> 
>>>>> Hi everyone,
>>>>> 
>>>>> I am Yash, and I need some discussion regarding the project "Refactor and Automating Infusion Documentation" and wanted to get your opinions on it.
>>>>> The HUGO code would be in a different repository; this would mean that the current repo would have to be cleaned of any rendering code, just .md and static files would remain. There is a way to clone another repo into a repo with the help of a script in the CI job.
>>>> Yes, ideally the content would be separate so that the Static Site Generator could be changed if needed, without the need to modify any of the content. Setting up a CI job, e.g. using GitHub Actions, could be one possible means of retrieving the content. Although this would also need to be easily testable for development so would likely involve some build task that could be run locally as well.
>>>> 
>>>> By testable, do you mean testing the code changes in the HUGO repo or the changes in the docs repo? Does the docs repo need testing?
>>> 
>>> I’d say both. For example if you make some docs changes, you’ll want to make sure that it renders correctly, and links to other pages are functioning. That might involve running an instance of the Static Site Generator locally. Actually thinking through this process would be helpful too. It may affect how we make other decisions related to this project.
>>> 
>>> So I was thinking that we could have a script for local testing of the docs repo - something like `bash run test.sh` - which would use the config files of HUGO to run a local instance of docs. We can have those config files in a tests/ folder. HUGO can be configured <https://gohugo.io/getting-started/configuration/> to look for necessary files other than at the root level. We can pull the required files directly from the remote server rendering repo and delete them or add them to .gitignore to avoid getting those in the repo.
>>>>> The CI should trigger whenever there is a push to main docs repo; this can be achieved by broadcasting a repository dispatch event from the main docs repo.
>>>> Will another GitHub repo be able to consume the “repository dispatch event”?
>>>> 
>>>> Yes, the dispatch event is a POST request sent to the repo we are targeting. 
>>> 
>>> Could you like to the docs for the dispatch event both for sending and consuming it?
>>> 
>>> You can find them here <https://developer.github.com/v3/repos/#create-a-repository-dispatch-event> and here <https://help.github.com/en/actions/reference/events-that-trigger-workflows#external-events-repository_dispatch>. 
>>>>> The cloning of the repo can be done tag-wise, i.e., instead of cloning the whole repo, cloning just a specific tag(s) would be done looped over all the available tags.
>>>> That could work. Does that actually save complexity and time over checking out the whole repo and locally iterating over tags?
>>>> 
>>>> I guess it would take slightly more time than cloning the whole repo, but HUGO doesn't have git variables that could extract tags from it, that's why I came up with this solution. 
>>>>> There are some libraries for indexing content and implementing the search functionality like hugo-lunr
>>>>> The front matter - since it is required to not include it in the files of the main docs repo - can be made as layouts. The only needed front matter seems to be the title of the page and the GitHub link - both of which can be determined with the help of FIle Variables in HUGO. 
>>>>> The file structure would result as follows: All the .md documentation files would be having the same organization with their tags at the topmost level. In other words, the file structure for the .md files would be content -> <tag> -> <files-in-their =-original-structure>. The static files would be placed in static/ at the same level as 'content' as required by HUGO.
>>>>> After cloning a tag, its contents would be copied to relevant directories, and then the clone would be deleted, and the process would be repeated for each tag, thus watching changes in each tag.
>>>>> These are some of my findings. What are your opinions? Should this be the way to do it? Or am I missing something that needs to be addressed? 
>>>> 
>>>> I don’t know for sure if this will work or is the best solution, but it looks like a good starting point to continue to research and test from. You can also look into other static site generators if there are others that may satisfy the requirements better.
>>>> 
>>>> Okay, I'll take a look at some other site generators as well. 
>>> 
>>> I think HUGO would be a good fit because of its extreme speed. In my opinion, a docs site should be as fast as possible since developers come there mostly when they are searching for something and no one wants a preloader to take most of their time. What do you think?
>>>> 
>>>>> 
>>>>> It would be great if I could get some insight into this. Also, it would be of great help if I get some guidance on how to start since the current docs repo has no standing issues on the tracker.
>>>> 
>>>> You can find more issues in the JIRA issue tracker: https://issues.fluidproject.org/browse/FLUID-6160?jql=project%20%3D%20FLUID%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20component%20%3D%20Website <https://issues.fluidproject.org/browse/FLUID-6160?jql=project%20=%20FLUID%20AND%20status%20in%20(Open,%20%22In%20Progress%22,%20Reopened)%20AND%20component%20=%20Website>
>>>> 
>>>> Oh, wasn't aware of this tracker, thanks for pointing out. 
>>>> 
>>>> But you should also start by making sure you can get the current site up and running locally. 
>>>> 
>>>> Do you mean using docpad? If yes, then I have already done that. 
>>> 
>>> Yes that’s what I meant. 
>>> 
>>>> 
>>>>> 
>>>>> Thanks and Regards, 
>>>>> Yash Jipkate
>>>>> Sophomore Undergraduate,
>>>>> IIT(BHU) Varanasi
>>>>> India
>>>>> 
>>>>> 
>>>>> _______________________________________________________
>>>>> fluid-work mailing list - fluid-work at lists.idrc.ocad.ca <mailto:fluid-work at lists.idrc.ocad.ca>
>>>>> To unsubscribe, change settings or access archives,
>>>>> see https://lists.idrc.ocad.ca/mailman/listinfo/fluid-work <https://lists.idrc.ocad.ca/mailman/listinfo/fluid-work>
>>>> 
>>>> 
>>>> Thanks and Regards, 
>>>> Yash Jipkate
>>>> Sophomore Undergraduate,
>>>> IIT(BHU) Varanasi
>>>> India
>>> 
>>> 
>>> 
>>> Thanks and Regards, 
>>> Yash Jipkate
>>> Sophomore Undergraduate,
>>> IIT(BHU) Varanasi
>>> India
>>> 
>>> Thanks and Regards, 
>>> Yash Jipkate
>>> Sophomore Undergraduate,
>>> IIT(BHU) Varanasi
>>> India
>>> 
>> 
>> 
>> Thanks and Regards, 
>> Yash Jipkate
>> Sophomore Undergraduate,
>> IIT(BHU) Varanasi
>> India
>> 
>> 
>> 
>> 
>> 
> 
> 
> 
>  
> 
> 
> 
> -- 
> Thanks and Regards, 
> Yash Jipkate
> Sophomore Undergraduate,
> IIT(BHU) Varanasi
> India
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idrc.ocad.ca/pipermail/fluid-work/attachments/20200309/741d1c75/attachment.html>


More information about the fluid-work mailing list