Discussion regarding the project "Using a Game Controller as a Navigation Aid"

Tony Atkins tony at raisingthefloor.org
Mon Mar 2 09:22:40 UTC 2020


Hi, Neetigya.

Thanks for getting in touch, and for going through the exercise.  Quite a
few candidates have had similar ideas about using mouse navigation.  This
may be a good addition to my original concept, which was more concerned
with keyboard navigation.  One of things that inspired me to write up this
project was the Steam Controller, the (now discontinued) gamepad that
allows you to use any of the control surfaces as an analog stick, keyboard
arrows, or even a mouse.  If you have a moment, there are lots of demos on
YouTube that might give you ideas about the challenges of following that
approach.

Although I put the technical details in up front to give people ideas, part
of the project would involve a more involved research effort to try out
both of the approaches I suggested, and also to look for others.  So I
wouldn't focus too heavily on chrome extensions or anything specific like
that.

Now that you've demonstrated your interest in the project and understanding
of how we might accomplish the goal, I would suggest stepping back and
getting to know the larger community and software ecosystem you'd be
working in.  Any successful candidate needs to demonstrate familiarity with
Infusion as well as our community practices and standards.

You've already found one of the places we communicate, namely the
fluid-work mailing list.  Take some time to observe how things work here,
and in other communication channels (such as our IRC channel, JIRA tickets,
github pull requests).  Try out Infusion, read through the documentation,
try the tutorials, ask (and maybe even answer) questions, et cetera.

Cheers,


Tony

On Fri, 28 Feb 2020 at 21:19, Neetigya Chahar <neetigyachahar784 at gmail.com>
wrote:

> Hello everyone! I am a sophomore CSE student and have some experiences in
> nodejs, creating and hosting chrome extensions on the store. I have also
> done some programming and controlling of IoT devices.
>
> As I was looking to work with open source community through gsoc2020, I
> came across the project "Using a Game Controller as a Navigation Aid"
>
> My 30 minutes experience with keyboard navigation:
> 1. Most of the time, it was tab which I used to navigate around.
> 2. It was very frustrating to press tab again to reach an element which is
> at the centre or near bottom of the page every time new page is loaded.
> 3. For many websites like stackoverflow it was very touch to spot the
> current focus. And for many sites, the focus was instantaneously lost on
> next press.
> 4. Even if focus was clearly visible, for websites like w3schools, first
> tab focus traverse through right side menu, which is too long many times.
> This again creates need to press tab again and again.
> 5. Some HTML elements were totally inaccessible for e.g. unable to access
> whatsapp web chat options.
> 6. Canvas elements were not accessible.
> 7. Although websites like Google search has provided good accessibility
> support.
> 8. Keys other than tab were pretty much usable, but still may have further
> improvements.
>
> Both proposed solutions uses chrome extension, as according to me using
> extension instead of an electron app is more productive and practical.
>
> What can be done if we don't consider mouse simulation along:
> 0. Basically we have to map keys on the keyboard to the game pad and read
> the game pad output using official Game pad API.
> 1. Every Page's :focus could be edited to make outline look more clearly.
> 2. Provide key combo (or press and hold) to automatically press tab
> multiple times.
> 3. We can create a page which tells people how keys of the game pad are
> mapped to keyboard keys.
> 4. We can provide option to edit these mappings and give suggestions.
>
> Using mouse pointer:
> Mouse pointer is not part of the web page or the browser.
> It is something directly provided by the OS.
> There exists no web api to control mouse movement as it is considered a
> security vulnerability.
> However, the movements can be tracked.
>
> My idea:
> I was to thinking of adding a virtual mouse to the sites through
> extension. I have tested that.
> We can define an element with image of a mouse and the element can be
> positioned absolutely.
> Now we can set and alter the top and left properties i.e y and x
> coordinates of the element. This way we can move the element wherever we
> want.
> This element can be controlled by joystick, arrow keys in the game pad or
> in the keyboard.
> Since we know the x and y position of the element, I wrote the following
> chuck to perform click action.
>
>     pointer.style.display = 'none'; //Remove our element.
>     let test = document.elementFromPoint(pointer.x, pointer.y); //select
> the element where our element was.
>     test.click(); //click it
>     pointer.style.display = 'block'; //Show up again.
>
> The action of removing and showing up again is instantaneous, goes
> unnoticed.
>
> What can be done if we consider mouse simulation along:
> 1. The work of the tab key can be totally shifted to this mouse.
> 2. Other keys like space, enter, arrow keys, key combos like for closing
> and opening tabs will still be usable as earlier.
> 3. All the problems related to the tab will be solved.
> 4. We can give a click animation feedback to users as in android, thus
> increasing accessibility.
> 5. Also, when the element is at top of some text (like paragraph), the
> user can press a key on game pad and we can read it aloud, bold it,
> increase font size, etc.
>
> I hope I didn't annoyed anyone by writing such a text to mailing list. I
> tried IRC but there was timezone issue, I will adjust that in future.
> Meanwhile I am learning about the community and its projects like infusion.
>
> Feedback awaited!
>
> This is most basic implementation of a virtual mouse.
> [image: fluid-project.png]
> [image: gsoc20201.png]
> _______________________________________________________
> fluid-work mailing list - fluid-work at lists.idrc.ocad.ca
> To unsubscribe, change settings or access archives,
> see https://lists.idrc.ocad.ca/mailman/listinfo/fluid-work
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.idrc.ocad.ca/pipermail/fluid-work/attachments/20200302/a59df211/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fluid-project.png
Type: image/png
Size: 23090 bytes
Desc: not available
URL: <https://lists.idrc.ocad.ca/pipermail/fluid-work/attachments/20200302/a59df211/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gsoc20201.png
Type: image/png
Size: 43059 bytes
Desc: not available
URL: <https://lists.idrc.ocad.ca/pipermail/fluid-work/attachments/20200302/a59df211/attachment-0001.png>


More information about the fluid-work mailing list