GSoC'20 : Using a Game Controller as a Navigation Aid

zaid junaid zaidjunaid3 at
Sun Mar 15 21:54:09 UTC 2020

Hi Abhishek
The solution you mentioned above for the second group seems to be good,
and I guess we can try to work on it
for the first group, we should try to think of a solution that can work
well to improve the experience with assisting devices as well.
and also it is not sure if we will be limiting ourselves to chrome
extension or electron framework
I was told to look at the working of Steam controllers to get a better idea.

Also due to the weekend mentors are not available, try connecting with them
through the IRC channel. particularly for this project  @the-t-in-rtf (IRC)
is the mentor
you can connect him during the European working hours.


On Sun, Mar 15, 2020 at 9:42 PM Abhishek Garain <abhi211199 at>

> Hi Zaid,
> Thanks for your suggestions. I had a look on the topics you pointed out.
> Visually impaired people usually don't work with a mouse or other
> navigating devices, instead, they rely on Talkback, TTS and STT features.
> On the other hand, mobile impaired people uses a special software which
> magnifies the nearby screen area surrounding the pointer. A similar
> solution has been discussed here:
> What we can aim to achieve is to increase the accessibility features
> associated with the gamepad, so as it helps the whole community to navigate
> in a better way.
> I am eagerly waiting for mentor's feedback.
> Regards,
> Abhishek Garain
> On Sun, Mar 15, 2020 at 12:00 AM zaid junaid <zaidjunaid3 at>
> wrote:
>> Hi  abhishek
>> I tried a similar thing and was given some input about the same, would
>> like to share it with you
>> Projects here tend to respect the full range of human diversity. the idea
>> you (and I) think of actually don't consider two major groups- people who
>> have mobile and visual impairment, so although this feature(navigating
>> through a pointer)  could act as an additional feature in the project, we
>> need to think of a better solution.
>> as for the frontend technology, we have our own Inclusion framework,
>> which we can use, you can have a look at it-
>> On Sat, Mar 14, 2020 at 11:38 PM Abhishek Garain <abhi211199 at>
>> wrote:
>>> Hey all, My name is Abhishek Garain, currently a 3rd year BTech student
>>> of IIIT Gwalior, India interested in Application Development using Electron
>>> and React. I am also into Android Development. I am interested to work on
>>> the Gamepad Navigation Aid project during Inclusive Design Institute's
>>> GSoC'20 program.
>>> Many a times, I have faced a problem while playing games and all of a
>>> sudden, have to reach out for a mouse to attend a notification or to change
>>> the game. This becomes even more troublesome when the laptop is connected
>>> to a TV or Monitor. This is a problem, which is faced by the wider
>>> community of gamers and other users, which I would like to solve.
>>> My 30 minutes of experience of browsing the web without a mouse can be
>>> found here:
>>> Talking about the technical aspects of the project,
>>> 1. After analysing my experiences of browsing the web without a mouse,
>>> the main difficulty was to navigate to a particular link, To overcome this,
>>> I need to get hold of the pointer.
>>> Javascript can get pointer co-ordinates but can't control the pointer
>>> movements. Python on the other hand, can do so. As the axes provides analog
>>> inputs, the pointer can be shifted according to the scaling factor
>>> multiplied by the analog value.
>>> 2. Secondly, the keys which are regularly used, needs to mapped to the
>>> gamepad. Like opening a new tab with Ctrl+N may me mapped with a button.
>>> Swiping between tabs and scroll bar can also be mapped to the axes.
>>> 3.Thirdly, Speech to Text may be implemented for filling up forms or for
>>> giving voice commands.
>>> Based on my observations, I have made a prototype(implemented in Python)
>>> which can be found here:
>>> As per my resarch, the incorporation of Chrome functionalities can be
>>> implemented using a Chrome Extension(chrome.tabs API). But controlling the
>>> pointer needs Python in background. And for mapping the keys to a gamepad,
>>> we need an UI built on Electron or within the Extension itself. I am
>>> currently researching on various frontend technologies which goes hand in
>>> hand with a gamepad.
>>> I would like to get valuable feedback from mentors regarding my thoughts
>>> and prototype.
>>> Regards,
>>> Abhishek Garain
>>> _______________________________________________________
>>> fluid-work mailing list - fluid-work at
>>> To unsubscribe, change settings or access archives,
>>> see
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>

More information about the fluid-work mailing list