GSoC'20 : Using a Game Controller as a Navigation Aid

Abhishek Garain abhi211199 at
Sun Mar 15 16:12:02 UTC 2020

Hi Zaid,
Thanks for your suggestions. I had a look on the topics you pointed out.
Visually impaired people usually don't work with a mouse or other
navigating devices, instead, they rely on Talkback, TTS and STT features.
On the other hand, mobile impaired people uses a special software which
magnifies the nearby screen area surrounding the pointer. A similar
solution has been discussed here:
What we can aim to achieve is to increase the accessibility features
associated with the gamepad, so as it helps the whole community to navigate
in a better way.

I am eagerly waiting for mentor's feedback.

Abhishek Garain

On Sun, Mar 15, 2020 at 12:00 AM zaid junaid <zaidjunaid3 at> wrote:

> Hi  abhishek
> I tried a similar thing and was given some input about the same, would
> like to share it with you
> Projects here tend to respect the full range of human diversity. the idea
> you (and I) think of actually don't consider two major groups- people who
> have mobile and visual impairment, so although this feature(navigating
> through a pointer)  could act as an additional feature in the project, we
> need to think of a better solution.
> as for the frontend technology, we have our own Inclusion framework,
> which we can use, you can have a look at it-
> On Sat, Mar 14, 2020 at 11:38 PM Abhishek Garain <abhi211199 at>
> wrote:
>> Hey all, My name is Abhishek Garain, currently a 3rd year BTech student
>> of IIIT Gwalior, India interested in Application Development using Electron
>> and React. I am also into Android Development. I am interested to work on
>> the Gamepad Navigation Aid project during Inclusive Design Institute's
>> GSoC'20 program.
>> Many a times, I have faced a problem while playing games and all of a
>> sudden, have to reach out for a mouse to attend a notification or to change
>> the game. This becomes even more troublesome when the laptop is connected
>> to a TV or Monitor. This is a problem, which is faced by the wider
>> community of gamers and other users, which I would like to solve.
>> My 30 minutes of experience of browsing the web without a mouse can be
>> found here:
>> Talking about the technical aspects of the project,
>> 1. After analysing my experiences of browsing the web without a mouse,
>> the main difficulty was to navigate to a particular link, To overcome this,
>> I need to get hold of the pointer.
>> Javascript can get pointer co-ordinates but can't control the pointer
>> movements. Python on the other hand, can do so. As the axes provides analog
>> inputs, the pointer can be shifted according to the scaling factor
>> multiplied by the analog value.
>> 2. Secondly, the keys which are regularly used, needs to mapped to the
>> gamepad. Like opening a new tab with Ctrl+N may me mapped with a button.
>> Swiping between tabs and scroll bar can also be mapped to the axes.
>> 3.Thirdly, Speech to Text may be implemented for filling up forms or for
>> giving voice commands.
>> Based on my observations, I have made a prototype(implemented in Python)
>> which can be found here:
>> As per my resarch, the incorporation of Chrome functionalities can be
>> implemented using a Chrome Extension(chrome.tabs API). But controlling the
>> pointer needs Python in background. And for mapping the keys to a gamepad,
>> we need an UI built on Electron or within the Extension itself. I am
>> currently researching on various frontend technologies which goes hand in
>> hand with a gamepad.
>> I would like to get valuable feedback from mentors regarding my thoughts
>> and prototype.
>> Regards,
>> Abhishek Garain
>> _______________________________________________________
>> fluid-work mailing list - fluid-work at
>> To unsubscribe, change settings or access archives,
>> see
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>

More information about the fluid-work mailing list