GSoC'20 : Using a Game Controller as a Navigation Aid

[Abhishek Garain]

Hey all, My name is Abhishek Garain, currently a 3rd year BTech student of
IIIT Gwalior, India interested in Application Development using Electron
and React. I am also into Android Development. I am interested to work on
the Gamepad Navigation Aid project during Inclusive Design Institute's
GSoC'20 program.

Many a times, I have faced a problem while playing games and all of a
sudden, have to reach out for a mouse to attend a notification or to change
the game. This becomes even more troublesome when the laptop is connected
to a TV or Monitor. This is a problem, which is faced by the wider
community of gamers and other users, which I would like to solve.

My 30 minutes of experience of browsing the web without a mouse can be
found here:
https://medium.com/@abhi211199/lets-browse-the-web-without-mouse-for-30-minutes-b9a0db5e3ec9

Talking about the technical aspects of the project,
1. After analysing my experiences of browsing the web without a mouse, the
main difficulty was to navigate to a particular link, To overcome this, I
need to get hold of the pointer.
Javascript can get pointer co-ordinates but can't control the pointer
movements. Python on the other hand, can do so. As the axes provides analog
inputs, the pointer can be shifted according to the scaling factor
multiplied by the analog value.
2. Secondly, the keys which are regularly used, needs to mapped to the
gamepad. Like opening a new tab with Ctrl+N may me mapped with a button.
Swiping between tabs and scroll bar can also be mapped to the axes.
3.Thirdly, Speech to Text may be implemented for filling up forms or for
giving voice commands.

Based on my observations, I have made a prototype(implemented in Python)
which can be found here: https://github.com/abhi211199/gamepad_navigator
As per my resarch, the incorporation of Chrome functionalities can be
implemented using a Chrome Extension(chrome.tabs API). But controlling the
pointer needs Python in background. And for mapping the keys to a gamepad,
we need an UI built on Electron or within the Extension itself. I am
currently researching on various frontend technologies which goes hand in
hand with a gamepad.

I would like to get valuable feedback from mentors regarding my thoughts
and prototype.

Regards,
Abhishek Garain
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idrc.ocad.ca/pipermail/fluid-work/attachments/20200314/57a76929/attachment.html>