Overview
Over the course of these past 10 weeks, I made an RTS that is entirely hands-free, using only eye-tracking and voice recognition. Over the course of this project, I had a lot of fun, and I also learned things that I will carry with me into future projects. In the effort of being thorough, I’ll go over what went wrong and what went right.
What Went Wrong?
Not a lot went terribly wrong over the course of this project. Adding in eye tracking and voice recognition and hooking them up to commands executed in the game only took about a week. Refining that system took much longer of course. So the chief thing that went wrong, in my opinion, was the medium of eye tracking itself.
Eye tracking is great for big targets on the screen or choosing specific areas on the screen that would be difficult to distinguish with voice alone. For example, placing down the barracks was a good use of eye tracking. However, apart from that, voice recognition is a much easier way to navigate a game like an RTS. Try as I did, eye tracking did not make for an engaging RTS experience, especially given the near-perfect lighting circumstances necessary to get it working reliably. Looking back, I would focus on the voice recognition aspect, and only use eye tracking where it provides an advantage.
Movement was surprisingly one of the biggest struggles I had in the development of this. I think the reason for that is composed of several smaller problems. There was the procedural generation, hex grid combined with square-ish grid, and most importantly, the short timeline to implement custom character movement and animation on top of developing a full game with other challenging elements. Looking back, I would have planned out how I wanted movement to work more thoroughly, and I might have used Unity’s character controller. Other than that, I think I simply needed more time if I wanted to dedicate the attention to movement that it required.
What Went Right?
Most things went right! I didn’t expect this game to be as fully fleshed out as it ended up. I was able to add a lot of smaller details and features that I didn’t intend to from the beginning, simply because I saw an opportunity to do so as I was developing.
Using Unity as my engine certainly helped a great deal. I have a lot of experience in Unity, and it’s incredibly fun for me to work in, almost like a game itself. C#’s ease of use, along with Microsoft’s suite of built in data structures and other systems, made coding an absolute breeze. Some features were implemented in times drastically lower than I allotted because slotting them into my existing infrastructure was a piece of cake.
Partially tied to Unity, already having software for capturing voice and translating it into a string I could then parse made voice recognition very exciting to work with. It allowed me to spend more time handling my speech tree and command system to make it more refined. The same can be said of the eye tracking software. I had very few technical issues on that end. That gave me more time to attempt to make eye tracking less fidgety and more user friendly.
I didn’t have a concrete plan going in, but I had a general outline for how I wanted to pace my project and which systems I wanted to work on and in what order. That foreknowledge allowed me to create a framework where I could easily slot in every new feature I added. I had managers that kept track of all of units and all of the buildings, so if I needed to refer to one, I could just search the manager. I had a manager to handle all of the menus, so adding menus was a matter of adding a reference to it to that manager and adding it to the list of commands. Speaking of commands, I had a class I melodramatically called CommandCenter that received all commands and then delegated tasks to the appropriate managers to handle them. My speech tree made adding new voice commands take a few minutes once the function for the command itself was created. My eye tracker worked in a similar way, where I had events tied to each item that could be selected, and adding a new function meant adding it as a listener to that event. Overall, the systems I set up made adding new things to my existing game very straightforward, and I will definitely remember that as I begin new projects.
Final Product
Keira Taylor - Game Development 