Yeah, it's a bad time.
Thank you.
Cindy, would you like to start the recording now? Thank you. Okay. Thank you for joining us today, Tony.
I'll pass to you.
Right, let me fire up the presentation. Welcome everyone. So thanks for coming today. Today I'm going to give an update on the game pad navigator.
And hope to hear your feedback. Some of you may remember the last time I did this. At the end of, well, sorry, that Devianchu did this at the end of.
Google Summer of Code, 2,020. And a lot of what you see here today is based on feedback from bad event, however long it took to get there.
And then for those of you who haven't seen the gamepad navigator before, I'm going to give you a little bit of a background and also since it's been almost 4 years it'll be a reminder for the rest of us.
So the game had navigator navigator was a new project created during Google Summer of Code, 2,020.
Based on an idea I had and the research and hard work of our extraordinary student who thankfully could join us tonight.
Givian Chu. So, I started focusing on game pads as an assistive technology when the Xbox Adaptive Controller was released in 2,018.
You can see the video of me demonstrating, this is one of the YouTube videos that I've recorded.
Recently, showing how to use the X. Spark adaptive controller with the game that navigator.
So it's a modular controller with headphone jacks where you can plug in adaptive switches.
Here you can see me plugging in. One of the buttons from. The Logitech Adapted Gaming Kit, which was released in 2,019.
So for me, the combination was ideal. They provide a low-cost solution. It's very customizable.
You can also build your own buttons, which I've done a lot with. Basically headphone jacks and arcade.
So I use these 2 products in a range of personal projects, mostly musical toys for my son. And I've been thinking about using them in new ways.
And right before the summer of 2,020 I was searching for accessible and affordable options to help my son control a browser.
And I wrote up the core of the idea as a potential project for Google Summer Code and started talking with candidates.
And thankfully I was able to find an ideal candidate, Divian Chu who's here to join us.
And with my input as his mentor, he worked hard to build the first prototypes and the initial release of the Game Pet Navigator.
So the first thing I asked him to do was to look around for the building blocks of a solution, a broadways who might solve the problem.
We looked at a range of solutions that would let you take a game pad and use it to control.
Computer and or browser. We looked at things like operating system extensions. There are small circuit boards that, let you wire a control in to take the place of every key.
101 key keyboard. There are solutions with Internet of Things boards, but in the end we ended up coming back to web technologies because They were low cost flexible easy for end users to install and could be used with assistive controllers like the Xbox Adapted Controller, the Fourier Flex Controller, and the PlayStation access controller.
So we ended up going with the Game Pad API, which has been. Well supported in at least the browsers we needed for a while and then we used the browser extension APIs used in Chrome and Chrome variants and to a certain extent Firefox.
So with a browser extension, we could provide a cross platform solution as well. And push out updates, which Hasn't been us relevant until now, but will soon be very relevant.
So after a summer of iterating on the on the early version We went from the prototype to version 0 point 2 of the game pet navigator which was released to the Chrome Web Store.
And in summary, there's a half an hour video that But to get on she recorded that gives you a full details.
But in summary with version 0 point 2 you could use a game pad to and navigate between focusable elements in order to use Tab or Shift Tab on a keyboard.
You can simulate a click on a focused element. You could manage a single window, for example. Scrolling, zooming, changing the window size.
You could manage multiple windows, for example, switching windows, opening windows, closing windows. And then you could configure it.
You could add and remove bindings. So binding in this case is just the association between an input. What you press or touch.
And an action. What happens when you do it? So version 0 point 2 was an impressive start and an incredible amount of work for the amount of time that we had.
I had always intended to continue working on the gamepad navigator, but never quite found the time and It turned out it was a breaking change, that would have resulted in the plug-in being delisted from the Chrome Web Store in late 2,023.
And that finally pushed me to make the time to work on it. And although I quickly fixed the small issue that I needed to, Once I had a chance to reread the code and use the Gamepad Navigator again.
I started seeing both the limitations and some realistic opportunities to improve.
So let's summarize the limitations. First of all, you could only follow links. In other words, a link had to appear on the page you were already on if you're going to be able to click it you you
Could do a good decent job with something like a phone directory or something where it's just links.
But You had no control over form inputs like text and number inputs, which meant something as simple as searching.
Wasn't really possible.
It was also too easy to lose control of the browser and by control I mean you would be in the context with game pad but no longer work, to control your browser.
And this is because it's a browser extension and it has to be injected into a page that allows extension content and there's a lot of cases where that isn't the case.
And in version 0 point 2 there was nothing to prevent you from ending up somewhere where you would no longer have control.
And then this is kind of a victims of our success and the incredible amount of work that. Do you put in there were so many actions that actually you couldn't use them all on a single controller?
You, and you would have to reconfigure them and reconfiguring them. Required a keyboard and a mouse.
So in other words, you couldn't configure. You're set up with the gamepad itself.
And these to me with the key. Limitations, which. Were each an opportunity for improvement.
When it came time to work on. Version 1 point. So the first thing I added was an onscreen keyboard, which I started working on in 2,022.
There's a lot of on screen keyboards including for the web, but they're a lot of them assume that you're going to be using a mouse that you want a mouse to take the place.
Of a keyboard when what we wanted was a keyword that could be basically controlled using a game pad or a keyboard and I ended up having to write one which I wrote using infusion and then finally incorporated into the gamepad navigator for version 1 point.
And you can see one of the onscreen keyboards here.
The second improvement was the action launcher, which is a modal dialogue. That lets you execute a supported action without having to bind it to a button or thumb stick.
So even though we have far more actions than you could ever bind to a single controller. You can now access the ones that you don't use as often.
As long as you have a single button to open the action launcher.
And then the configuration panel, I ended up refactoring from the ground up. To work with a game pad or keyboard or mouse.
I wanted to keep the functionality we had where you could choose to configure it with a keyboard a mouse.
But I also wanted to work, with just a game pad and in addition I added a lot more general preferences and fine-grained control over individual bindings.
Which we'll see in the demo. And then we talked about control issues, the new version adds a few guardrails to keep you from.
Losing control over the browser if there's no controllable window when the browser starts one is open by default.
You can turn this off if you don't like it. The new version will also not switch to a window it can't control and it won't control.
Sorry, it won't close the last controllable window. So whenever possible, you are still going to be in a window where the game pet navigator can respond to your Gaype and inputs.
And then once all of that was more or less nailed down, I started testing with.
Well, real is kind of subjected, but with with sites that were complex enough to show me what we could and couldn't do.
I went to a lot of sites like Wikipedia, create dot Weavely. Org the fluid project homepage including pages that use UIO.
And then major search engines because part of what I wanted to do was make it so that navigating included Knowing what page you wanted to go to searching for it and clicking that link.
So I did this over and over again and made a note of things that didn't work. And fixed as many issues as I could and although there are still a lot of room for improvement you can operate a lot of sites without ever taking your hands off the keyboard and reaching for a mouse.
Sorry, taking your hands off the game pattern, reaching for a mouse or keyboard. So that's a lot of talking, but now we go for.
Demo time. So let me. Switch away and I'm gonna have to reconfigure briefly.
This is the point where You might want to PIN my video. Cause it's not just going to be talking.
So. Just a moment when we wake up my game pad. I've been talking for long enough that it's decided to take a break.
Okay, so on the bottom right you see representation of the game pad. That will.
Respond as I click. Buttons or it should. I hope it will.
Alright, well hopefully that will be visible enough. So let me focus back here. Right now I'm in the.
Configuration interface, which I'll show you in a minute. I'm just scrolling up for a moment.
The first thing I wanted to show you is the action launcher which is bound to the start button by default.
If I press the start button I get a list of all the actions. That I can perform, I'm going to.
Open a new window by hitting the click button, which is the bottom button. In the right hand cluster.
Is we kept the I kept the default from the last time. It switched to dark mode because it's 8 pm here, but, this is Google search.
It's in Dutch, but I think most of you will understand what's happening here. If I click on this text input that I'm on.
Automatically the keyboard pops up and I can search for So, and let me just explain about the modules it is.
Arrow and bumper navigable or you could think of it as air and tab navigable tab moves between the controls and The keys, so the buttons and the, the cancel button update feel button and the key so that you can easily move back and forth and then within the keyboard you move with the errors and the arrows.
Because we can now simulate sending keys, the errors are now bound to the DPAD, which stimulates sending.
The appropriate arrow key. To the on screen elements. So if I search by hitting the click button, if I search for IDRC.
I'm gonna intentionally make a typo to show you. Something. This actually does support. Changing the cursor position.
And fixing typos without having to erase everything. Which I wish Apple TV and other things did.
So if I update the field, you can see that the search input now has what I was searching for if I use the bumpers.
I can navigate and activate the search button. And there we have it. I'm able to search.
And navigate through a range of pages. Including something as complex as the Google search. With the game pad.
So I'm not gonna do a lot of real world demos if you want to see more examples of how this works.
You should check out the series of YouTube videos I recorded. Which go through all of the functions to manage a single page, to manage individual pages, to work with videos and also to work with the Xbox Adaptive Controller.
But for now, I'm gonna go back to open the settings panel, which I can do with the launcher.
There is a key for it. But let's. I've been recording, doing a bunch of dry run, so I ended up focused in a weird area.
Let me just navigate back to the top.
We start with general preferences and what I want to show you here is this is a range input. And this supports arrows.
So I'm using the DPAD to change the value. Of the range input. If I find a toggle like this.
If I click it with the click button. It behaves as a toggle wood if I clicked it with the mouse or activated it with the keyboard.
I've shown you entering text, which I can also do here. I will mention you can turn off arrow navigation through the features that I talked about.
I'm not going to I can't really demonstrate it because you have to hold. The controller on the microphone, but it does actually support vibration when I if like for example, I don't figure who this.
When I scroll up and there's no nowhere left to go, it vibrates to let me know that it can't do what I wanted to do.
And this is configurable. Here. And then the rest of the interface lets you control what bindings.
Are, available. So bindings are just again. The button you press or the thumb stick you press and then what I should do.
And I'll show you one of the key features here.
Alright, nearly all actions are repeatable. It used to be that thumb sticks were the only things that repeated.
So when you were scrolling, if you held down. It would keep scrolling. So often.
And now anything can be repeatable and you can you can choose to turn it off. You can say I don't want it to repeat.
Or assisted default for the bumpers. Repeat every point 5 s. So if I hold down one of the bumpers It keeps going.
You can turn this as fast as you want. You make it as slow as you want. Which means you can have kind of a scanning like.
Way of going through. The interface just by holding a button and letting go.
Right, so I realize that's a lot. Again, there's YouTube videos that go into a lot more of the the the features but I think that gives you a good feel for what it's possible to do.
With the new version and I am going to switch back to.
My main screen and. Switch over to questions and comments. I'm actually gonna have to kill the slide so that I can see hold of participants.
So here we go. I tried to keep it short and know it's kind of a whirlwind but I wanted to make sure there's plenty of time for.
People to ask questions and Anyway.
Anyone? Hello?
And you go ahead.
Hey, Tony, this is Daniel. Yeah. Thanks for the demo. As an example, you showed us like opening a new window and I was wondering if you can like go back to the previous window or is there a way to view or the windows those are open?
Right now there's not a way to view all the windows that are open, but I'll show you there definitely is a way to switch to the previous window next window, previous tab, next tab.
Close the current tab, close the current window. I added the the an action to duplicate the current tab.
You also have the ability to move back and forward in history. So I thought of that in some kind of menu to let you Go through the whole list of open.
Windows and tabs. In fact, I've got our feature request written up for that. But for now, you have to switch through it one window at a time.
I see. Good, thank you.
And Joseph?
Thanks. That is great Tony. May, have missed it, but what can you do with the thumbs?
Oh, that's a great point. Let me just scroll down. I'll show you some of the there's unique features.
For the thumb sticks which are kind of You can think of them as wrappers. So for example, if there's an action for a button to go up or down, you have an action for the thumb stick where you can Go either up or down.
Oh, sometimes this happens and I don't know why.
There's a subtle focus problem which happens when I switch back and forth a lot. Okay, so, for example, you can scroll right.
No, no, no.
You can scroll. How did I get back there? No, no, no, I'm on buttons and triggers again.
Thank you, time.
Sorry, just like. Okay. So you can, I don't know what is going on with the focus here.
I need to look into this, but, You can see you can navigate through history with a stick input.
So in other words, I found that one to be kind of. Disorienting, so I didn't find that by default.
You can send errors. With a with a stick input you can move through focus so you could use at a stick input to tab navigate basically.
And then you have several scrolling options and then you have a few things about zooming in and out and then changing the window size.
So the window size that just means maximize or not maximized. But those were some of the features that we had before and then preserved here.
The only one that is bound by the fault. Is is The left stick is found to scrolling by default.
Scrolling perfectly. Okay.
So yes, exactly. I found that if you had scrolling horizontally bound as well, what ended up happening is you would try to scroll down and accidentally scroll left and right.
And then an awful lot of pages don't really support scrolling left and right anyway so you can add it if you need it, but by default, vertical scrolling is the only one that's.
Got it.
Cool. I'm interested in what we're working on something like a keyboard and I would love to be able to use something like a joystick to scan up, down, left, right, and diagonally.
Right.
To get to to get the keys rather than tabbing endlessly.
Right and and that's part of the reason I went for error navigation for some things is that Tabbit you end up having an inlessly if you don't have something like errors occasionally available in your design toolkit.
Yeah.
And so for something like the I'll go back to the to the on-screen keyboard, something like the onsite keyboard.
When I start, I was so excited to add this and so happy to have it and then I tried to use it and I was sitting here tabbing through.
Right. Right?
100 keys to get to the update field button and then going back and forth and it was it was like every nightmare you've had about using a badly designed remote control search, you know, like where you, you're trying to search a YouTube video by hitting 2 buttons backward and forward.
So this turned out to be a decent pattern. This happened this, DPAD.
Up and down, left, right, yeah, okay.
Yeah, yeah. I mean it does actually support I'll show you it does actually support.
Shift and caps like as well. I the only thing that I realized far too late is that I didn't have any mechanism for international characters, which obviously I need to fix.
But anyway, I'll show you a feature that I have training before. I, because we now have the ability to send keys by default.
Right.
The right hand button hits escape, which it. So you can close the modal without. Just so, so the demo that I did for the Xbox Adapt controller, the 2 big buttons are bound to click and escape.
The D-pad does what you saw here and then I just attached bumpers. So with basically I think the 4 directions of the DPAD.
One button to click and one button, 2 buttons for tab navigation and then something for the action launcher.
So. You know, 7 or 8 buttons, you can actually do everything. Everything else is just.
What you prefer to have easy access to.
Okay, thanks.
Sure.
Hi, this time just to sort of jump off of what Joseph was asking. In a way, regarding the joysticks.
I know what the XC there is a X one and X 2 jack. Are those still usable as the joysticks or can you only go to the USB ports on the side if you want to use any of these stick.
I've never had anything that would work in either of those ports. So I'm not sure.
Okay.
I know that it will work with the USB joystick. And I know that you can, you can get breakout boards to let you turn nearly anything.
But I don't know. I haven't looked at it recently to see how that works.
I think in part it's because any of the analog controls, they need to have a certain range of resistance and I never figured out the right level of resistance to make something act as an analog input even the triggers I could only get them to do.
Binary inputs and I think for something like an access you would want It should be able to deliver a range of, I guess.
Voltages. Yeah, so I, I don't know. I don't know.
Okay. No, no worries. We've been No, I guess we have an adaptive game program here and we've been using some of the joysticks for makers making change and they lot of their joysticks go through the X one and X 2 and sports and I was just wondering it
Yes.
Oh, thank you very much. I'm gonna write that down. Makers making change. Making change joysticks.
Okay, so, Wonderful.
Yup, they're working on a whole plethora of joysticks right now. For their, they have an adaptive game going on.
We're actually partnered with them here. As one of their game checkpoints where one of 8 across the country.
So they're doing a lot of work in the adaptive gaming world right now. And yeah, yes, I'm trying to see if any of the products will work.
I'm currently in the augmented of an alternate communication. Field right now as well.
Nice.
So I'm trying to pull some of their stuff into working with computers as well. So this is very interesting to hear all your work on this and I'd be I'll probably be exploring myself with some of us up here.
Sounds good. I would like to mention that there are operating system specific plugins that will let you Have a joystick.
Game Pad act as a keyboard and or mouse. I haven't found one that's particularly well supported, especially in an environment like OSX where you get breaking changes every couple versions of.
Of the operating system, but there are at least some solutions out there that will let you handle it at the operating system level.
So one thing I can't do is change from a browser to any other browser or any of the program, I can't control any of the program.
But a system like that, there's like joy key and a few others will let you. Do that.
Yep.
Yeah, I just discovered that one, right? I recently, well, I was, searching for Ig's, technologies and gaming as well.
I found that one. So I'm planning on playing around with that one. So I'm planning on playing around with that one in the next few weeks as well.
Cool.
So yeah. No, but this is very, interesting. And I think Even if you're, as you said, the Chrome in itself is quite complex, so to be able to navigate it.
Sure, please.
So cleanly using a game pad is quite good. And Sorry, I have one more question about I'll save that after Sunday.
Okay.
Actually team, if you I also have multiple questions. So Tim, if you wanna go ahead with your next question, I'm totally fine.
Okay, thank you. I just want to quickly ask you mentioned that it's also works with the PlayStation access controller and the Hori.
I will, I it should. I don't actually own all 3 of them. I just wanted to mention it because It should
If you can.
Work with anything that can connect. So I have I have Too many game controllers. I don't, but I can't justify spending another.
3 or 400 to get those 2. So I mainly focus on the first one that was available.
But I have, I'm I have PlayStation controllers and if it acts like a PlayStation controller, it should work with the gamepit navigator.
I have switch controllers. If it acts like a switch controller, it should work with the Gamepad Navigator.
Okay.
So I would love for someone to tell me How it is.
One wonderful.
If you, if you wanna connect, I do have access to all 3 through our game program here. So.
I think you and I have a lot to talk about.
So, that's what you want.
Worked at writing AAC apps in the past. So it seems like we have a lot to talk about.
Yeah, so definitely we can talk. On a separate thing as well.
Great, sounds great. Cool.
Okay, so and, Tony, thank you for.
Demonstrating this. This is really great.
Yep.
Yes.
So one question I have is I'm not familiar with their like, game contours like using a in path, we're using PlayStation controllers.
So I wonder how does it get connected to the Chrome? Like
But you that's it.
Right, right. So you have to do, you have to connect it to your operating system first. And in most cases, we're talking about or Bluetooth.
And then once you've done that bit of setup, it works through the gamepad API, which has been supported in most browsers for quite a few years.
So basically
Any connected controller, you have to move it for it to take for it to do something, but any connected controller, is visible to your browser through the window navigator.
There's an API to check for connected controllers and to read all of the values and to vibrate them and things like that.
Really good good good features and that that's probably why that you can use. Like, you can use, browser or you can use the tab to do their gimpad configuration so you can send those configuration back through the API to controller game pad.
Behavior. So is that
Right, and there's only a few things that you can do to the game head. Like I've never found a way to play sound for example through the
Getting paid.
Which a lot of G pads support headphone jacks. I haven't found a way to do that.
But vibration is pretty much the only thing you can But really the only thing you can send back is.
Indication of whether it
Whether or not to vibrate and unfortunately There's only one profile that works really. So it's it can't even do some of the subtle vibrations that newer controllers that the switch controller can do and it can't do complex.
Triggers and haptics. Like something like the the PlayStation 5 controller can do so it's in a way it's kind of a least common denominator for kind of became a few years ago, but it's still very powerful and the advantages, if you're curious about it, you can find a compatible controller.
Even at a thrift store or Sorry, yard sale, which I often do. For, you know, lunch money.
It's, it's not an expensive hobby. Compared to synthesizers.
Yeah.
Yeah, I'm, so curious to find out like how well the support is for like the EC devices, like I, IDs devices or switches and those kind of things that it would be very interesting to find out.
So.
I know there are people working to connect. I gaze devices I think to the to the Xbox control in particular because it's very well supported on Windows.
I have seen articles about that, but I think probably Tim could comment better on that.
Yeah, right now I think actually the, Toby has released, well, Toby has her AGE camera and that's here.
Oh nice.
I think right now they're mostly focusing on PC and actually through the Hori Flex controller for the attendance switch actually so they seem to have that down that part figured out.
They haven't touched too much on the X the XC the Xbox adaptive controller yet I do know that they also just have their own software that helps do a lot of games where looking around using a secondary joystick is a problem.
Now you can just use your eyes and head movements to control that right choice. I know they've done that.
I'll be playing around. I love this is more things I've seen. I plan to actually try them out in the next few weeks.
But yeah.
Great. Yeah, unfortunately Justin just left, but he and I did do a little work to actually push out the new version.
I wanted to give everyone a chance to see it and maybe. To a little less. Finger crossing in QA, but we should be pushing out 1 point out where you could just install it from the Google.
From web stored in the next week or so. And just try it. I'd love to hear what you think.
Yeah, that's cool. I would love to try. And so, and Tony, I have another question about their, onscreen keyboard.
Yes.
So whenever you focus on our tax field, Issac when I, actually my question is, How is it?
Is it working like whenever you focus on our tax field on our page? The onscreen keyboard we're But.
Right, right. So we, yeah, yeah, we, we have what are called actions, which basically one of these is a click action and this click action It does.
It has to check and see what kind of element it's focused on and it decides to do a few special things for different elements.
And one of those things is for a text input and I have a whole bunch of rules about what a text input could be.
If it thinks it's a text input. Then when you click it won't send a click to that element.
Instead, it will try to open the on screen keyboard. If it's a link or a button, and it knows that it's not a text input and it will just trigger the click activity of that element.
So it's based on what you're focused on and it has rules around. What it should do when you click on it.
Yeah. That's really cool. So, yeah, is there written in Chrome extension?
Right, so, there's, it is, it is a chrome extension and there's 2 parts of the chroma extension.
There's the part that is injected into the page. Which for this version is injected into a, an element with a shadow root.
So that it doesn't affect the markup of the page any more than we can help. And then there's a part that lives in the background, which is in the case of Chrome, they've migrated to using service workers.
So there's a service worker that has access to different APIs and then there's a mechanism for communicating between the client side where the code is injected and the service worker.
And, and in this version, we, it's actually bi-directional. So that if you tell the service worker to do something and it knows it can't, it sends information back to the client side to say I couldn't do that and then the client side vibrates the controller.
So there's There's kind of a couple of moving parts, but it's all, stuff that is published as part of the I guess they call it the web extension API, which is mostly just.
A curve extension that now seems to work. Almost everywhere.
Yeah, thank you. Yeah, this is very cool. Yeah, that's where I wanna ask about.
So adding questions from anyone?
No.
I'm bright. Well, thank you everyone for your time. It's been really great. I always enjoy hearing your questions as I as I pointed out we're gonna have a release very soon where you won't have to do much to try it.
There's also ways to install it and try the version that we're working on. That they're not any more difficult than anything else we have to do as developers.
So, I would be happy for any of you to try it. I'd love to hear what you think, the project and all of the feature requests and issues are tracked on GitHub.
And, I'm always open to. New ideas about What could work better?
Cool. Thank you very much.
Thank you.
All right, thanks everyone.