WEBVTT 00:00:00.418 --> 00:00:07.993 Yeah, it's a bad time. 00:00:07.993 --> 00:00:10.220 Thank you. 00:00:10.220 --> 00:00:12.068 Cindy, would you like to start the recording now? Thank you. Okay. Thank you for joining us today, Tony. 00:00:12.068 --> 00:00:22.470 I'll pass to you. 00:00:22.470 --> 00:00:30.019 Right, let me fire up the presentation. Welcome everyone. So thanks for coming today. Today I'm going to give an update on the game pad navigator. 00:00:30.019 --> 00:00:38.969 And hope to hear your feedback. Some of you may remember the last time I did this. At the end of, well, sorry, that Devianchu did this at the end of. 00:00:38.969 --> 00:00:49.570 Google Summer of Code, 2,020. And a lot of what you see here today is based on feedback from bad event, however long it took to get there. 00:00:49.570 --> 00:00:57.570 And then for those of you who haven't seen the gamepad navigator before, I'm going to give you a little bit of a background and also since it's been almost 4 years it'll be a reminder for the rest of us. 00:00:57.570 --> 00:01:03.470 So the game had navigator navigator was a new project created during Google Summer of Code, 2,020. 00:01:03.470 --> 00:01:14.769 Based on an idea I had and the research and hard work of our extraordinary student who thankfully could join us tonight. 00:01:14.769 --> 00:01:20.870 Givian Chu. So, I started focusing on game pads as an assistive technology when the Xbox Adaptive Controller was released in 2,018. 00:01:20.870 --> 00:01:26.320 You can see the video of me demonstrating, this is one of the YouTube videos that I've recorded. 00:01:26.320 --> 00:01:31.869 Recently, showing how to use the X. Spark adaptive controller with the game that navigator. 00:01:31.869 --> 00:01:40.270 So it's a modular controller with headphone jacks where you can plug in adaptive switches. 00:01:40.270 --> 00:01:47.621 Here you can see me plugging in. One of the buttons from. The Logitech Adapted Gaming Kit, which was released in 2,019. 00:01:47.621 --> 00:01:55.270 So for me, the combination was ideal. They provide a low-cost solution. It's very customizable. 00:01:55.270 --> 00:02:04.920 You can also build your own buttons, which I've done a lot with. Basically headphone jacks and arcade. 00:02:04.920 --> 00:02:12.520 So I use these 2 products in a range of personal projects, mostly musical toys for my son. And I've been thinking about using them in new ways. 00:02:12.520 --> 00:02:20.120 And right before the summer of 2,020 I was searching for accessible and affordable options to help my son control a browser. 00:02:20.120 --> 00:02:25.169 And I wrote up the core of the idea as a potential project for Google Summer Code and started talking with candidates. 00:02:25.169 --> 00:02:33.320 And thankfully I was able to find an ideal candidate, Divian Chu who's here to join us. 00:02:33.320 --> 00:02:41.819 And with my input as his mentor, he worked hard to build the first prototypes and the initial release of the Game Pet Navigator. 00:02:41.819 --> 00:02:47.120 So the first thing I asked him to do was to look around for the building blocks of a solution, a broadways who might solve the problem. 00:02:47.120 --> 00:03:00.719 We looked at a range of solutions that would let you take a game pad and use it to control. 00:03:00.719 --> 00:03:24.520 Computer and or browser. We looked at things like operating system extensions. There are small circuit boards that, let you wire a control in to take the place of every key. 00:03:24.520 --> 00:03:41.070 101 key keyboard. There are solutions with Internet of Things boards, but in the end we ended up coming back to web technologies because They were low cost flexible easy for end users to install and could be used with assistive controllers like the Xbox Adapted Controller, the Fourier Flex Controller, and the PlayStation access controller. 00:03:41.070 --> 00:03:52.620 So we ended up going with the Game Pad API, which has been. Well supported in at least the browsers we needed for a while and then we used the browser extension APIs used in Chrome and Chrome variants and to a certain extent Firefox. 00:03:52.620 --> 00:04:04.920 So with a browser extension, we could provide a cross platform solution as well. And push out updates, which Hasn't been us relevant until now, but will soon be very relevant. 00:04:04.920 --> 00:04:12.221 So after a summer of iterating on the on the early version We went from the prototype to version 0 point 2 of the game pet navigator which was released to the Chrome Web Store. 00:04:12.221 --> 00:04:20.969 And in summary, there's a half an hour video that But to get on she recorded that gives you a full details. 00:04:20.969 --> 00:04:29.219 But in summary with version 0 point 2 you could use a game pad to and navigate between focusable elements in order to use Tab or Shift Tab on a keyboard. 00:04:29.219 --> 00:04:36.821 You can simulate a click on a focused element. You could manage a single window, for example. Scrolling, zooming, changing the window size. 00:04:36.821 --> 00:04:43.469 You could manage multiple windows, for example, switching windows, opening windows, closing windows. And then you could configure it. 00:04:43.469 --> 00:04:54.470 You could add and remove bindings. So binding in this case is just the association between an input. What you press or touch. 00:04:54.470 --> 00:05:10.220 And an action. What happens when you do it? So version 0 point 2 was an impressive start and an incredible amount of work for the amount of time that we had. 00:05:10.220 --> 00:05:20.619 I had always intended to continue working on the gamepad navigator, but never quite found the time and It turned out it was a breaking change, that would have resulted in the plug-in being delisted from the Chrome Web Store in late 2,023. 00:05:20.619 --> 00:05:27.369 And that finally pushed me to make the time to work on it. And although I quickly fixed the small issue that I needed to, Once I had a chance to reread the code and use the Gamepad Navigator again. 00:05:27.369 --> 00:05:42.370 I started seeing both the limitations and some realistic opportunities to improve. 00:05:42.370 --> 00:05:48.069 So let's summarize the limitations. First of all, you could only follow links. In other words, a link had to appear on the page you were already on if you're going to be able to click it you you 00:05:48.069 --> 00:05:55.470 Could do a good decent job with something like a phone directory or something where it's just links. 00:05:55.470 --> 00:05:58.519 But You had no control over form inputs like text and number inputs, which meant something as simple as searching. 00:05:58.519 --> 00:06:08.620 Wasn't really possible. 00:06:08.620 --> 00:06:20.569 It was also too easy to lose control of the browser and by control I mean you would be in the context with game pad but no longer work, to control your browser. 00:06:20.569 --> 00:06:27.719 And this is because it's a browser extension and it has to be injected into a page that allows extension content and there's a lot of cases where that isn't the case. 00:06:27.719 --> 00:06:39.320 And in version 0 point 2 there was nothing to prevent you from ending up somewhere where you would no longer have control. 00:06:39.320 --> 00:06:47.419 And then this is kind of a victims of our success and the incredible amount of work that. Do you put in there were so many actions that actually you couldn't use them all on a single controller? 00:06:47.419 --> 00:06:52.469 You, and you would have to reconfigure them and reconfiguring them. Required a keyboard and a mouse. 00:06:52.469 --> 00:06:58.571 So in other words, you couldn't configure. You're set up with the gamepad itself. 00:06:58.571 --> 00:07:08.769 And these to me with the key. Limitations, which. Were each an opportunity for improvement. 00:07:08.769 --> 00:07:17.371 When it came time to work on. Version 1 point. So the first thing I added was an onscreen keyboard, which I started working on in 2,022. 00:07:17.371 --> 00:07:36.370 There's a lot of on screen keyboards including for the web, but they're a lot of them assume that you're going to be using a mouse that you want a mouse to take the place. 00:07:36.370 --> 00:07:39.072 Of a keyboard when what we wanted was a keyword that could be basically controlled using a game pad or a keyboard and I ended up having to write one which I wrote using infusion and then finally incorporated into the gamepad navigator for version 1 point. 00:07:39.072 --> 00:07:49.670 And you can see one of the onscreen keyboards here. 00:07:49.670 --> 00:07:56.871 The second improvement was the action launcher, which is a modal dialogue. That lets you execute a supported action without having to bind it to a button or thumb stick. 00:07:56.871 --> 00:08:01.670 So even though we have far more actions than you could ever bind to a single controller. You can now access the ones that you don't use as often. 00:08:01.670 --> 00:08:11.720 As long as you have a single button to open the action launcher. 00:08:11.720 --> 00:08:16.171 And then the configuration panel, I ended up refactoring from the ground up. To work with a game pad or keyboard or mouse. 00:08:16.171 --> 00:08:27.121 I wanted to keep the functionality we had where you could choose to configure it with a keyboard a mouse. 00:08:27.121 --> 00:08:36.220 But I also wanted to work, with just a game pad and in addition I added a lot more general preferences and fine-grained control over individual bindings. 00:08:36.220 --> 00:08:42.670 Which we'll see in the demo. And then we talked about control issues, the new version adds a few guardrails to keep you from. 00:08:42.670 --> 00:08:51.421 Losing control over the browser if there's no controllable window when the browser starts one is open by default. 00:08:51.421 --> 00:09:02.420 You can turn this off if you don't like it. The new version will also not switch to a window it can't control and it won't control. 00:09:02.420 --> 00:09:09.019 Sorry, it won't close the last controllable window. So whenever possible, you are still going to be in a window where the game pet navigator can respond to your Gaype and inputs. 00:09:09.019 --> 00:09:16.371 And then once all of that was more or less nailed down, I started testing with. 00:09:16.371 --> 00:09:25.120 Well, real is kind of subjected, but with with sites that were complex enough to show me what we could and couldn't do. 00:09:25.120 --> 00:09:36.421 I went to a lot of sites like Wikipedia, create dot Weavely. Org the fluid project homepage including pages that use UIO. 00:09:36.421 --> 00:09:52.671 And then major search engines because part of what I wanted to do was make it so that navigating included Knowing what page you wanted to go to searching for it and clicking that link. 00:09:52.671 --> 00:10:00.270 So I did this over and over again and made a note of things that didn't work. And fixed as many issues as I could and although there are still a lot of room for improvement you can operate a lot of sites without ever taking your hands off the keyboard and reaching for a mouse. 00:10:00.270 --> 00:10:07.870 Sorry, taking your hands off the game pattern, reaching for a mouse or keyboard. So that's a lot of talking, but now we go for. 00:10:07.870 --> 00:10:16.570 Demo time. So let me. Switch away and I'm gonna have to reconfigure briefly. 00:10:16.570 --> 00:10:24.270 This is the point where You might want to PIN my video. Cause it's not just going to be talking. 00:10:24.270 --> 00:10:32.120 So. Just a moment when we wake up my game pad. I've been talking for long enough that it's decided to take a break. 00:10:32.120 --> 00:10:40.420 Okay, so on the bottom right you see representation of the game pad. That will. 00:10:40.420 --> 00:10:48.819 Respond as I click. Buttons or it should. I hope it will. 00:10:48.819 --> 00:10:54.369 Alright, well hopefully that will be visible enough. So let me focus back here. Right now I'm in the. 00:10:54.369 --> 00:10:58.870 Configuration interface, which I'll show you in a minute. I'm just scrolling up for a moment. 00:10:58.870 --> 00:11:06.920 The first thing I wanted to show you is the action launcher which is bound to the start button by default. 00:11:06.920 --> 00:11:14.420 If I press the start button I get a list of all the actions. That I can perform, I'm going to. 00:11:14.420 --> 00:11:23.120 Open a new window by hitting the click button, which is the bottom button. In the right hand cluster. 00:11:23.120 --> 00:11:29.221 Is we kept the I kept the default from the last time. It switched to dark mode because it's 8 pm here, but, this is Google search. 00:11:29.221 --> 00:11:37.720 It's in Dutch, but I think most of you will understand what's happening here. If I click on this text input that I'm on. 00:11:37.720 --> 00:11:55.921 Automatically the keyboard pops up and I can search for So, and let me just explain about the modules it is. 00:11:55.921 --> 00:12:02.370 Arrow and bumper navigable or you could think of it as air and tab navigable tab moves between the controls and The keys, so the buttons and the, the cancel button update feel button and the key so that you can easily move back and forth and then within the keyboard you move with the errors and the arrows. 00:12:02.370 --> 00:12:14.119 Because we can now simulate sending keys, the errors are now bound to the DPAD, which stimulates sending. 00:12:14.119 --> 00:12:23.991 The appropriate arrow key. To the on screen elements. So if I search by hitting the click button, if I search for IDRC. 00:12:23.991 --> 00:12:30.820 I'm gonna intentionally make a typo to show you. Something. This actually does support. Changing the cursor position. 00:12:30.820 --> 00:12:38.770 And fixing typos without having to erase everything. Which I wish Apple TV and other things did. 00:12:38.770 --> 00:12:44.920 So if I update the field, you can see that the search input now has what I was searching for if I use the bumpers. 00:12:44.920 --> 00:12:53.620 I can navigate and activate the search button. And there we have it. I'm able to search. 00:12:53.620 --> 00:12:59.821 And navigate through a range of pages. Including something as complex as the Google search. With the game pad. 00:12:59.821 --> 00:13:12.421 So I'm not gonna do a lot of real world demos if you want to see more examples of how this works. 00:13:12.421 --> 00:13:16.520 You should check out the series of YouTube videos I recorded. Which go through all of the functions to manage a single page, to manage individual pages, to work with videos and also to work with the Xbox Adaptive Controller. 00:13:16.520 --> 00:13:25.869 But for now, I'm gonna go back to open the settings panel, which I can do with the launcher. 00:13:25.869 --> 00:13:28.720 There is a key for it. But let's. I've been recording, doing a bunch of dry run, so I ended up focused in a weird area. 00:13:28.720 --> 00:13:38.020 Let me just navigate back to the top. 00:13:38.020 --> 00:13:45.720 We start with general preferences and what I want to show you here is this is a range input. And this supports arrows. 00:13:45.720 --> 00:13:53.671 So I'm using the DPAD to change the value. Of the range input. If I find a toggle like this. 00:13:53.671 --> 00:14:03.120 If I click it with the click button. It behaves as a toggle wood if I clicked it with the mouse or activated it with the keyboard. 00:14:03.120 --> 00:14:13.321 I've shown you entering text, which I can also do here. I will mention you can turn off arrow navigation through the features that I talked about. 00:14:13.321 --> 00:14:19.520 I'm not going to I can't really demonstrate it because you have to hold. The controller on the microphone, but it does actually support vibration when I if like for example, I don't figure who this. 00:14:19.520 --> 00:14:27.469 When I scroll up and there's no nowhere left to go, it vibrates to let me know that it can't do what I wanted to do. 00:14:27.469 --> 00:14:36.570 And this is configurable. Here. And then the rest of the interface lets you control what bindings. 00:14:36.570 --> 00:14:42.220 Are, available. So bindings are just again. The button you press or the thumb stick you press and then what I should do. 00:14:42.220 --> 00:14:49.272 And I'll show you one of the key features here. 00:14:49.272 --> 00:14:54.721 Alright, nearly all actions are repeatable. It used to be that thumb sticks were the only things that repeated. 00:14:54.721 --> 00:15:02.520 So when you were scrolling, if you held down. It would keep scrolling. So often. 00:15:02.520 --> 00:15:12.670 And now anything can be repeatable and you can you can choose to turn it off. You can say I don't want it to repeat. 00:15:12.670 --> 00:15:17.320 Or assisted default for the bumpers. Repeat every point 5 s. So if I hold down one of the bumpers It keeps going. 00:15:17.320 --> 00:15:27.670 You can turn this as fast as you want. You make it as slow as you want. Which means you can have kind of a scanning like. 00:15:27.670 --> 00:15:40.320 Way of going through. The interface just by holding a button and letting go. 00:15:40.320 --> 00:15:45.520 Right, so I realize that's a lot. Again, there's YouTube videos that go into a lot more of the the the features but I think that gives you a good feel for what it's possible to do. 00:15:45.520 --> 00:15:55.020 With the new version and I am going to switch back to. 00:15:55.020 --> 00:16:01.621 My main screen and. Switch over to questions and comments. I'm actually gonna have to kill the slide so that I can see hold of participants. 00:16:01.621 --> 00:16:09.369 So here we go. I tried to keep it short and know it's kind of a whirlwind but I wanted to make sure there's plenty of time for. 00:16:09.369 --> 00:16:22.069 People to ask questions and Anyway. 00:16:22.069 --> 00:16:24.119 Anyone? Hello? 00:16:24.119 --> 00:16:41.920 And you go ahead. 00:16:41.920 --> 00:16:49.992 Hey, Tony, this is Daniel. Yeah. Thanks for the demo. As an example, you showed us like opening a new window and I was wondering if you can like go back to the previous window or is there a way to view or the windows those are open? 00:16:49.992 --> 00:16:57.471 Right now there's not a way to view all the windows that are open, but I'll show you there definitely is a way to switch to the previous window next window, previous tab, next tab. 00:16:57.471 --> 00:17:06.370 Close the current tab, close the current window. I added the the an action to duplicate the current tab. 00:17:06.370 --> 00:17:14.670 You also have the ability to move back and forward in history. So I thought of that in some kind of menu to let you Go through the whole list of open. 00:17:14.670 --> 00:17:17.992 Windows and tabs. In fact, I've got our feature request written up for that. But for now, you have to switch through it one window at a time. 00:17:17.992 --> 00:17:27.530 I see. Good, thank you. 00:17:27.530 --> 00:17:33.020 And Joseph? 00:17:33.020 --> 00:17:37.520 Thanks. That is great Tony. May, have missed it, but what can you do with the thumbs? 00:17:37.520 --> 00:17:52.970 Oh, that's a great point. Let me just scroll down. I'll show you some of the there's unique features. 00:17:52.970 --> 00:17:55.270 For the thumb sticks which are kind of You can think of them as wrappers. So for example, if there's an action for a button to go up or down, you have an action for the thumb stick where you can Go either up or down. 00:17:55.270 --> 00:18:03.819 Oh, sometimes this happens and I don't know why. 00:18:03.819 --> 00:18:07.119 There's a subtle focus problem which happens when I switch back and forth a lot. Okay, so, for example, you can scroll right. 00:18:07.119 --> 00:18:08.020 No, no, no. 00:18:08.020 --> 00:18:14.623 You can scroll. How did I get back there? No, no, no, I'm on buttons and triggers again. 00:18:14.623 --> 00:18:16.820 Thank you, time. 00:18:16.820 --> 00:18:23.222 Sorry, just like. Okay. So you can, I don't know what is going on with the focus here. 00:18:23.222 --> 00:18:28.921 I need to look into this, but, You can see you can navigate through history with a stick input. 00:18:28.921 --> 00:18:39.070 So in other words, I found that one to be kind of. Disorienting, so I didn't find that by default. 00:18:39.070 --> 00:18:45.469 You can send errors. With a with a stick input you can move through focus so you could use at a stick input to tab navigate basically. 00:18:45.469 --> 00:18:53.471 And then you have several scrolling options and then you have a few things about zooming in and out and then changing the window size. 00:18:53.471 --> 00:19:03.620 So the window size that just means maximize or not maximized. But those were some of the features that we had before and then preserved here. 00:19:03.620 --> 00:19:07.470 The only one that is bound by the fault. Is is The left stick is found to scrolling by default. 00:19:07.470 --> 00:19:14.521 Scrolling perfectly. Okay. 00:19:14.521 --> 00:19:24.970 So yes, exactly. I found that if you had scrolling horizontally bound as well, what ended up happening is you would try to scroll down and accidentally scroll left and right. 00:19:24.970 --> 00:19:28.270 And then an awful lot of pages don't really support scrolling left and right anyway so you can add it if you need it, but by default, vertical scrolling is the only one that's. 00:19:28.270 --> 00:19:38.471 Got it. 00:19:38.471 --> 00:19:41.769 Cool. I'm interested in what we're working on something like a keyboard and I would love to be able to use something like a joystick to scan up, down, left, right, and diagonally. 00:19:41.769 --> 00:19:44.770 Right. 00:19:44.770 --> 00:19:56.620 To get to to get the keys rather than tabbing endlessly. 00:19:56.620 --> 00:19:58.819 Right and and that's part of the reason I went for error navigation for some things is that Tabbit you end up having an inlessly if you don't have something like errors occasionally available in your design toolkit. 00:19:58.819 --> 00:20:01.770 Yeah. 00:20:01.770 --> 00:20:09.271 And so for something like the I'll go back to the to the on-screen keyboard, something like the onsite keyboard. 00:20:09.271 --> 00:20:14.869 When I start, I was so excited to add this and so happy to have it and then I tried to use it and I was sitting here tabbing through. 00:20:14.869 --> 00:20:25.920 Right. Right? 00:20:25.920 --> 00:20:31.823 100 keys to get to the update field button and then going back and forth and it was it was like every nightmare you've had about using a badly designed remote control search, you know, like where you, you're trying to search a YouTube video by hitting 2 buttons backward and forward. 00:20:31.823 --> 00:20:33.570 So this turned out to be a decent pattern. This happened this, DPAD. 00:20:33.570 --> 00:20:37.670 Up and down, left, right, yeah, okay. 00:20:37.670 --> 00:20:47.520 Yeah, yeah. I mean it does actually support I'll show you it does actually support. 00:20:47.520 --> 00:20:55.020 Shift and caps like as well. I the only thing that I realized far too late is that I didn't have any mechanism for international characters, which obviously I need to fix. 00:20:55.020 --> 00:21:04.269 But anyway, I'll show you a feature that I have training before. I, because we now have the ability to send keys by default. 00:21:04.269 --> 00:21:09.670 Right. 00:21:09.670 --> 00:21:18.071 The right hand button hits escape, which it. So you can close the modal without. Just so, so the demo that I did for the Xbox Adapt controller, the 2 big buttons are bound to click and escape. 00:21:18.071 --> 00:21:26.070 The D-pad does what you saw here and then I just attached bumpers. So with basically I think the 4 directions of the DPAD. 00:21:26.070 --> 00:21:31.871 One button to click and one button, 2 buttons for tab navigation and then something for the action launcher. 00:21:31.871 --> 00:21:36.020 So. You know, 7 or 8 buttons, you can actually do everything. Everything else is just. 00:21:36.020 --> 00:21:39.619 What you prefer to have easy access to. 00:21:39.619 --> 00:21:42.319 Okay, thanks. 00:21:42.319 --> 00:21:50.321 Sure. 00:21:50.321 --> 00:22:00.171 Hi, this time just to sort of jump off of what Joseph was asking. In a way, regarding the joysticks. 00:22:00.171 --> 00:22:05.170 I know what the XC there is a X one and X 2 jack. Are those still usable as the joysticks or can you only go to the USB ports on the side if you want to use any of these stick. 00:22:05.170 --> 00:22:06.969 I've never had anything that would work in either of those ports. So I'm not sure. 00:22:06.969 --> 00:22:15.169 Okay. 00:22:15.169 --> 00:22:19.670 I know that it will work with the USB joystick. And I know that you can, you can get breakout boards to let you turn nearly anything. 00:22:19.670 --> 00:22:33.320 But I don't know. I haven't looked at it recently to see how that works. 00:22:33.320 --> 00:22:41.989 I think in part it's because any of the analog controls, they need to have a certain range of resistance and I never figured out the right level of resistance to make something act as an analog input even the triggers I could only get them to do. 00:22:41.989 --> 00:22:48.670 Binary inputs and I think for something like an access you would want It should be able to deliver a range of, I guess. 00:22:48.670 --> 00:23:01.770 Voltages. Yeah, so I, I don't know. I don't know. 00:23:01.770 --> 00:23:07.120 Okay. No, no worries. We've been No, I guess we have an adaptive game program here and we've been using some of the joysticks for makers making change and they lot of their joysticks go through the X one and X 2 and sports and I was just wondering it 00:23:07.120 --> 00:23:08.520 Yes. 00:23:08.520 --> 00:23:15.269 Oh, thank you very much. I'm gonna write that down. Makers making change. Making change joysticks. 00:23:15.269 --> 00:23:15.720 Okay, so, Wonderful. 00:23:15.720 --> 00:23:22.071 Yup, they're working on a whole plethora of joysticks right now. For their, they have an adaptive game going on. 00:23:22.071 --> 00:23:29.994 We're actually partnered with them here. As one of their game checkpoints where one of 8 across the country. 00:23:29.994 --> 00:23:33.720 So they're doing a lot of work in the adaptive gaming world right now. And yeah, yes, I'm trying to see if any of the products will work. 00:23:33.720 --> 00:23:36.420 I'm currently in the augmented of an alternate communication. Field right now as well. 00:23:36.420 --> 00:23:46.321 Nice. 00:23:46.321 --> 00:23:53.720 So I'm trying to pull some of their stuff into working with computers as well. So this is very interesting to hear all your work on this and I'd be I'll probably be exploring myself with some of us up here. 00:23:53.720 --> 00:24:06.320 Sounds good. I would like to mention that there are operating system specific plugins that will let you Have a joystick. 00:24:06.320 --> 00:24:11.920 Game Pad act as a keyboard and or mouse. I haven't found one that's particularly well supported, especially in an environment like OSX where you get breaking changes every couple versions of. 00:24:11.920 --> 00:24:19.670 Of the operating system, but there are at least some solutions out there that will let you handle it at the operating system level. 00:24:19.670 --> 00:24:27.520 So one thing I can't do is change from a browser to any other browser or any of the program, I can't control any of the program. 00:24:27.520 --> 00:24:31.069 But a system like that, there's like joy key and a few others will let you. Do that. 00:24:31.069 --> 00:24:33.819 Yep. 00:24:33.819 --> 00:24:38.619 Yeah, I just discovered that one, right? I recently, well, I was, searching for Ig's, technologies and gaming as well. 00:24:38.619 --> 00:24:41.219 I found that one. So I'm planning on playing around with that one. So I'm planning on playing around with that one in the next few weeks as well. 00:24:41.219 --> 00:24:51.170 Cool. 00:24:51.170 --> 00:24:59.992 So yeah. No, but this is very, interesting. And I think Even if you're, as you said, the Chrome in itself is quite complex, so to be able to navigate it. 00:24:59.992 --> 00:25:00.420 Sure, please. 00:25:00.420 --> 00:25:05.420 So cleanly using a game pad is quite good. And Sorry, I have one more question about I'll save that after Sunday. 00:25:05.420 --> 00:25:15.169 Okay. 00:25:15.169 --> 00:25:20.870 Actually team, if you I also have multiple questions. So Tim, if you wanna go ahead with your next question, I'm totally fine. 00:25:20.870 --> 00:25:26.171 Okay, thank you. I just want to quickly ask you mentioned that it's also works with the PlayStation access controller and the Hori. 00:25:26.171 --> 00:25:33.219 I will, I it should. I don't actually own all 3 of them. I just wanted to mention it because It should 00:25:33.219 --> 00:25:34.919 If you can. 00:25:34.919 --> 00:25:44.471 Work with anything that can connect. So I have I have Too many game controllers. I don't, but I can't justify spending another. 00:25:44.471 --> 00:25:51.270 3 or 400 to get those 2. So I mainly focus on the first one that was available. 00:25:51.270 --> 00:25:53.620 But I have, I'm I have PlayStation controllers and if it acts like a PlayStation controller, it should work with the gamepit navigator. 00:25:53.620 --> 00:25:57.020 I have switch controllers. If it acts like a switch controller, it should work with the Gamepad Navigator. 00:25:57.020 --> 00:25:59.920 Okay. 00:25:59.920 --> 00:26:06.569 So I would love for someone to tell me How it is. 00:26:06.569 --> 00:26:06.973 One wonderful. 00:26:06.973 --> 00:26:11.069 If you, if you wanna connect, I do have access to all 3 through our game program here. So. 00:26:11.069 --> 00:26:11.270 I think you and I have a lot to talk about. 00:26:11.270 --> 00:26:14.275 So, that's what you want. 00:26:14.275 --> 00:26:18.670 Worked at writing AAC apps in the past. So it seems like we have a lot to talk about. 00:26:18.670 --> 00:26:21.469 Yeah, so definitely we can talk. On a separate thing as well. 00:26:21.469 --> 00:26:27.919 Great, sounds great. Cool. 00:26:27.919 --> 00:26:32.919 Okay, so and, Tony, thank you for. 00:26:32.919 --> 00:26:34.969 Demonstrating this. This is really great. 00:26:34.969 --> 00:26:37.669 Yep. 00:26:37.669 --> 00:26:38.970 Yes. 00:26:38.970 --> 00:26:43.120 So one question I have is I'm not familiar with their like, game contours like using a in path, we're using PlayStation controllers. 00:26:43.120 --> 00:26:51.219 So I wonder how does it get connected to the Chrome? Like 00:26:51.219 --> 00:26:51.570 But you that's it. 00:26:51.570 --> 00:26:59.771 Right, right. So you have to do, you have to connect it to your operating system first. And in most cases, we're talking about or Bluetooth. 00:26:59.771 --> 00:27:03.619 And then once you've done that bit of setup, it works through the gamepad API, which has been supported in most browsers for quite a few years. 00:27:03.619 --> 00:27:17.420 So basically 00:27:17.420 --> 00:27:33.471 Any connected controller, you have to move it for it to take for it to do something, but any connected controller, is visible to your browser through the window navigator. 00:27:33.471 --> 00:27:40.821 There's an API to check for connected controllers and to read all of the values and to vibrate them and things like that. 00:27:40.821 --> 00:27:43.320 Really good good good features and that that's probably why that you can use. Like, you can use, browser or you can use the tab to do their gimpad configuration so you can send those configuration back through the API to controller game pad. 00:27:43.320 --> 00:27:50.320 Behavior. So is that 00:27:50.320 --> 00:27:53.073 Right, and there's only a few things that you can do to the game head. Like I've never found a way to play sound for example through the 00:27:53.073 --> 00:28:21.572 Getting paid. 00:28:21.572 --> 00:28:24.119 Which a lot of G pads support headphone jacks. I haven't found a way to do that. 00:28:24.119 --> 00:28:29.569 But vibration is pretty much the only thing you can But really the only thing you can send back is. 00:28:29.569 --> 00:28:32.321 Indication of whether it 00:28:32.321 --> 00:28:42.719 Whether or not to vibrate and unfortunately There's only one profile that works really. So it's it can't even do some of the subtle vibrations that newer controllers that the switch controller can do and it can't do complex. 00:28:42.719 --> 00:28:50.270 Triggers and haptics. Like something like the the PlayStation 5 controller can do so it's in a way it's kind of a least common denominator for kind of became a few years ago, but it's still very powerful and the advantages, if you're curious about it, you can find a compatible controller. 00:28:50.270 --> 00:28:55.985 Even at a thrift store or Sorry, yard sale, which I often do. For, you know, lunch money. 00:28:55.985 --> 00:29:09.569 It's, it's not an expensive hobby. Compared to synthesizers. 00:29:09.569 --> 00:29:11.421 Yeah. 00:29:11.421 --> 00:29:14.219 Yeah, I'm, so curious to find out like how well the support is for like the EC devices, like I, IDs devices or switches and those kind of things that it would be very interesting to find out. 00:29:14.219 --> 00:29:22.471 So. 00:29:22.471 --> 00:29:26.819 I know there are people working to connect. I gaze devices I think to the to the Xbox control in particular because it's very well supported on Windows. 00:29:26.819 --> 00:29:35.420 I have seen articles about that, but I think probably Tim could comment better on that. 00:29:35.420 --> 00:29:44.169 Yeah, right now I think actually the, Toby has released, well, Toby has her AGE camera and that's here. 00:29:44.169 --> 00:29:47.820 Oh nice. 00:29:47.820 --> 00:30:03.620 I think right now they're mostly focusing on PC and actually through the Hori Flex controller for the attendance switch actually so they seem to have that down that part figured out. 00:30:03.620 --> 00:30:11.021 They haven't touched too much on the X the XC the Xbox adaptive controller yet I do know that they also just have their own software that helps do a lot of games where looking around using a secondary joystick is a problem. 00:30:11.021 --> 00:30:17.370 Now you can just use your eyes and head movements to control that right choice. I know they've done that. 00:30:17.370 --> 00:30:23.320 I'll be playing around. I love this is more things I've seen. I plan to actually try them out in the next few weeks. 00:30:23.320 --> 00:30:27.219 But yeah. 00:30:27.219 --> 00:30:36.070 Great. Yeah, unfortunately Justin just left, but he and I did do a little work to actually push out the new version. 00:30:36.070 --> 00:30:44.219 I wanted to give everyone a chance to see it and maybe. To a little less. Finger crossing in QA, but we should be pushing out 1 point out where you could just install it from the Google. 00:30:44.219 --> 00:30:52.576 From web stored in the next week or so. And just try it. I'd love to hear what you think. 00:30:52.576 --> 00:30:56.520 Yeah, that's cool. I would love to try. And so, and Tony, I have another question about their, onscreen keyboard. 00:30:56.520 --> 00:31:03.820 Yes. 00:31:03.820 --> 00:31:14.272 So whenever you focus on our tax field, Issac when I, actually my question is, How is it? 00:31:14.272 --> 00:31:23.221 Is it working like whenever you focus on our tax field on our page? The onscreen keyboard we're But. 00:31:23.221 --> 00:31:30.920 Right, right. So we, yeah, yeah, we, we have what are called actions, which basically one of these is a click action and this click action It does. 00:31:30.920 --> 00:31:35.270 It has to check and see what kind of element it's focused on and it decides to do a few special things for different elements. 00:31:35.270 --> 00:31:43.270 And one of those things is for a text input and I have a whole bunch of rules about what a text input could be. 00:31:43.270 --> 00:31:52.428 If it thinks it's a text input. Then when you click it won't send a click to that element. 00:31:52.428 --> 00:32:00.520 Instead, it will try to open the on screen keyboard. If it's a link or a button, and it knows that it's not a text input and it will just trigger the click activity of that element. 00:32:00.520 --> 00:32:07.570 So it's based on what you're focused on and it has rules around. What it should do when you click on it. 00:32:07.570 --> 00:32:13.121 Yeah. That's really cool. So, yeah, is there written in Chrome extension? 00:32:13.121 --> 00:32:19.272 Right, so, there's, it is, it is a chrome extension and there's 2 parts of the chroma extension. 00:32:19.272 --> 00:32:32.820 There's the part that is injected into the page. Which for this version is injected into a, an element with a shadow root. 00:32:32.820 --> 00:32:44.272 So that it doesn't affect the markup of the page any more than we can help. And then there's a part that lives in the background, which is in the case of Chrome, they've migrated to using service workers. 00:32:44.272 --> 00:32:56.470 So there's a service worker that has access to different APIs and then there's a mechanism for communicating between the client side where the code is injected and the service worker. 00:32:56.470 --> 00:33:06.270 And, and in this version, we, it's actually bi-directional. So that if you tell the service worker to do something and it knows it can't, it sends information back to the client side to say I couldn't do that and then the client side vibrates the controller. 00:33:06.270 --> 00:33:12.671 So there's There's kind of a couple of moving parts, but it's all, stuff that is published as part of the I guess they call it the web extension API, which is mostly just. 00:33:12.671 --> 00:33:21.720 A curve extension that now seems to work. Almost everywhere. 00:33:21.720 --> 00:33:34.469 Yeah, thank you. Yeah, this is very cool. Yeah, that's where I wanna ask about. 00:33:34.469 --> 00:33:40.670 So adding questions from anyone? 00:33:40.670 --> 00:33:51.020 No. 00:33:51.020 --> 00:33:59.992 I'm bright. Well, thank you everyone for your time. It's been really great. I always enjoy hearing your questions as I as I pointed out we're gonna have a release very soon where you won't have to do much to try it. 00:33:59.992 --> 00:34:09.020 There's also ways to install it and try the version that we're working on. That they're not any more difficult than anything else we have to do as developers. 00:34:09.020 --> 00:34:18.119 So, I would be happy for any of you to try it. I'd love to hear what you think, the project and all of the feature requests and issues are tracked on GitHub. 00:34:18.119 --> 00:34:24.723 And, I'm always open to. New ideas about What could work better? 00:34:24.723 --> 00:34:24.769 Cool. Thank you very much. 00:34:24.769 --> 00:34:26.119 Thank you. 00:34:26.119 --> 00:34:29.119 All right, thanks everyone.