WEBVTT 00:00.000 --> 00:08.000 The next one is the state of open source XR, 00:08.000 --> 00:12.000 and another million by Christoph Hack. 00:12.000 --> 00:13.000 Yeah. 00:13.000 --> 00:15.000 Thanks for being through. 00:15.000 --> 00:17.000 You took my words for me. 00:17.000 --> 00:20.000 Thank you. 00:20.000 --> 00:21.000 Yeah. 00:21.000 --> 00:24.000 As I was introduced, my talk is about the state of open source XR, 00:24.000 --> 00:28.000 which is basically about Monado and what you can do with it. 00:28.000 --> 00:35.000 My name is Christoph Hack, and I've been working at Colabora for Monado for a long time now. 00:35.000 --> 00:41.000 So in this talk, we will go very, very quickly about the R hardware and tracking systems, 00:41.000 --> 00:46.000 because this is important for the open source of their support, 00:46.000 --> 00:52.000 which is basically the bigger half is exactly these tracking systems, 00:52.000 --> 00:55.000 and these smaller half is basically everything else, 00:55.000 --> 00:58.000 like how to communicate over your speed of these, 00:58.000 --> 01:00.000 how to correct what the lenses do. 01:00.000 --> 01:04.000 Then we will take a quick look at the state of Monado, 01:04.000 --> 01:08.000 and then lastly, what you can actually do with it, 01:08.000 --> 01:11.000 what how does the ecosystem look like. 01:11.000 --> 01:16.000 hardware in the R starts 60 years ago, 01:16.000 --> 01:20.000 but for this talk, this is basically uninteresting, 01:20.000 --> 01:23.000 because these old VR heads are super, 01:23.000 --> 01:27.000 like the super loud resolution, if they are affordable, 01:27.000 --> 01:32.000 and only in 2012, then the Oculus Rift DK1 was released. 01:32.000 --> 01:35.000 Smartphone displays, they are coming up, 01:35.000 --> 01:38.000 and they are small, and allow latency and low persistence, 01:38.000 --> 01:43.000 you then get ghosting, so it's also finally a good experience with those. 01:43.000 --> 01:46.000 So this Oculus Rift DK1, 01:46.000 --> 01:48.000 only then I am new. 01:48.000 --> 01:50.000 This means this has that noise, 01:50.000 --> 01:53.000 because VR gravity is because gravity is an acceleration, 01:53.000 --> 01:56.000 and it can track your rotation, 01:56.000 --> 01:58.000 but it doesn't know your position. 01:58.000 --> 02:00.000 If you take a step to the left, 02:00.000 --> 02:05.000 it has that does not know that it needs to kind of be involved to the right. 02:05.000 --> 02:13.000 This tracking system system is not developed in commercial, 02:13.000 --> 02:17.000 consumer heads, there are basically three systems for this. 02:17.000 --> 02:21.000 One was LED tracking, 02:21.000 --> 02:23.000 nicknamed constellation tracking. 02:23.000 --> 02:27.000 The second one was light horse tracking, 02:27.000 --> 02:29.000 and the third one is slim tracking. 02:29.000 --> 02:32.000 We will take a very, very quick look at those. 02:32.000 --> 02:35.000 LED tracking looks like this. 02:35.000 --> 02:38.000 In the second iteration of their headset, 02:38.000 --> 02:40.000 Oculus put LEDs on the headset, 02:40.000 --> 02:42.000 and ship it with a camera, 02:42.000 --> 02:44.000 but watchers these LEDs, 02:45.000 --> 02:50.000 and because these software allows the configuration of the LED, 02:50.000 --> 02:55.000 you can basically take what's in for how this headset, 02:55.000 --> 02:57.000 very little is positioned in space, 02:57.000 --> 02:59.000 if you have an image of it, 02:59.000 --> 03:02.000 and you use computer vision for that. 03:02.000 --> 03:07.000 Valve came out with the light horse tracking system, 03:07.000 --> 03:09.000 you don't have a camera for that, 03:09.000 --> 03:12.000 you have these boxy, 03:13.000 --> 03:16.000 stations that sweep your room with lasers, 03:16.000 --> 03:18.000 they don't send anything to your PC, 03:18.000 --> 03:22.000 everything that is sent to the PC comes from the headset. 03:22.000 --> 03:25.000 There are sensors on the headset that detect 03:25.000 --> 03:29.000 when the lasers sweep of these stations hit each sensor, 03:29.000 --> 03:31.000 and you do basically the same, 03:31.000 --> 03:35.000 if it was infer how your headset has to be positioned 03:35.000 --> 03:40.000 for this timing of each sensor being hit to be possible. 03:42.000 --> 03:46.000 And the third one is slam tracking. 03:46.000 --> 03:51.000 Microsoft started with a whole suit of vendors 03:51.000 --> 03:54.000 who made headsets for this. 03:54.000 --> 03:56.000 This one in the image is from ASA, 03:56.000 --> 04:00.000 I have one from Neodyon here. 04:00.000 --> 04:04.000 You don't see any camera or base station in the room here. 04:04.000 --> 04:07.000 You only have camera as on the headset itself, 04:07.000 --> 04:11.000 and you'll use a more complicated computer vision algorithm. 04:11.000 --> 04:14.000 To detect features in your environment, 04:14.000 --> 04:17.000 feature something like an edge or a corner, 04:17.000 --> 04:21.000 and apparently it's possible to detect these from different angles. 04:21.000 --> 04:26.000 So if you store all of these features that you have detected 04:26.000 --> 04:30.000 and can read detect them, then you can also detect your position 04:30.000 --> 04:32.000 relative to them. 04:32.000 --> 04:35.000 If this was too fast, then don't worry, 04:35.000 --> 04:37.000 this was just an overview. 04:37.000 --> 04:42.000 And what you would take from it is that there are basically three tracking systems 04:42.000 --> 04:47.000 that you need to implement if you want to cover all the health advantage out there. 04:47.000 --> 04:51.000 Here's a list of a lot of these headsets. 04:51.000 --> 04:55.000 I'm not going over all of them in the lighthouse column. 04:55.000 --> 04:57.000 You have a lot of high end headsets, 04:57.000 --> 04:59.000 and enthusiast class headsets. 04:59.000 --> 05:01.000 On these slam tracking columns, 05:01.000 --> 05:04.000 you have more affordable ones. 05:04.000 --> 05:09.000 And LED tracking has been kind of all out of favor, 05:09.000 --> 05:11.000 except for one case. 05:11.000 --> 05:15.000 And that is even those mixed reality headsets 05:15.000 --> 05:17.000 are slam tracking themselves. 05:17.000 --> 05:22.000 But you can see that the other person has controllers in their hands, 05:22.000 --> 05:27.000 and there are LEDs on the control of us. 05:27.000 --> 05:31.000 And the cameras on the headsets will track these controllers. 05:31.000 --> 05:34.000 So even if you have a slam track headset, 05:34.000 --> 05:41.000 you still need LED tracking algorithm to track the controllers for this headset. 05:41.000 --> 05:44.000 There are also standalone headsets. 05:44.000 --> 05:48.000 Everything I mentioned so far is your plug-in and HDMI, 05:48.000 --> 05:51.000 HDMI and this platform, 05:51.000 --> 05:54.000 or USB to your PC, 05:54.000 --> 05:57.000 and the PC does everything. 05:57.000 --> 06:00.000 And the lower headsets, like the Metacrest picture here, 06:00.000 --> 06:04.000 basically have an Android device inside them. 06:04.000 --> 06:07.000 These are typically bootloader locked. 06:07.000 --> 06:08.000 You can't loot them. 06:08.000 --> 06:10.000 You can't let your own software on them. 06:10.000 --> 06:13.000 So they are not very open source friendly. 06:13.000 --> 06:15.000 But there are some exceptions. 06:15.000 --> 06:18.000 The links are one for example. 06:18.000 --> 06:20.000 Links is an European startup, 06:20.000 --> 06:23.000 and they had a lot of issues with the supply chain 06:23.000 --> 06:24.000 and shipping them. 06:24.000 --> 06:27.000 It's unfortunately very hard to get one of those. 06:27.000 --> 06:29.000 But if you get one of those, 06:29.000 --> 06:31.000 here's a first market link. 06:31.000 --> 06:32.000 You can install it. 06:32.000 --> 06:34.000 It's not very useful yet. 06:34.000 --> 06:36.000 But you can hack on it. 06:36.000 --> 06:41.000 That's also the steam frame on this slide here. 06:41.000 --> 06:45.000 Steam frame is what the release very soon. 06:45.000 --> 06:48.000 We don't know exactly yet for the outgoing to ship. 06:48.000 --> 06:51.000 Most likely they will not ship their slam tracking 06:51.000 --> 06:54.000 or their controller tracking open source. 06:54.000 --> 06:58.000 But probably they will have more open hardware platform 06:58.000 --> 07:02.000 than most, and will be hackable. 07:02.000 --> 07:06.000 So how does this software for all of this look like? 07:06.000 --> 07:10.000 In 2013, it allows to be an announcement 07:10.000 --> 07:12.000 of the Oculus Rift DK1. 07:12.000 --> 07:15.000 Open H&B was started. 07:15.000 --> 07:20.000 You can see a picture here that was taken at first in 2019. 07:20.000 --> 07:26.000 I am behind the VLC guy of the traffic cone. 07:26.000 --> 07:29.000 It was in the changing over there. 07:29.000 --> 07:32.000 But open H&B is not very active anymore. 07:32.000 --> 07:35.000 It's no basically unmeted. 07:35.000 --> 07:39.000 Who has the R was another SDK that came before. 07:39.000 --> 07:42.000 Kronos made a unified API. 07:42.000 --> 07:45.000 That was an effort by our company called SENSIX, 07:45.000 --> 07:49.000 which made the software and in cooperation with Razor, 07:49.000 --> 07:52.000 who made a headset. 07:52.000 --> 07:54.000 And that complete came up tracking solution 07:54.000 --> 07:57.000 and complete open source driver. 07:57.000 --> 08:01.000 But for one another reason they didn't make it in the market. 08:01.000 --> 08:03.000 And they seized operations. 08:03.000 --> 08:07.000 And OS for the RS basically unmeted to now. 08:07.000 --> 08:12.000 I say basically because we have one of the original main trainers 08:12.000 --> 08:14.000 of OSVR and Cullabora. 08:14.000 --> 08:16.000 And she has the foot access. 08:16.000 --> 08:21.000 And sometimes maintains it a little bit, but not much anymore. 08:21.000 --> 08:24.000 In comes Monado. 08:24.000 --> 08:28.800 Kronos in 2018 finally made the push to make 08:28.800 --> 08:35.000 a unified API for XR. 08:35.000 --> 08:38.000 And there is Mesa for OpenGL. 08:38.000 --> 08:42.000 In full can he wanted to have an open source implementation 08:42.000 --> 08:44.000 for OpenXR. 08:44.000 --> 08:46.000 And that's how Monado was born. 08:46.000 --> 08:49.000 And it's loosely based on the Mesa design. 08:49.000 --> 08:52.000 For example, how it implements OpenXR 08:52.000 --> 08:54.000 is we are a state tracker. 08:54.000 --> 08:58.000 Just like these are implement APIs and state strikers. 08:58.000 --> 09:02.000 There's a very long list of drivers that currently exist 09:02.000 --> 09:04.000 for diverse hardware. 09:04.000 --> 09:07.000 And of course, there's this incomplete. 09:07.000 --> 09:11.000 You can help by expanding it by writing a driver. 09:11.000 --> 09:19.000 And like I said, the bigger half of supporting VR hardware 09:19.000 --> 09:24.000 is the persistent tracking system. 09:24.000 --> 09:27.000 For LED tracking, we have a simple tracking framework 09:27.000 --> 09:30.000 that is dotted for PSVR. 09:30.000 --> 09:35.000 But I guess people thought it's not theory heavy enough. 09:35.000 --> 09:38.000 The board from this thing approach is what community 09:39.000 --> 09:41.000 then one named Faiton. 09:41.000 --> 09:44.000 Young Schmidt has started in OpenHMB. 09:44.000 --> 09:49.000 This is currently being plotted over to Monado in various branches. 09:49.000 --> 09:51.000 It's not obscene yet. 09:51.000 --> 09:55.000 But if anyone wants to help, then please come to our community 09:55.000 --> 09:59.000 generous and talk to the people who are doing this. 09:59.000 --> 10:03.000 For Lighthouse, there is an open source library. 10:03.000 --> 10:07.000 It's almost usable, but it's usual people who are 10:08.000 --> 10:12.000 who have this skill set are very busy and currently 10:12.000 --> 10:17.000 lips the wife is minimally maintained by Colabora. 10:17.000 --> 10:21.000 What most people actually use is a close source 10:21.000 --> 10:23.000 and a wrapper for steam. 10:23.000 --> 10:26.000 We are a close source lighthouse tracking 10:26.000 --> 10:32.000 because that's basically usable for using for playing real games. 10:32.000 --> 10:36.000 And for slim tracking, we chose an academic 10:37.000 --> 10:40.000 tracking system called basalt. 10:40.000 --> 10:43.000 We improved it, but it's still quite CPU heavy. 10:43.000 --> 10:45.000 Clicking quality is not quite reaching, 10:45.000 --> 10:51.000 commercial systems yet, but you can use it. 10:51.000 --> 10:54.000 At this point, I can maybe shower. 10:54.000 --> 11:01.000 Oh, this is actually looking light. 11:02.000 --> 11:06.000 With this headset here. 11:06.000 --> 11:09.000 It's very loud as a solution. 11:09.000 --> 11:14.000 So like I said, this slim tracking system is detecting 11:14.000 --> 11:17.000 stuff in the environment and can, 11:17.000 --> 11:22.000 oh, it's very, very loud as this screen. 11:22.000 --> 11:26.000 And it can determine its own position 11:26.000 --> 11:30.000 relative to all the features that are detecting 11:30.000 --> 11:35.000 in the environment. 11:35.000 --> 11:41.000 And more for that later, if I have the time, 11:41.000 --> 11:47.000 we also have a hand tracking implementation that I, 11:47.000 --> 11:49.000 geez. 11:50.000 --> 12:03.000 I don't know if you can actually see this in this lower screen here. 12:03.000 --> 12:05.000 Let's skip this. 12:05.000 --> 12:07.000 You just have to trust me on that. 12:07.000 --> 12:12.000 We trained model on open and on synthetic data 12:12.000 --> 12:16.000 that is included and usable with monado. 12:16.000 --> 12:20.000 So with these great cameras can watch your hands 12:20.000 --> 12:26.000 and make our hand model and detect very little joins are. 12:26.000 --> 12:35.000 Let's skip a little bit ahead because I'm that one time. 12:35.000 --> 12:39.000 One thing you can use whenado for is, 12:39.000 --> 12:41.000 if you were actually very tricky, 12:41.000 --> 12:45.000 or you could have seen an Android driver in the long list of drivers, 12:45.000 --> 12:48.000 you can install monado on an Android smart phone. 12:48.000 --> 12:51.000 And you get basically what Google had with Google 12:51.000 --> 12:54.000 cardboard, which was like this cardboard, 12:54.000 --> 12:58.000 little cardboard, smart phone holder. 12:58.000 --> 13:01.000 You can use this with monado, 13:01.000 --> 13:05.000 but I don't have an Android phone with me today. 13:05.000 --> 13:09.000 I have a first market OS smart phone. 13:10.000 --> 13:16.000 You can also run monado with an example application here. 13:16.000 --> 13:18.000 It's probably very small, 13:18.000 --> 13:21.000 but you can see that it's running. 13:21.000 --> 13:24.000 This is completely open source, 13:24.000 --> 13:28.000 stack here with 200 as the will can driver. 13:28.000 --> 13:31.000 So if anyone wants to build us, 13:31.000 --> 13:32.000 then allow a headset, 13:32.000 --> 13:37.000 you can use this as a platform on Linux or on Android. 13:40.000 --> 13:47.000 And if you want to start hacking on any of the drivers, 13:47.000 --> 13:50.000 or any of the other software in monado, 13:50.000 --> 13:54.000 then it's a very good idea to come to the community. 13:54.000 --> 13:56.000 This code is, 13:56.000 --> 13:58.000 where most of the people hang out, 13:58.000 --> 13:59.000 iOS is unfortunate, 13:59.000 --> 14:01.000 and he's not very much used. 14:01.000 --> 14:05.000 But we do pledge all of our discord channels to matrix two, 14:05.000 --> 14:08.000 so you don't have to actually sign up for this code. 14:10.000 --> 14:14.000 So there's a lot to be done for the future. 14:14.000 --> 14:17.000 The trend goes to mixed reality. 14:17.000 --> 14:19.000 There are headset, 14:19.000 --> 14:22.000 detect all sorts of stuff in your environment. 14:22.000 --> 14:25.000 They can detect keyboard, 14:25.000 --> 14:27.000 mouse, objects, 14:27.000 --> 14:29.000 semantically labeled them, 14:29.000 --> 14:33.000 construct a mesh from the environment. 14:33.000 --> 14:36.000 Stuff like this is not in monado yet. 14:37.000 --> 14:38.000 For a long time, 14:38.000 --> 14:41.000 it seemed this would be very difficult to implement, 14:41.000 --> 14:43.000 but in the last few years, 14:43.000 --> 14:46.000 the commutation of AI, 14:46.000 --> 14:48.000 looks very promising. 14:48.000 --> 14:53.000 AI is a lot more than just LLMs or image generators. 14:53.000 --> 14:56.000 A lot of AI is actually used for perception, 14:56.000 --> 15:03.000 and for making a model of the real world. 15:03.000 --> 15:06.000 That can be very helpful for open source, 15:06.000 --> 15:08.000 because it's not. 15:08.000 --> 15:11.000 You don't need our computer vision, 15:11.000 --> 15:14.000 PhD department to use these. 15:14.000 --> 15:18.000 So this could be also an interesting thing. 15:18.000 --> 15:22.000 So what can you actually do with it? 15:22.000 --> 15:31.000 One thing is that people want to use their desktop's, 15:32.000 --> 15:35.000 while they are in VR. 15:35.000 --> 15:39.000 You can imagine, if you put a headset like this on your head, 15:39.000 --> 15:41.000 you don't actually see your PC, 15:41.000 --> 15:44.000 you don't see your keyboard, you don't see your mouse, 15:44.000 --> 15:48.000 and people want to put their desktop windows into VR. 15:48.000 --> 16:06.000 So this is a demo video, 16:06.000 --> 16:09.000 I shot about two years ago. 16:09.000 --> 16:15.000 There's a Raspberry Pi connected to a Valve index headset, 16:15.000 --> 16:18.000 and what's running on VR is WXOD, 16:18.000 --> 16:27.000 which is a desktop solution that we at Colabora made. 16:27.000 --> 16:29.000 There is Colaboyant, 16:29.000 --> 16:33.000 and I'm splitting on it with a index controller. 16:33.000 --> 16:36.000 I think it's a high controller. 16:36.000 --> 16:40.000 But that is something you can do with it, 16:40.000 --> 16:54.000 even though... 17:06.000 --> 17:08.000 Even though we don't have a lot of time to maintain 17:08.000 --> 17:09.000 our desktop at the moment, 17:09.000 --> 17:12.000 there are alternative options, 17:12.000 --> 17:15.000 where VR is made by the community. 17:15.000 --> 17:18.000 Start-up, XOR is a lot more ambitious, 17:18.000 --> 17:22.000 it's aiming to create a whole comprehensive display server 17:22.000 --> 17:26.000 and support most of the open XOR application at the same time. 17:26.000 --> 17:29.000 So there's a lot of movement in this space. 17:29.000 --> 17:32.000 You can develop games, 17:32.000 --> 17:34.000 and applications with a lot of frameworks, 17:34.000 --> 17:36.000 go to open XORs, 17:36.000 --> 17:40.000 and the biggest engine here, 17:40.000 --> 17:46.000 the VR is on the lower engine, 17:46.000 --> 17:50.000 OSXGR is used in flight gear. 17:50.000 --> 17:54.000 The rear kit is another small framework, 17:54.000 --> 17:58.000 maybe all of those run on Monado, 17:58.000 --> 18:03.000 and are running on a complete open XORs pipeline. 18:04.000 --> 18:06.000 The web ecosystem, 18:06.000 --> 18:08.000 unfortunately, does not look as good. 18:08.000 --> 18:12.000 Firefox is right through here. 18:12.000 --> 18:16.000 It's just support some that VR. 18:16.000 --> 18:18.000 It's support is open VR only, 18:18.000 --> 18:20.000 and not open XORs, 18:20.000 --> 18:22.000 so it doesn't run out of the box on Monado. 18:22.000 --> 18:24.000 It only runs on Windows. 18:24.000 --> 18:27.000 The graphics are support is not complete, 18:27.000 --> 18:29.000 and they fired their XORs. 18:29.000 --> 18:31.000 They develop us a few years ago. 18:32.000 --> 18:35.000 So currently, they are not putting a lot of effort into this. 18:35.000 --> 18:37.000 Chrome, Chrome, 18:37.000 --> 18:39.000 looks kind of better. 18:39.000 --> 18:41.000 It has an open XOR implementation, 18:41.000 --> 18:44.000 but it appears not to be interested in Linux support, 18:44.000 --> 18:47.000 and I'm basing this on having waited ten years 18:47.000 --> 18:51.000 for them to support it on Linux and they have been so far. 18:51.000 --> 18:52.000 Thought who knows, 18:52.000 --> 18:55.000 with the release of Valve Steam Frame, 18:55.000 --> 18:56.000 they might change their tune, 18:56.000 --> 19:00.000 because Steam Frame ships with Steam OS. 19:01.000 --> 19:04.000 Currently, what's out there is WebKit. 19:04.000 --> 19:06.000 Egalya is working on it, 19:06.000 --> 19:10.000 and there is partially working WebExour support. 19:10.000 --> 19:13.000 I do have a demo for this. 19:31.000 --> 19:34.000 If you start WebKit, 19:34.000 --> 19:36.000 you get a browser here. 19:36.000 --> 19:39.000 This is a WebExour test page. 19:39.000 --> 19:44.000 This is the sponsor scene, 19:44.000 --> 19:46.000 that many people will know, 19:46.000 --> 19:48.000 that's a WebG application, 19:48.000 --> 19:50.000 and there is a button, enter VR. 19:50.000 --> 19:52.000 If you click it, 19:52.000 --> 19:54.000 you get permission for up, 19:54.000 --> 19:55.000 do you want to go into VR, 19:55.000 --> 19:57.000 and if you do, 19:58.000 --> 19:59.000 oh, 19:59.000 --> 20:01.000 outside of the model right now, 20:01.000 --> 20:03.000 but you can see it. 20:03.000 --> 20:05.000 This is working WebExour, 20:05.000 --> 20:06.000 it's tracking the head, 20:06.000 --> 20:10.000 and you will see this on your headset, 20:10.000 --> 20:13.000 if my HDMI cable is what the headset actually worked. 20:13.000 --> 20:15.000 And I see my timers up. 20:15.000 --> 20:18.000 Thank you.