WEBVTT 00:00.000 --> 00:13.960 So, hello. I am going to complain about where land and about input methods. Yeah, 00:13.960 --> 00:20.080 complain about wayland and this is kind of graphic room. Graphics room, but I'm not going 00:20.080 --> 00:25.320 to go to talk about graphics. Graphics is the output. I'm more interested in input. Input 00:25.640 --> 00:32.680 methods. Who of you has ever used an input method to hands up? How the crap? I was like 00:32.680 --> 00:42.680 trying to make a point. No more questions. Who is using it regularly? Okay, that's a bit better. 00:42.680 --> 00:50.280 I'm still impressed. Even better impressed with what I will sell later. Back then, we had x 00:50.280 --> 00:56.280 got replaced by wayland and with wayland came some changes. We still wanted to support input 00:56.280 --> 01:02.360 methods, but we had to use different mechanisms. Then, with x so, you had to build it up from scratch. 01:02.360 --> 01:06.840 And someone did it, like there was an institutional implementation. The initial implementation was 01:06.840 --> 01:14.680 not great. Well, it was not perfect. It was not perfect, but it had some problems, some imperfections, 01:14.760 --> 01:21.480 it had some missing features. I'm not going to go into details, but someone pointed out those 01:21.480 --> 01:28.920 issues. Hey, maybe we can do better. And the people who did that, some of them were, 01:29.960 --> 01:36.440 without the necessary experience, maybe incompetence. There's out of that, where three protocols, 01:36.440 --> 01:43.480 texting put V3, input method V2 and virtual keyboard. That didn't solve the problem. The 01:43.560 --> 01:48.360 protocols still had broken features, like with the edge cases that don't work. Some of the 01:48.360 --> 01:53.560 features were kind of bad, but you could work around them and some of nice things. They were just 01:53.560 --> 02:00.520 not there. And I come in and I asked, and let's could you please fund my work? Maybe I can make it 02:00.520 --> 02:06.760 better? And then let's say yes, and I'm here now. That was almost a year ago. So, I started fixing things. 02:06.760 --> 02:12.680 One of the things that I tried to fix was deleting text. The deleting text had a problem. In the 02:12.920 --> 02:18.600 protocols, if existing protocols, if it was even specified, it was deleting text by bytes. 02:19.560 --> 02:24.600 Before bytes, how many before the cursor to delete and how many after the cursor to delete. And this 02:24.600 --> 02:32.440 leads to problems. In the heavy path, it's time. In the heavy path, you have a text, the application 02:32.440 --> 02:38.920 tells the input method. This is the current text that I have. The input method is free to delete 02:39.000 --> 02:45.800 one by turn of text. What happens? What happens? Something gets deleted. So, we have one character 02:45.800 --> 02:51.880 less. That's good. But the way land is distributed protocol is asynchronous. Something can happen 02:51.880 --> 02:57.640 while something else is happening. So, when the input method gets a, get the notification of the 02:57.640 --> 03:02.680 text, maybe the text is already different for some other reason. Maybe it came out like a backspace, 03:02.680 --> 03:10.120 the backspace from the keyboard. The evil keyboard messes up our deletion because the update 03:10.120 --> 03:15.240 doesn't reach the input method in time. And the message tries to delete something. What happens 03:15.240 --> 03:23.240 here? Anyone can anyone see the problem? Hands up if you see the problem. Thank you. See, 03:23.240 --> 03:28.440 it's a UTF. It's not an ASCII character. It takes up two bytes. So, you're going to try to delete 03:28.440 --> 03:33.800 one byte. And this is where things blow up because if you try to delete one byte over two byte 03:33.800 --> 03:42.120 character, you end up in an invalid UTFH string and game over. So, this is like one corner case. 03:42.120 --> 03:49.000 I decided this cannot be. So, instead, let's, it's not void. So, let's just delete entire 03:49.000 --> 03:55.400 characters or something. So, that was that synchronization. It's kind of similar to the issue 03:55.400 --> 04:00.120 where things happen at the same time. You want to synchronize it. I tried to find solutions. I tried 04:00.120 --> 04:08.360 to do like retransmission stuff. It was so complicated. I had my Zen moment. The best code is code that 04:08.360 --> 04:14.360 is not written at all. And I decided to just say ignore it. No, no problem. It's going to be a 04:14.360 --> 04:20.360 problem of the user if something comes out wrong. And I think that might actually work. It took 04:20.360 --> 04:24.920 three attempts until I reached enlightenment. So, kind of that's pop up. That wasn't really 04:24.920 --> 04:30.120 amazing. The pop up couldn't be positioned with the previous implementations. So, I fixed it. 04:30.120 --> 04:41.400 It took another two attempts. Whoa, what did I do? Whoa. That went too far. Anyway, keyboard filtering. 04:41.400 --> 04:47.320 If you are typing your text in Chinese or Japanese or Korean, you don't want that text to end up 04:47.320 --> 04:52.680 in your input field immediately. You want the text to be held up by the input method. Like here, 04:53.560 --> 05:00.360 you type into the input method and it just presents the suggestion to the application. So, I had to 05:00.360 --> 05:06.360 it actually sounds a lot simpler than it is. It took three attempts with two complete implementations. 05:07.640 --> 05:13.480 And focusing. This is a screenshot from my phone. There is like a suggestion where some of the text 05:13.480 --> 05:19.160 is in so-called pre-edits mode. It's a candidate of what might be input. And in this case, 05:19.720 --> 05:25.320 what happens if I try to unfocus it? I try to like switch the next text field. So, the applications 05:25.320 --> 05:31.800 are, I'm going to just throw it because you didn't confirm it. This is not what Korean users expect. 05:31.800 --> 05:36.680 Korean users want to input their text. And what they have right now, if they change focus, 05:36.680 --> 05:41.880 the applications should keep it. It's all in under control of the applications. So, you need to tell 05:41.960 --> 05:49.000 the application how to deal with it. This also needs to be implemented as a wayland protocol. 05:49.000 --> 05:54.760 When on protocols is a repository and free desktop, which takes care of gathering things, 05:54.760 --> 06:00.840 where all the protocols, the input method protocols existing are. And this is where the fixes go. 06:00.920 --> 06:11.880 So, well and protocols has this very smart way of dealing with things. If the works, 06:12.600 --> 06:19.880 don't break it. So, modifications are kind of conservative. It has its downsides because if we want, 06:19.880 --> 06:26.760 if you know that the protocols are already broken, it takes a very long time to do something about it. 06:26.840 --> 06:31.480 This is a screenshot from yesterday. There wasn't times two, there isn't a time two, 06:31.480 --> 06:38.040 fixed the protocols. But it has not been merged despite being there for two years. And those are the 06:38.040 --> 06:44.920 things that everyone is supposed to be agreeing on. The conservativeness is maybe not so great. 06:45.800 --> 06:51.960 But, well and protocols, the maintainers, they agree that maybe sometimes they are a little slow. 06:51.960 --> 06:57.000 So, they opened a new experimental path. It is better in the way that it allows 06:57.000 --> 07:04.600 expectations, which you can have your protocol upstream two weeks after submitting, which is great. 07:04.600 --> 07:11.400 The protocol doesn't have to be super backwards compatible. But, if you don't do anything with 07:11.400 --> 07:18.280 the protocol after 13 months, we can just delete it. I managed to use this path to merge three 07:18.280 --> 07:28.680 protocols. So, it actually kind of works. But, how do I do it? So, I don't just ask to merge 07:28.680 --> 07:33.640 a draft. That would be kind of rude because, yeah, how do I, how do I prove that it is a good thing? 07:33.640 --> 07:41.400 So, first I draft, then I collect feedback. Then I try to implement a demo. And in the process, 07:41.400 --> 07:45.960 I usually find out that something will run with a draft. So, I dated draft and I tried to 07:46.280 --> 07:54.360 implement it, then I collect approval. And I asked to merge this protocol. And I lied a little 07:54.360 --> 08:01.320 bit here. One of those points doesn't happen, which one? Hands up for the number. 08:04.360 --> 08:12.280 Most of you guessed correctly. I said to say, but I haven't heard, I don't recall hearing one 08:12.520 --> 08:19.880 example of, yes, please merge it. This doesn't happen. So, I have my suspicion that people 08:19.880 --> 08:26.840 don't care that much about my work. And looking at this, I think people don't care about input 08:26.840 --> 08:33.880 methods in general. It takes two years to merge things that everyone is on board with. I don't know. 08:34.600 --> 08:42.200 Let's check it out. This is a timeline of the protocol. Initially in 2018, the fixing effort 08:42.200 --> 08:50.280 started. It was implemented in Weroots in GTK upstream. In 2024, that proposal that I just showed you appeared. 08:50.280 --> 08:58.920 2026, well, it took six years to fix a broken protocol. And it takes two years to implement the things. 09:01.320 --> 09:07.640 There's also the side of how are people implementing this in actual products. In Chromium, 2021, 09:08.280 --> 09:14.360 first Apache appeared. The Apache got merged in 2025, partially because everyone knew that it was 09:14.360 --> 09:21.080 broken and they were reluctant to merge it. So, maybe there's something missing. Maybe the 09:21.080 --> 09:25.480 right people are not working on it. The right person would have user experience. They would actually 09:25.480 --> 09:31.240 know what the user needs. And maybe they are going to need some skill to actually pull it off. 09:31.240 --> 09:35.320 Right away, I'm protocol, right implementations, and they also need interest. Maybe I've 09:35.400 --> 09:42.280 fixed them personally. Maybe they're getting paid by LLET. Or maybe they feel guilty about 09:42.280 --> 09:50.680 sucking it up in the first place because that that was me kind of. Not even enough. This all has 09:50.680 --> 09:55.480 to come at the same time. One person is not enough. One person who's like great is not going to 09:55.480 --> 10:01.240 pull it off if the protocol affects the entire ecosystem. So, you need multiple people at the same 10:01.240 --> 10:07.080 time who have all those skills to build a trust. This protocol is going to work. This is actually 10:07.080 --> 10:13.800 not going to break things worse. Those are all the necessary checks that need some political 10:13.800 --> 10:19.480 capital to pass. They are necessary. But it also means that it's hard to do something as a one 10:19.480 --> 10:25.640 person. I'm not seeing a lot of people that are working on this at the same time. I am seeing people 10:25.720 --> 10:32.200 which have user experience. They are using the input methods every day and have to skill 10:32.200 --> 10:37.880 but they are doing something else. I know people who have like who leaves like enthusiastic 10:37.880 --> 10:42.760 comments on the B-shoot tracker, but they don't like any sense at all. They have no skill. 10:42.760 --> 10:47.640 And there's me. I have the interest. I have the skill. I don't even use any of the methods. 10:47.640 --> 10:52.920 I like this is what you get people. I'm sorry. I'm the worst person to do it, but this is all you 10:53.000 --> 11:01.000 have right now. Yeah, and I'm making, makes me wonder, like where are all the people? This is 11:01.000 --> 11:06.360 affecting a billion people in China. I'm sure that some of them have all three qualities. Where 11:06.360 --> 11:11.800 are they? And my suspicion is maybe maybe we're developing this in a Western context. And this is 11:11.800 --> 11:17.080 like a barrier that we can't cross. Maybe they're different culture approaches do that. Maybe there 11:17.160 --> 11:22.120 is maybe they're all stuck in on the X11 that they are happy with it. I don't know. It's 11:22.120 --> 11:25.720 happening behind the great firewall because it's difficult to talk to those people. I don't know 11:25.720 --> 11:34.280 any Chinese. And maybe there's a language barrier. And what about Linux Mobile? There should be a 11:34.280 --> 11:38.120 bunch of people who have the skill. Maybe they just don't have interest. Maybe that just works. 11:38.120 --> 11:45.640 Well, I don't know. Please, please help me. Anyway, all right, the kid, the kid is me and the 11:45.720 --> 11:51.720 other kid that I presented earlier, they won about incompetence. And yeah, incompetence and 11:51.720 --> 11:56.840 lack of experience, that was also me. I fucked this up. Just full disclosure. I'm, 11:58.760 --> 12:03.880 I'm responsible for this. But like implementations, I see like a very similar thing with implementations. 12:04.600 --> 12:10.200 But first off, how does it look like? You need the input method, you need applications, and you need 12:10.200 --> 12:15.080 the composer to let them talk together. The basic I already told you, well, I'm protocols. 12:15.720 --> 12:21.320 Rust bindings. So back in 2019, when I implemented this first, I used to see, because that was 12:21.320 --> 12:29.400 the language for GTK and for dev downloads. And people were finding bugs half a year after I was finished. 12:30.040 --> 12:36.680 Then I decided, I am too stupid to write C. Now I wrote everything in Rust. So, full disclosure. 12:36.680 --> 12:43.720 I wrote an input method in Rust. I used the flow in library for application, like just a demo to 12:43.720 --> 12:48.600 test things with, and I used and will as the composer. So those are the layers. There's a lot 12:48.600 --> 12:54.280 of the quite a lot. What's in between the middle where there is a, okay, that's count. One, two, three, 12:54.280 --> 13:01.560 four, five, six projects that I had to contribute to to get anything working. That is kind of a lot. 13:01.560 --> 13:10.040 And yeah, how did it go? Under one year, I merged three experimental protocols. I made 53 changes. 13:10.120 --> 13:17.720 Some of that didn't go through. Some of them were 36 of them were upstream. I opened seven 13:17.720 --> 13:25.000 issues and I was across six projects. I think it's not bad, but it is not enough. It is not enough 13:25.000 --> 13:29.480 to let people try this out. It's not enough for you to build it without spending a lot of effort 13:29.480 --> 13:35.640 trying to build all this crap. And there is a problem. I mean, yes, those are the protocols, 13:36.200 --> 13:42.440 those are the efforts, but there's a chicken and egg problem. People say, yes, we are going to 13:42.440 --> 13:47.160 start implementing this. Once it is tested a bit more. Once we know that this is the right thing, 13:48.040 --> 13:53.560 but people are not going to be testing this in everyday life until it's implemented. So I don't 13:53.560 --> 14:01.080 know. I think we should as developers try to implement, please implement my crap. This is, 14:01.080 --> 14:06.840 this is me crying. Please implement this. Please implement this because it's a group effort. 14:06.840 --> 14:12.440 We need to implement this in order for this to take off. Those are experimental protocols. 14:12.440 --> 14:17.320 You can implement them alongside existing ones. So it doesn't necessarily break the existing stuff. 14:18.040 --> 14:22.280 We just have to do it. We have to give the user slags. We have to give the user a quick ability 14:22.280 --> 14:28.920 to test this out. I don't count on cute or gdk to implement it now, but like smaller libraries, 14:29.000 --> 14:33.640 maybe we can try and test it. I didn't have the time to make all implementations, 14:33.640 --> 14:37.240 but with some, I had better success with some, I had less success. 14:38.840 --> 14:44.520 Yeah, so I'm going to clean this up. Honestly, I'm tired of this and I need to hold it. 14:46.520 --> 14:53.080 Yeah, it's not really finished, but I need the rest of the community to step up and help me 14:53.080 --> 14:59.320 finish this up. I will be able to like, get feedback, but I'm not going to spend a huge amount 14:59.320 --> 15:07.000 of time coding anymore. And it doesn't catch up. You remember, well, I'm protocols had three months 15:07.000 --> 15:12.520 timeout for protocols. If there is no activity, well, there's just going to clean it up, 15:12.520 --> 15:18.840 and the effort is going to be wasted, which would be a shame. But thankfully, 15:19.560 --> 15:24.040 the analytics grant is not over yet. It goes until June. And I can include you in background. 15:24.040 --> 15:28.680 I can like kind of hire you or something. There is still a bunch of money for upstreaming. 15:28.680 --> 15:34.440 Like, I took care of most of drafting, but upstreaming, please come into me, speak to me. 15:34.440 --> 15:39.880 We can find something out. If you know Rust, if you will, maybe you're willing to learn Rust, I don't know. 15:39.880 --> 15:45.800 If you know Wayland, this could be upstream. It could be like, I could use some negotiation 15:45.800 --> 15:52.760 with other people. If you are a user, you could maybe try to be nice and not overly enthusiastic, 15:52.760 --> 15:58.120 and comment on the issue tracker. Yes, I would like it for this way or that way. I don't know. 15:59.880 --> 16:05.880 So yeah, that was basically the status before I try to push the project away and work 16:05.880 --> 16:09.080 on something else. Thank you. 16:22.840 --> 16:25.960 There's a question. So you spoke last year, right? 16:25.960 --> 16:28.440 I believe I did speak last year. 16:28.440 --> 16:30.600 Can you say that? That was it? 16:31.560 --> 16:41.880 I promise, I did say that this was the end. I promise this time it's not, it is actually the end, 16:41.880 --> 16:48.600 like soft end, but it is the end. So I don't want to be enthusiastic. Thank you very much for working with us. 16:48.600 --> 16:50.600 Thank you for thanking me. 16:50.920 --> 16:52.920 Thank you. 16:56.920 --> 17:02.120 Do you see a lot of improvements in feedback or upstreaming or whatever as well 17:02.120 --> 17:04.440 it gets more and more used for people? 17:04.440 --> 17:08.440 Is there improvement in feedback or upstreaming? 17:08.440 --> 17:15.160 I don't know. I think it would be pretty similar to 2019, but it would be people who were 17:15.160 --> 17:19.080 more enthusiastic back then because they didn't realize that I am very good at hacking things up. 17:21.160 --> 17:25.400 I'm not sure. Because like you say wayland is being used more now, 17:26.760 --> 17:32.760 I am not really observing that. Maybe that makes people more careful because they don't want 17:32.760 --> 17:37.880 a broken protocol to make people unhappy. They don't want to make a broken protocol 17:37.880 --> 17:44.200 make customers unhappy. This was the conversation I had today with the PopOS developer. 17:45.160 --> 17:49.000 The people, if we start breaking things, if you start adding experimental things, 17:49.000 --> 17:53.400 we are going to have more complaints for the customers and that is actually a negative thing 17:53.400 --> 17:56.360 for trying to make big changes. 18:03.160 --> 18:05.720 What kind of feedback would be useful to me from users? 18:06.840 --> 18:13.560 Ideally, some kind of user would come to me and ask me how do I build this to test it out? 18:13.560 --> 18:21.400 That would be pretty much the best feedback. The problem is, in order to work with this, 18:21.400 --> 18:26.600 you also need an input method. I try to build my own demo and the input method is not really 18:26.600 --> 18:32.040 usable. It's so slow because I'm not an expert in this on like graphical building a graphic 18:32.040 --> 18:40.120 application from scratch. The users can't really give me great feedback in terms of testing things 18:40.120 --> 18:44.360 and testing things would be the best feedback. I would need to get a first to get some authors 18:44.360 --> 18:51.320 of input methods on board. As a user, you come with the ability to say what you need. 18:51.880 --> 18:57.080 Maybe if you look at the protocol and if you're able to understand the purpose of the protocol, 18:57.080 --> 19:00.440 you can say yes, I want this protocol, yes, I want this feature. 19:00.440 --> 19:06.360 I think that would be not like a groundbreaking piece of feedback but it would send a message to people 19:07.320 --> 19:11.320 who are the main painters of the protocols that there is an interest and maybe some effort 19:11.320 --> 19:15.160 would be put into it. I don't know if it would be working actually. 19:19.160 --> 19:22.760 Wasn't the main goal of the unstable protocol to have some way 19:23.720 --> 19:29.240 and that because in the past it was implemented by the toolkit directly, like GTK, 19:29.480 --> 19:36.840 as it's on custom protocol to do the IMF to IME code. Same thing for Q. 19:38.280 --> 19:45.640 I think, yeah, I mean, you can kind of on the architecture level level do it either way. I don't 19:45.640 --> 19:50.600 think the toolkits do not want to do it on their own anymore. 19:51.400 --> 19:57.800 So the idea has shifted from the toolkit does it to the toolkit talks to the compositor and the 19:57.880 --> 20:01.080 compositor does it. That's my understanding. 20:01.080 --> 20:08.120 Yeah, it's why there's not much adoption but most of the way in compositor already use a specific toolkit. 20:08.920 --> 20:13.400 So there's no change for the user. So you're saying maybe that's the reason there's no adoption 20:13.400 --> 20:18.920 that the toolkit are already doing this and I would say I don't think so because I have seen 20:18.920 --> 20:25.000 the great adoption of the broken protocol already. Like pretty much everything moved to the broken 20:25.080 --> 20:30.120 protocol. That means they are not interested in doing the toolkits are not interested in doing 20:30.120 --> 20:35.400 their own thing anymore. So fixing this is actually important because people are using this. They 20:35.400 --> 20:38.440 want to use it. It's just not great. 20:43.640 --> 20:49.160 I made a question here so I kind of got lost there but was there, was there any point to doing 20:49.160 --> 20:57.800 the 3.2 version or was that also technically broken? Is there any point in doing 3.2 version 20:58.920 --> 21:03.800 or is it still broken? No, I don't think that 3.2 is like fundamentally broken. The effort is 21:03.800 --> 21:08.840 right. I gave a little bit of feedback and the feedback seems to be like just hanging there. 21:09.560 --> 21:13.640 It seems that there is also suffering from the same problem that there's not enough interest to 21:13.720 --> 21:15.640 actually fix it and put it through. 21:31.000 --> 21:38.760 So the question is about the key repeats on the client. I did include a provision for like 21:38.760 --> 21:42.920 repeating keys in the keyboard filtering protocol but otherwise it's not really 21:43.720 --> 21:54.600 part of what an input method does. It would be an issue with an issue with the toolkits. I don't know. 21:54.600 --> 21:59.560 I would have to look at it. I never dealt with it. 22:00.520 --> 22:07.240 Yeah. You mentioned when it's mobile? Yes. When I think of it, it would method or I mean 22:07.960 --> 22:12.600 I see like the policy show for example. Yes. How does that relate to the keyboard on the phone? 22:12.600 --> 22:20.200 This is a great question because I forgot. Linux mobile, Linux mobile, how does it relate to input methods? 22:20.200 --> 22:26.680 In Linux mobile or an Android or iOS, you have a panel which serves as a kind of thing that 22:26.680 --> 22:30.920 includes text. It's not a keyboard because the phone doesn't have a keyboard and it is not 22:30.920 --> 22:36.840 forwarding kernels. The kernel events to the composer. It has to be generated in a special way. 22:37.400 --> 22:43.720 So like the input panel for the keyboard where the keys change like the submit keys sometimes it's 22:43.720 --> 22:48.520 magnifying glass sometimes it's something else and it doesn't really do the same thing 22:48.520 --> 22:53.240 all the time. It changes layers dynamically. This is something that falls into the input method 22:53.320 --> 23:03.720 concept that we are employing here in a way laden. Or like swiping keyboards. T9 or T9. 23:04.520 --> 23:06.920 One of the demos that I might in my input method is T9. 23:16.920 --> 23:20.840 Looks like I exhausted you. Sorry. 23:21.800 --> 23:24.520 Do I want a user story? Of course I do. 23:37.800 --> 23:42.520 I think matter would be in the way itself. I don't think this adds thing. 23:44.360 --> 23:48.040 You need a protocol for this but I don't know what those things use for all the time. 23:48.120 --> 23:50.040 They could use an input method. They could use like pen. 23:51.400 --> 23:58.200 Right. So I think there would be one more way to do it but it wouldn't get in the way. 24:02.040 --> 24:03.240 Thank you and of time.