WEBVTT 00:00.000 --> 00:14.240 Yeah, alright. So, welcome everybody. Right. Let's get down to it. Thank you all for letting 00:14.240 --> 00:22.640 me open this deaf room for giving me the chance to present this interesting topic. I just 00:22.640 --> 00:27.920 realized that this year is going to be my 20th anniversary of using Linux. I'm full-time 00:27.920 --> 00:35.440 kernel developer since 2020. I've joined Kalabra in 25. And I mostly hang around in Linux, 00:35.440 --> 00:43.760 media, and Linux, Rockship, and in that capacity I am a charge of bringing the video capture 00:43.760 --> 00:50.080 and camera support mainline. So, this is the goal. Right. And this is for recent Rockship 00:50.080 --> 00:59.440 SOCs. So, by that I mean something RK 35, something something. And if it comes to camera and video 00:59.440 --> 01:06.080 capture we can group those into two roughly because they somewhat share the same architecture. And that's 01:06.080 --> 01:14.320 the 66 and the 68. And then there's the 88 and the 76 and the 62 and whatnot in the other group. 01:14.320 --> 01:22.640 So, just as a bit of an info there. Now, we do capture on camera. First of all, it's amazing. Right. 01:23.200 --> 01:30.240 Then it's pretty much about capturing video frames from some companion chip to memory. 01:32.480 --> 01:37.120 However, a camera does require more. So, the companionship could be some 01:38.000 --> 01:44.160 HDMI to mepices, I convert it, but also it could be an image sensor of course, and it delivers 01:44.160 --> 01:51.120 raw frames in the sense that you need to do debaireing and then it's hard to green. So, you need to 01:51.120 --> 01:58.480 do corrections and all the valve balance and all those things. At some point you'll need to 01:58.480 --> 02:06.480 eliminate it. You may have some optics and yeah please like this. So, I won't be covering this too 02:06.480 --> 02:10.880 much because they're interesting talks tomorrow, so make sure not to miss those. 02:14.160 --> 02:19.520 I don't feel it's required to say why we want to use upstream here or why we want to go main line. 02:20.880 --> 02:24.400 Rockship SOCs are already well supported in main line Linux. 02:25.040 --> 02:36.560 Yeah, woo for that one. So, my colleague Nikola will be giving a kind of an overview of what goes 02:36.560 --> 02:44.560 and what happened last year, also tomorrow. So, if you're into Rockship SOCs in general, make sure 02:44.560 --> 02:51.920 to drop by. Yeah, and we do capture on camera is a bit of a gap here and the game here is to close 02:51.920 --> 02:59.760 this gap right. I've given a similar talk recently in Amsterdam last August, so we'll be covering 02:59.760 --> 03:06.160 what changed since then and so that the goal here is just to give you a bit of an update. Still, 03:06.160 --> 03:14.480 first we're going to look at the hardware roughly and then give the overview of the current state of 03:14.480 --> 03:23.520 the art and then let's see what comes next. So, I said, well, okay, there's roughly two groups and 03:23.520 --> 03:32.000 the first group here, there's the video capture you didn't and the iSpeed image signal processor 03:32.000 --> 03:40.400 and they work mostly parallel, right? So, they share the the phi, this phi is special because 03:40.480 --> 03:46.880 you can share it between the two or use it for one of them. So, this is configurable. 03:48.080 --> 03:54.960 And then, of course, you've got a ton of DMA engines and converters and ESpeed iSpeed cores just 03:54.960 --> 04:02.240 well, all the processing blocks and lots of things in it. So, again, the spit mode is something 04:02.240 --> 04:07.920 I'm going to mention quite often because this is well, something we need to take care of 04:07.920 --> 04:16.720 soonish because it blocks things and yeah, okay, the iSpees have different versions I'm not going to 04:16.720 --> 04:25.120 into. Keep in mind that the iSpeed has different operating modes so you could use them in 04:25.120 --> 04:32.320 in line where you take the image from the sensor and pass it to the iSpeed or you can buffer it into 04:32.320 --> 04:39.600 memory and then use it like in memory to memory operating mode and yeah, there are use cases with 04:39.600 --> 04:47.760 raw capture and HDR and but yeah, this is a bit beyond this code right now. Of course, there is a 04:47.760 --> 04:56.640 receiver that like me PCSI is used for camera sensors often and it features some virtual channels 04:56.720 --> 05:03.040 so you can have different streams over the same CSI link. So, this is also some interesting bit. 05:05.520 --> 05:12.000 The other group features an architecture like this where there is first the recap that is the 05:12.000 --> 05:20.240 interface and then the iSpees are sub well at the next stage of it and you can do raw capture 05:20.240 --> 05:28.000 with the recap itself and it features a huge crossbar max and then you can select certain streams 05:28.000 --> 05:34.240 to go to the iSpeed and some to memory and so well whole lot of possibilities. 05:35.840 --> 05:42.480 Again, we have different numbers of different files and different iSpees over all those variants 05:42.480 --> 05:49.760 the iP cores are roughly mostly the same but not exactly so sometimes it's easy to port 05:49.840 --> 05:58.320 them from one SOC to the other and sometimes it's more difficult so there's a bit of a variation there. 06:00.720 --> 06:07.280 So the recap again this max 2 iSpeed block this is something we'll need to take care of 06:07.280 --> 06:14.480 because there's enables the usage of the iSpeed in line mode and this should be one of the 06:14.480 --> 06:23.120 most frequent use cases i'd say. Right so in comparison to Amsterdam in August 06:25.440 --> 06:33.040 this CSI defy was merged after a bit of well some i think it went up to reform or something 06:36.080 --> 06:42.880 this split mode is something i mentioned and so this is new because i didn't realize that we 06:42.960 --> 06:50.240 really need to take care of it it has some implications on how things are exposed to user space 06:50.240 --> 06:58.880 how they're exposed to the device tree so how do the bindings look like and things so this 06:59.760 --> 07:06.560 is what something we should be looking at but there is initial work going on i heard from 07:07.520 --> 07:15.680 i user that he was at least able to to bring up say the other pair of lanes and so 07:16.560 --> 07:24.960 yeah things will look good not just say not promising anything but i indicate that. 07:26.720 --> 07:35.440 Right um yeah the dc5 is on the more modern rpc bases this needs some loving care 07:36.240 --> 07:44.080 the receiver was renamed a few times but now finally accepted woohoo and with that we can go forward 07:44.080 --> 07:54.160 and this should be easy to port on say if you have 75-76 or 62 machine so this should be easy 07:54.160 --> 08:02.880 i hope um then the the mp and the dvp capture so the dvp means digital video port it's just a 08:02.880 --> 08:13.760 parallel interface so this went mainline recently so it's just the wall initial driver comes with 08:13.760 --> 08:23.440 all that and was a bit of a relief that this was finally accepted yeah and basing on this there are 08:23.520 --> 08:34.720 first efforts towards mp capture on the 3588 and also on say the towards this max to i's bbit 08:35.920 --> 08:43.600 and well i can show you a picture of the mp capture thing so this is just a rock 5b plus a 08:43.600 --> 08:51.440 raxa camera and it's just capturing to memory and then using g streamer by air to rgb 08:51.520 --> 08:59.120 to convert to do the demosic and it's awfully green and noisy and whatnot but well you can 08:59.120 --> 09:13.920 recognize me so first success you can right well anyway so okay um however well we kind of stop here right 09:13.920 --> 09:24.240 so um yeah and here's nothing really easy so um i did start so i was here two years ago and promised 09:24.240 --> 09:33.360 that i'll be mainlining patches for the isp on the 66 and 68 i never managed to do so so sorry about that at this 09:33.360 --> 09:43.600 point sorry say laurel sorry for me anyway right now there is focus on the 3588 because 09:43.680 --> 09:50.480 this is like the flagship right and so um and the good news is that there are different 09:50.480 --> 09:57.680 parties working on the isp support so there's us there is ideas on board of course and there is also 09:57.680 --> 10:02.880 rock chip doing some bits and i hope they really so this really takes off and we've got the 10:03.840 --> 10:11.200 contributing here so in terms when it comes to SOC camera SOC vendors rock chip 10:11.200 --> 10:18.880 is somewhat one of the best right so it's still not good but there's somewhat the best so just the 10:18.880 --> 10:32.800 others are way more worse anyway it's just about this closing having their openness right 10:33.120 --> 10:40.560 so rock chip is some some company you can actually approach and you get the information at least 10:40.560 --> 10:48.240 the register contents to some degree it's not easy but it's at least possible right so 10:50.080 --> 11:01.600 and yeah so i'm really looking forward how this develops so right so isp support is somewhat 11:01.680 --> 11:07.040 in the making but it will be some churnian till we get there and then of course we'll need 11:07.040 --> 11:10.880 flip camera support right and looking at you guys see 11:15.280 --> 11:22.560 hi in which SOC rock chip is contributing there's a way like so there's an initial 11:23.520 --> 11:35.680 driver for the RP 3588 isp from them but it's just basically throwing something over the fence 11:35.680 --> 11:44.240 and then let's see what comes around but we'll try to get them i think most work will be done 11:44.240 --> 11:52.080 on the 3588 and then we can see what it of course we'll try to make sure that the 76 11:52.160 --> 11:57.680 in the 62 can be supported at some point the ice piece different but the architecture is the same 11:57.680 --> 12:08.800 overall so i think we can share some code there and yeah right so what's next 12:11.040 --> 12:15.760 yeah there are certain things to discuss mostly about how to expose hardware using 12:16.480 --> 12:26.080 video for Linux to the user space i mentioned the split mode quite a bit it's about okay 12:26.080 --> 12:31.040 how does the media graph look like should there be two of them or one or whatever 12:31.920 --> 12:37.280 it's about how to expose memory to memory mode and inline mode can use which between the two 12:38.160 --> 12:46.640 in the 88 there is the chance of two ISB's working in parallel in unison and how can this be 12:46.640 --> 12:56.960 activated if need be um we definitely looks already offers the streams API which is a good match for 12:56.960 --> 13:03.840 the mep virtual channels i mentioned and then there is multi context for the time division 13:03.840 --> 13:10.160 multiplex because you can have like up to six or seven sensors on the 88 but you have only 13:10.160 --> 13:16.960 have two ISP so you need to do some multiplexing here and this leads then to the question again 13:16.960 --> 13:22.080 together with memory to memory operation do we need a scheduler here and so there are certain 13:22.080 --> 13:29.680 things to discuss and well this may take a while for now i'm going to continue my work 13:30.640 --> 13:41.600 regarding the 88 weakup in particular the max unit and then yeah a preliminary 13:42.480 --> 13:50.240 initial ISB driver that does the inline processing right so still a green picture but at least 13:50.240 --> 13:57.920 debaired this will be the first goal here what you can do is just test all this as far as it 13:57.920 --> 14:06.560 possible and well with different setups different use cases feel free to reach out and just in case 14:07.840 --> 14:16.800 well you want to have an ISP there is a software ISB lip camera now lip camera 07 0 right 14:16.800 --> 14:25.200 it's called Brussels for a reason and right and there will be an interesting talk about that one too 14:25.200 --> 14:31.920 so make sure not to miss this if you're looking for a new challenge we are hiring and 14:31.920 --> 14:36.880 for the time being let me thank you all for listening and i'm open for your questions now 14:37.760 --> 14:47.280 ah sure yet go ahead 14:59.040 --> 15:04.720 yes um so it's the question is about capturing HDMI 15:07.840 --> 15:21.200 yeah um okay about capturing HDMI audio as far as i'm or as far as i know it's 15:22.080 --> 15:29.280 something there's a companionship and in the this driver you will register some also device 15:29.920 --> 15:37.040 as a sock something so this is at least the capture i see's i know 15:39.920 --> 15:48.240 if this but let's have a chat afterwards so uh you mentioned that you are the fairing 15:48.240 --> 15:53.440 basically support for statistics and parameters which usually is not the hardest part of supporting 15:53.440 --> 15:59.440 now it's been because that's just basically about exposure actually i was going that for 15:59.440 --> 16:03.440 question of time you don't have time for you missing the presentation for that 16:03.440 --> 16:08.880 okay the question was about kind of short parameters statistics is it a time issue or a 16:08.880 --> 16:16.640 documentation issue um pretty sure the documentation is there in the sense of code is documentation 16:16.720 --> 16:25.440 we've got downstream uh the biggest downstream BSP and so we'll we'll extract it from there i'd say 16:27.120 --> 16:32.480 i reckon i'm not too experienced when it really comes to parameters and stats i see this from an 16:32.480 --> 16:39.680 abstract point of view and my job is to get a stream running and this is why i'd just accept that 16:40.640 --> 16:48.080 debaring yes and then just some stream and the the what say to make it nice is the next step 16:48.080 --> 16:54.240 done there was some what yeah for a question we're going to be uh downstream and rocks is 16:54.240 --> 17:00.400 what so how much I guess in production of these accounts uh it was better if it was 17:00.400 --> 17:05.680 how much I get it from the downstream is it's so how how how 17:06.000 --> 17:12.720 how feature which is the downstream channel code so it was basically so that you can enable and 17:14.080 --> 17:20.720 can most of all the ice cream okay so the question was about the downstream BSP in terms of features right 17:21.760 --> 17:30.880 or or uh you can get more from the raw chip themselves as adults uh so the downstream BSP i must have 17:30.880 --> 17:39.600 met i tend to not look at it so i i hear yeah of course they implemented it on their basis so 17:39.600 --> 17:47.040 it's pretty much every feature should be there right but i only use it just a reference and so yeah 17:48.080 --> 17:54.000 what is the step to sort of a fixed 30 in main line for the video in the end of the video code 17:54.160 --> 18:07.120 so the the question was the px 30 uh and how the status is uh i during this work i brought the 18:07.120 --> 18:13.040 video input process of for the px 30 main line but i never really had this board on my desk or this 18:13.040 --> 18:20.400 chip on my desk and it was just a continuation work so i wouldn't know but there is the rock 18:20.400 --> 18:28.400 chip maintainer here somewhere i think two rows behind you and uh 18:28.640 --> 18:33.680 all of them are not high at least just like what you put three nine nine and if you put it 18:33.680 --> 18:43.200 or like like you can just put the camera on the whole thing and you will get the real 18:43.200 --> 18:49.040 nice and much other thank you so just for the record uh after the lecture way 18:50.560 --> 18:57.600 for the recording so high co claims that is covered well by the RKI SP1 driver and Kira 18:57.600 --> 19:02.480 mentions it should be feasible with a bit of tuning uh filling around don't expect 19:02.480 --> 19:08.240 perfect images out of box but you can definitely get there okay not perfect but some images 19:08.240 --> 19:18.720 and this is a good start at say right okay and thanks again y'all for listening