WEBVTT 00:00.000 --> 00:11.800 Our next speaker, just admitted to me, he is not a go developer. 00:11.800 --> 00:14.600 Wow, exciting. 00:14.600 --> 00:17.680 People like that, you know, the go developer for some reason? 00:17.680 --> 00:20.720 So why did I invite Dimash for you today? 00:20.720 --> 00:25.800 Well, you don't build Go, oh well, we don't write Go, but you build it. 00:25.800 --> 00:30.280 And this is most interesting, because those go binaries, they get them everywhere. 00:30.280 --> 00:35.160 And we want to know how this is called, we've written very badly, and it's a positive 00:35.160 --> 00:36.160 compiler. 00:36.160 --> 00:37.160 A positive entry. 00:37.160 --> 00:46.080 And yeah, hello, welcome. 00:46.080 --> 00:47.080 My name is Dimitri. 00:47.080 --> 00:49.680 I am an OS distribution developer. 00:49.680 --> 00:52.360 So I've built lots of operating systems. 00:52.360 --> 00:54.000 I am the Indian developer. 00:54.000 --> 00:55.880 I'm a bunch of core developer. 00:55.880 --> 01:02.280 I started Intel Clear Linux project currently working at Chingar Building WulfioS. 01:02.280 --> 01:07.520 And I'm mostly focused on building production grade container images, such that all the 01:07.520 --> 01:13.560 go code that I compile over a thousand projects, we deploy it in Kubernetes clusters in 01:13.560 --> 01:17.240 production for ourselves and our customers. 01:17.240 --> 01:23.480 And this talk I'm going to dwell into how we build all of the go projects that we ship. 01:23.480 --> 01:28.480 And hopefully, such that other people adopt our best practices. 01:28.480 --> 01:35.200 It will be Linux focused, right, such that majority of this applies to Linux binaries on 01:35.200 --> 01:37.240 x86 and ARM. 01:37.240 --> 01:43.080 However, most of it is transferable to go on other operating systems and architectures as 01:43.080 --> 01:44.080 well. 01:44.080 --> 01:47.200 But you may need to do your own testing. 01:47.200 --> 01:55.080 Yeah, I've been doing this for a while, so hopefully you'll find something useful here. 01:55.080 --> 01:57.440 First we'll start with simple things. 01:57.440 --> 02:04.480 So by default, when you run go build, it will produce binaries with debug symbol attached. 02:04.480 --> 02:12.600 I do want to ask you a question, who has here run GTB or DELF on a go binary to debug 02:12.600 --> 02:13.600 it? 02:14.520 --> 02:15.640 OK, that's a lot. 02:15.640 --> 02:21.160 Keep your hand up if you've done it, well, the binary you're attaching to is running 02:21.160 --> 02:22.480 in a production cluster. 02:24.480 --> 02:26.160 OK, still a few hands. 02:26.160 --> 02:32.520 OK, I want to posit that the debug information is often unused in production, right? 02:32.520 --> 02:36.160 Such that when you build your production container, you can strip that. 02:36.160 --> 02:42.560 So there is LD flags, dash W flag, which will remove debug symbols from the binary 02:42.600 --> 02:44.040 you produce. 02:44.040 --> 02:48.600 And if you want to keep a copy of your debug information, there is another option where 02:48.600 --> 02:53.920 you can use strip and you can strip debug and store it in your debug info server, which 02:53.920 --> 02:55.520 you are running locally. 02:55.520 --> 03:00.160 Because then GTB can still connect, and you can still get all of your debug information 03:00.160 --> 03:05.880 for the five people who connect to their production binaries, right? 03:05.880 --> 03:08.560 They're saving on the binary stage. 03:08.600 --> 03:14.240 It can be like 35, 40% of the binary size, such that you can potentially, you know, 03:14.240 --> 03:18.000 half of all of your container sizes. 03:18.000 --> 03:20.560 Next up, trim path. 03:20.560 --> 03:27.800 So when you build binary, it encodes the file path of your build system inside the 03:27.800 --> 03:31.120 binaries for majority of files that you're compiling. 03:31.120 --> 03:36.600 It leaks the username, it leaks the file system layout of your build infrastructure into 03:36.600 --> 03:37.680 the binary. 03:37.680 --> 03:42.960 And if you strip that, you can save between like 2% and 5% of your binary size, and 03:42.960 --> 03:48.400 you can also make your binary script reproducible and also be able to compile it on multiple 03:48.400 --> 03:51.160 places and get the same hash every time. 03:51.160 --> 03:52.880 It's a useful property. 03:52.880 --> 04:00.480 It removes any extended attributes that you've set with LD flags from the build info file. 04:00.480 --> 04:03.880 So there is an upstream bug about it. 04:03.920 --> 04:07.800 They're hopefully going to fix it in the compiler, and we'll see if we do patch that out 04:07.800 --> 04:13.840 because we want to still see all of the x flags that you pass to the binary and build info. 04:13.840 --> 04:15.240 Okay. 04:15.240 --> 04:20.480 How many people here disable Cgo when they build binaries? 04:20.480 --> 04:24.040 Why? 04:24.040 --> 04:28.000 So, static linking, that's very common ounces. 04:28.000 --> 04:31.800 So back in the day, you have to pass it to get a static binary. 04:31.800 --> 04:34.200 That's no longer the case. 04:34.200 --> 04:39.840 By default, on Linux, go use this, a Glypsy, and name switch service to do a resolution 04:39.840 --> 04:42.360 of DNS and the user names. 04:42.360 --> 04:48.960 However, there is build tags, netgo, and OS usergo, which you can pass, and you can keep 04:48.960 --> 04:53.120 Cgo and a vote, and you can still get static binary. 04:53.120 --> 04:57.000 And then in your build info, it will also save those tags. 04:57.000 --> 05:01.560 And then your build becomes the same whether you're cross compiling or natively compiling 05:01.560 --> 05:06.320 the binary, which helps all of the developers who use like a MacBook to deploy to Linux 05:06.320 --> 05:08.320 containers. 05:08.320 --> 05:14.520 And you get compatibility with all of the FIPs toolchains and forks of official go-tool-chay, 05:14.520 --> 05:19.840 such that you can compile the same project with boring crypto, go-open SSL, go-mikers 05:19.840 --> 05:20.840 of FIPs. 05:20.840 --> 05:25.800 If somebody ever tells you that you need FIPs compliance for your binary, because all 05:25.800 --> 05:30.760 of those forks, they require Cgo to be an able to implement TLS and cryptography. 05:30.760 --> 05:36.520 So I want to keep your projects behaving the same, still have static linking for the regular 05:36.520 --> 05:46.200 go-tool-chay, but be able to use Cgo for accelerated code, optimized code, if you need it. 05:46.200 --> 05:47.960 And there are two options for that. 05:47.960 --> 05:48.960 There is a caveat. 05:48.960 --> 05:55.080 If you do need name switch service resolution, then you sort of keep Cgo enabled, and 05:55.080 --> 05:59.960 there is no mirror tags for that to opt into that behavior. 05:59.960 --> 06:06.160 Who has production hardware, which is older than 20 years? 06:06.160 --> 06:07.160 Excellent. 06:07.160 --> 06:13.000 This slide is not for you three people. 06:13.000 --> 06:17.320 You should set environmental variables to the highest API level for your production 06:17.320 --> 06:22.600 great hardware, such that most people V2 is, if your hardware is newer than 20 years 06:22.600 --> 06:27.640 old, if your hardware is like newer than 10 years, or you use latest instance types, 06:27.640 --> 06:29.280 you can go to V3. 06:29.280 --> 06:32.600 If you use AI ML only, you can go to V4. 06:32.600 --> 06:37.040 Your binaries will run faster, and they'll have less code in them. 06:37.040 --> 06:38.600 That's as simple as that. 06:38.600 --> 06:44.280 And most Linux distributions, they'll rate you race the C, C++, minimum, and the I2, V2, 06:44.280 --> 06:45.280 or V3. 06:45.280 --> 06:47.800 However, they haven't done it for Go binaries. 06:47.800 --> 06:53.520 So Go is starting to be slower than C, and I don't like that. 06:53.560 --> 06:58.520 So export those environmental variables, and you'll get better performance out of your 06:58.520 --> 06:59.520 binaries. 06:59.520 --> 07:00.520 To do. 07:00.520 --> 07:05.760 Next up, position independent executables encode. 07:05.760 --> 07:12.960 So most Linux distributions in their toolchains, they enable hardening flags to make sure 07:12.960 --> 07:19.920 that all binaries are linked with bind now, and position independent executables encode. 07:19.920 --> 07:24.960 And they do it consistently for Rust, C, C++. 07:24.960 --> 07:28.800 Not many people do it for Go, but they should. 07:28.800 --> 07:35.240 For static binaries, it might not gain you much security, but it will not set off your security 07:35.240 --> 07:41.040 scanners and your audits, because when security companies come into the audit of your deployments, 07:41.040 --> 07:45.000 they will look for that, because lots of security scanners look at your whole container, 07:45.000 --> 07:49.880 and they're like, okay, everything has position independent executables apart from the 07:49.880 --> 07:54.160 five things that you are your entry point and your go binaries. 07:54.160 --> 07:56.360 So you may want to enable it. 07:56.360 --> 08:02.360 It does gain you security features if your go binary calls into C code, or if it has any 08:02.360 --> 08:06.920 C go enabled, or if it has any hard work acceleration instructions. 08:06.920 --> 08:14.080 You may not know this, but some of your dependencies may have assembly inside the Go binaries, 08:14.080 --> 08:19.160 which can then be secured by the compiler. 08:19.160 --> 08:22.640 It's an each thing. 08:22.640 --> 08:30.360 Version numbers, who has difficulty tracking what binary is deployed in production, or 08:30.360 --> 08:35.000 know what its version number is, a few people. 08:35.000 --> 08:44.920 So when you run Go build, it doesn't bet VCS information into the binary, but it does not. 08:44.920 --> 08:52.560 It does, but it encodes only the commit, shaw, and times them, but it doesn't include 08:52.560 --> 08:53.880 the tag name. 08:53.880 --> 08:59.600 If you run Go install, which you can for private builds with like Go private exported, 08:59.600 --> 09:06.520 it will encodes the tag name, but not the commit, or the timestamp. 09:06.520 --> 09:11.840 For that, I recommend to set a custom argument to your build environment where you get 09:11.880 --> 09:17.000 described your tag, and then it will end up in the build information of your binary, because 09:17.000 --> 09:21.240 then it doesn't matter if you go build or go install, you will have the same field and 09:21.240 --> 09:26.400 build info, which you can then statically analyze and discover in all of your production 09:26.400 --> 09:28.400 clusters. 09:28.400 --> 09:33.240 C go hardening. 09:33.240 --> 09:39.080 Who has set C go under course C flags in their life? 09:39.080 --> 09:44.520 One, two, ooh, seven people, okay, that's good. 09:44.520 --> 09:51.520 When you build C++ binary using like RPM build with RPM spec files, or in David, if 09:51.520 --> 09:57.880 you use S build, normally they have lots of tooling, which will export really long C flags, 09:57.880 --> 10:02.880 and LD flags and things like that, or they'll generate spec files, and all of that works 10:02.880 --> 10:11.160 with like Mason and whatnot, but Go tool chain doesn't honor C flags environmental variable, 10:11.160 --> 10:17.320 it has its own environmental variable code C go C flags, and none of the tooling out there 10:17.320 --> 10:22.800 currently sets that by default, such that even if you package your software properly and 10:22.800 --> 10:29.440 use a given OS vendors hardening flags, they will actually not be enforced for any C code, 10:29.440 --> 10:34.880 which happens to be called by Go, and at that point, you need to reset those environmental 10:34.880 --> 10:40.960 variables again, so if you detect in your binaries, that C goes enabled, then you need those 10:40.960 --> 10:45.400 hardening options to be on, because otherwise your binary will be less secure and less 10:45.400 --> 10:51.320 optimized as well, and for people who are interested in that instead of sending environmental 10:51.320 --> 10:57.520 variables, I normally do wrappers around my GCC such that I can specify spec file elsewhere, 10:57.520 --> 11:02.720 and I know it will propagate throughout my build systems, irrespective of what you use to compile 11:02.720 --> 11:08.880 your code, because they all call GCC, and then I redirect and say, please set all of these things 11:08.880 --> 11:18.960 for me, which are recommended by OpenSSF, so who has to deal with vulnerabilities in your 11:19.040 --> 11:29.840 dependencies in the codes that you deploy? Who wants to not deal with that? 11:31.840 --> 11:38.000 So, this is probably the best slide that I can do, so whenever I get the hello golfer 11:38.080 --> 11:48.880 CML, I just panic, because usually it means that some big dependency library has vulnerability, 11:48.880 --> 11:54.800 normally if you get an email two days earlier saying we will announce this next week, then you know 11:54.800 --> 11:59.760 it's going to be like higher critical, and that there is a lot of reverse dependencies, right? 11:59.760 --> 12:05.200 And at that point, you realize that suddenly, next week, you're stopping development, 12:05.200 --> 12:10.080 you're going to upgrade your dependencies, rebuild everything and redeploy, right, because that's 12:10.080 --> 12:16.720 what you need to prepare for. And there is a tool called GoVonCheck that you can run yourself, 12:16.720 --> 12:22.720 and it is able to detect all of the CVs and all of your dependencies if you point a binary at it. 12:22.720 --> 12:31.040 However, normally if you strip symbols out of your binaries, it will look at module level CVs only. 12:32.000 --> 12:39.600 And that's not good enough, because despite you importing accident or despite you importing 12:39.600 --> 12:45.120 accident HTML or something like that, you might not be using every single symbol, every single 12:45.120 --> 12:51.040 functionality inside your binary, or inside your dependencies which you've pulled in, right? 12:51.040 --> 12:56.080 And when vulnerabilities are announced, they're usually for a specific function call or a specific 12:56.160 --> 13:02.000 feature. And all of the vulnerabilities canters without symbol level information, they will say 13:02.000 --> 13:08.080 everything is vulnerable in your cluster container everywhere. However, if you keep symbols 13:08.080 --> 13:13.760 information inside your binary, the vulnerabilities canters will be able to go into tech that, 13:13.760 --> 13:20.320 yes, you're using vulnerable XNet. Now, you're not using the symbol that has security, 13:20.320 --> 13:25.840 feature a security vulnerability inside of it, and we can skip patching and you don't need to 13:25.840 --> 13:31.360 redeploy anything. So, for example, the previous email, which announced XNet vulnerability, 13:31.360 --> 13:38.160 in WolfOS, we found that there was over a thousand binaries which imported that module, that package, 13:38.160 --> 13:44.960 and then we found that like 200 of them had the symbols that used the HTML subcomponent 13:45.440 --> 13:53.760 thing of XNet, and then because we keep symbol tables in all of our binaries, despite the 13:53.760 --> 13:59.920 binary size, we were able to eliminate majority of them and close all of those city vulnerabilities 13:59.920 --> 14:05.200 as code not present, because only 13 of them were actually using the vulnerable functionality. 14:06.080 --> 14:14.800 And I can show you an example of this. So, like, if you go install Core DNS, 111 tag, 14:15.360 --> 14:21.040 and if you run the go vulnerability checker on it, it will find 15 vulnerabilities. 14:22.400 --> 14:29.360 If I do exactly the same release, and I keep the symbols information inside the binary, 14:29.360 --> 14:36.400 despite it costing me 15% of the binary size, when I run vulnerabilities can on it, it will 14:36.400 --> 14:43.360 automatically detect that, yes, the module input has two vulnerabilities or five or whatever, 14:43.360 --> 14:49.200 but you're not using that functionality, and you don't need to patch or upgrade for that stuff. 14:49.200 --> 14:55.120 So, I want to encourage everybody to keep symbols tables inside your binaries, 14:55.120 --> 15:01.440 because that will help you later on to analyze and inspect whether your binary has 15:01.440 --> 15:10.240 discovered vulnerabilities inside of it. Confused looks in the room. So, how do you do this? 15:11.200 --> 15:16.640 This is the not slide, right? Do not use LD flags dashes, do not strip all. 15:17.600 --> 15:23.600 If you look at the binary published by all major Linux distributions, they currently strip all, 15:23.600 --> 15:29.200 and they currently remove all symbol tables from all of the go binaries, such that if you're using 15:29.200 --> 15:34.160 go binaries that came from Linux distributions, a nameable debug symbol servers, 15:34.160 --> 15:39.680 reattached debug symbols, and then rerun the vulnerabilities scanner that will help you to say 15:39.680 --> 15:45.120 actually know the half or two thirds of vulnerabilities that are being detected by secure 15:45.120 --> 15:55.040 it's scanner's false positive. To do do. There is small caveat, like in the previous slide, 15:55.040 --> 16:01.360 it says it doesn't appear. If you craft your code to use reflection and to do certain tricky 16:01.360 --> 16:06.880 things, there is a small percentage of cases where you're do not appear to use the symbol, 16:06.880 --> 16:13.280 but it's linked in your binary, and you're actually using it via tricks and hacks. But if you have 16:13.280 --> 16:20.320 very complicated go code, then you shouldn't be a go developer, and maybe try to write some 16:20.320 --> 16:33.440 blur code. I have this slide of encouraging people to do this often. Many times people in production, 16:33.440 --> 16:39.680 they run codes that was released a while ago. They're not on the latest release, not on the latest 16:39.680 --> 16:46.720 point release. However, it would help a lot if you're able to continuously bump the go tool chain 16:46.800 --> 16:53.360 stanza in your go mod. If you could bump all of your dependencies that have CVs resolved, 16:53.920 --> 16:59.600 and if you would retag your upstream projects again. You don't have to make a huge announcement about it. 16:59.600 --> 17:05.840 You don't have to write meaningless posts, PR statements, and whatnot, but it would help a lot for people 17:05.840 --> 17:16.240 like the case with Core DNS. If they would have pushed out a micropoint release of V1.11.1.1.1, 17:16.720 --> 17:23.200 right? Just to push out an extra release which uses a neural tool chain and uses neural dependencies, 17:23.200 --> 17:28.960 which have all of these CVs resolved without changing the Core DNS project itself. That would help 17:28.960 --> 17:35.040 all of the downstream users who still want to keep using that release, but they want to meet 17:35.040 --> 17:44.720 to get CVs without changing anything else. Today, if you followed the best practices, and for example, 17:44.720 --> 17:51.040 use Renovator depend about, normally it mitigates stuff for your master or main branch only. 17:51.040 --> 17:58.080 It doesn't mitigate your stable branches or any of the tags. I hope that maybe somebody writes actions 17:58.080 --> 18:04.320 to say, hey, I'm not changing any of your code. I only updating your go mod go sums, and I'm going 18:04.320 --> 18:10.240 to help you retag all the releases to just keep maintaining them because that would help people. 18:10.240 --> 18:14.640 Well, we do this a changer, but maybe upstream's want to do it as well. 18:16.640 --> 18:23.120 To do it. I hope to contribute these things with examples, and with benchmark 2, 18:23.120 --> 18:29.280 open SSF working group because they have an amazing hardening guide for CC++, 18:29.280 --> 18:34.240 but they do not have something similar for go, and there is a lot of things that people should 18:34.240 --> 18:40.400 be thinking about when they're building go binaries. At the front, I have some glitter stickers, 18:40.400 --> 18:48.800 and I have about 10 minutes for questions or answers, or even I can go over some of these things 18:48.800 --> 18:55.200 again, because some people look confused. Questions? I'm quickly going to him. 18:56.400 --> 19:02.800 Very close to the microphone. You have to kiss it. Thanks. So, just a question. I'm coming from 19:03.200 --> 19:13.200 OpenSusa, and you tell us to up the project to up the version number of the two things. 19:13.200 --> 19:18.960 Yes. That's problematic. I just yesterday came about the project. I wanted to package my project 19:18.960 --> 19:24.960 on devion, and that has go 115. So, what the hell? If you upgrade, all tell all projects to upgrade, 19:24.960 --> 19:30.480 then I cannot package it for devion again. So, what to do? Excellent questions. So, 19:31.360 --> 19:38.320 there is this conflict where security scanners look at the go version of the binary, 19:38.320 --> 19:44.240 and the tool chain version that the binary was compiled with is encoded in every binary, 19:44.240 --> 19:52.160 and it is the upstream version of go. So, it's like 1.15.2, for example, right? The latest point 19:52.240 --> 19:58.800 releases 0.7, right? And devion, for example, cherry-picked security vulnerabilities, 19:59.440 --> 20:08.000 and they have multiple releases of go 115.2, but there is no information in the compiled binary, 20:08.000 --> 20:15.680 whether or not you use the vulnerable devion 115.2, or they are re-release of it, which they call 20:15.680 --> 20:23.520 dash 2, or dash 3, or dash 4, or dash 5. I believe that we should fix upstream go tool chain 20:23.520 --> 20:30.400 to enable vendors of the tool chain to set their own vendor version of the go compiler, which they 20:30.400 --> 20:36.400 maintain for longer than upstream. Such that security scanners don't have to rely on the upstream 20:36.400 --> 20:45.600 version number, such that they can say, oh, yes, it's go 115, whatever, but I see in VCS or go vendor 20:45.600 --> 20:51.200 metadata or something that it was a patched version, because then we would be able to solve this 20:51.200 --> 20:56.960 in a way that exposes information to the scanners and lets up things correctly. So, yeah. 20:56.960 --> 21:03.280 Until then, please upgrade your tool chain standard, because that's the only way to go and see 21:03.280 --> 21:09.200 that all of your downstreams and users will build production rate binary. 21:09.200 --> 21:18.560 Yeah. One of your early slides was about ABI versions for architectures. Do you have 21:18.560 --> 21:23.760 a way like some advice on how I can determine what the maximum save value I can use for that is for 21:23.760 --> 21:29.680 my environments? How can I find out what all of my machine support? Yeah. If you have SSA checks 21:30.000 --> 21:38.480 to all of your machines, if you run LD so dash dash help, it will actually print in the output 21:38.480 --> 21:46.560 all of the ABI versions that are supported by your LD so tool chain, and it will say that, oh, 21:46.560 --> 21:52.800 I know about V4 V3 V2, and it will also say in brackets, and your hardware supports this one. 21:53.360 --> 21:57.680 If you can run that command across all of your clusters, all of your hardware, you can gather 21:57.840 --> 22:03.920 that information and you can say actually everything apart from that one staging server supports V4 22:03.920 --> 22:11.040 or supports V3, and then I can set it there. I did it for Wolfie by analyzing all of the 22:11.040 --> 22:16.960 instance types and all of the big public clouds that all of our customers use, because then I'm like, 22:16.960 --> 22:23.600 oh, I can go to V2 and in a year or two, I'll be able to go to V3 once a few deprecated instance 22:23.600 --> 22:30.880 types get eliminated from public clouds. I should publish that. Okay, I will publish that. 22:32.000 --> 22:38.320 Yeah, question. Last question? So in case of GoVone check, it technically is possible to run 22:38.320 --> 22:45.360 it not in mode binary, but also in mode on analyzing the source code itself, in which case 22:45.360 --> 22:51.840 you probably run and before you can buy the binary. Would the same set of problems basically 22:51.840 --> 22:58.320 be solved if you run it on the sources? So you would basically not be, you will be able to strip 22:58.320 --> 23:04.160 the symbols in that case, right? Correct. Correct. You will, if you have source code locally, 23:04.160 --> 23:10.080 you yourself will be able to run it. However, the people who run your binaries may not be the people 23:10.080 --> 23:16.320 who have the source code. If you distribute your application to run on prem or if you offer it as a 23:16.320 --> 23:21.280 software as a service where people deploy your code elsewhere, they may not necessarily have all 23:21.360 --> 23:27.760 of the Go code nor want to check it out. And also the symbols that end up in your binary 23:27.760 --> 23:33.280 are dependent on the toolchain you compiled it with. So source code will analyze it against your 23:33.280 --> 23:38.720 current compiler, not the compiler that was used in the past to build the binary that's running 23:38.720 --> 23:43.680 in production. So there is small sliver of things that will show discrepancies. 23:45.440 --> 23:50.240 While I still have all your attention and your very silent, I want to remind you that you don't 23:50.320 --> 23:55.440 have to be a go developer to give a talk here. And there is still a chance that you can now 23:55.440 --> 24:01.040 decide, I want to give a talk here. And we have space for you the last hour of the day, 24:01.040 --> 24:06.640 we will have eight minute lightning toss. You can submit them at gofirst.love slash lights. 24:06.640 --> 24:12.960 There are QR codes around the room to scan and submit them. But he proved that you did, 24:12.960 --> 24:17.040 don't have to be a go developer to give an amazing talk and that's why I asked an extra lot of the 24:17.040 --> 24:26.000 post.