Episode: 3329
Title: HPR3329: Linux Inlaws S01E29: The (one and only) Linux Kernel Contributor Panel
Source: https://hub.hackerpublicradio.org/ccdn.php?filename=/eps/hpr3329/hpr3329.mp3
Transcribed: 2025-10-24 20:57:41

---

This is Hacker Public Radio Episode 3329 for Thursday, 6th of May 2021.
To its show is entitled, Linux in laws S0129.
The one and only, Linux kernel contributor panel and is part of the series Linux
in laws it is hosted by Monocromic and is about 84 minutes long and carries an explicit flag.
The summary is an eclectic panel of Linux contributors,
discuss technology, anger management and other things.
This episode of HPR is brought to you by Ananasthost.com.
Get 15% discount on all shared hosting with the offer code HPR15.
That's HPR15.
Better web hosting that's honest and fair at Ananasthost.com.
This is Linux in laws, a podcast on topics around free and open source software,
any associated contraband, communism, the revolution in general and whatever else,
fans is critical. Please note that this and other episodes
may contain strong language, offensive humor and other certainly not politically correct language.
You have been warned. Our parents insisted on this disclaimer.
Happy mum? That's the content is not suitable for consumption in the workplace,
especially when played back on a speaker in an open plan office or similar environments.
Any miners under the age of 35 or any pets including fluffy little killer bunnies,
your trusted guide dog unless on speed and Q2T Rexes are other associated dinosaurs.
This is a season one episode 29 of something called Linux in laws Martin Howe things.
Howe things. Things are great. We have some avians land and they wanted to join the show apparently.
Really? Looking forward to that one.
So much for hosting.
Which I'm a little confused about because we have this exclusive round of kernel contributors
on a panel talking about of course Linux and kernels and stuff.
So without further you sorry aliens you have to wait.
Just simply send a mail to gig at linuxin laws.eu or even better to sponsor at linuxin laws.eu if you
want to sponsor us and you thought you out right away so you can appear on a further future episode.
And without further you as I said let's get rolling and welcome the panel.
So tonight we have an eclectic panel of kernel contributors around a virtual table.
And the idea is that you shed some light on the on Linux itself as the biggest open source
project on the planet but before we go into the details why don't we do a short intro round.
Darry why don't you start?
Sure. So hello everyone. I am Dario. I am from Italy and I live in Italy.
I currently work for a Sousa individualization team and I guess I'll say that I started doing
kernel hacking some 12 see 12 years ago and I was told back then by the whenever different person
that what I was working on was bullshit but I'm still here. That's it.
Excellent. Very interesting. Mr. Jan Engel.
Yeah I'm as you may know just by the name German currently working and living in the US for 10 years
or so started kernel hacking think around 2000. At the time I wanted to some sort of
versioning file system basically I wanted version control the file system which
is a horrible idea but it got me into kernel hacking and 20 years later I'm still doing it
about half my time. Wow. Okay. Yes. On to Harylian. Sorry. Sorry. Sorry. Yes.
That's fine. So as the name suggests maybe for some people. I'm French. I work for Sousa as well
from Nuremberg in Germany. I work on the SMB stack at Sousa so I maintain the
the Samba server and I slowly migrate it over the years or over to the kernel clients to mount
remote chairs. Okay. Dave, over to you.
I'm Dave Frankler. German living in France were tired. I'm been hacking a kernel since I can't
remember. My main mission is to support the test and measurement drivers. So I support the
USBTMC driver in the mainstream tree and then I support a whole bunch of added tree drivers in
the GPIB Linux package. Okay. Chocopo. Yeah. I'm back. I'm sorry. I keep dropping off. No worries.
So it's my time. Sorry. I missed that. Yeah. Sorry. Just introduce yourself. Okay. Hi again.
I'm Jakobo. I'm Italian. I live in Italy as well and I'm doing a kernel development in
for like five years or so. So I think I'm the newest one. You're probably. I didn't mostly work
in the multimedia subsystems for camera integration. So I do mostly driver development to the kernel
and I still find it very funny even if I'm doing some more open user space development recently.
So I can mix back and so yeah, I'm still enjoying that and even if it's not been too long. So I
also sent my first batch to Greg with Thunderbird and I was arguing with him that I was right and
he was wrong for like three or four emails but then I'm still around so I think it's a good
side. Okay. Thank you. Look. Hi. Yeah. So my name's Luke Leiden. I actually started on
Samber. This was 1995 and I downloaded a Slack worth 3.1 by walking in on also raised into
the Cambridge computer lab and downloading it on 150 floppies. Then I did my first
reverse, after that reverse engineering that I did my first actual Linux kernel hacking was
the HTC Linux product, the Xanadux project, well reported. Reverse engineer 9 HTC smartphones and
got in a row and got the Linux kernel 2.420 something up and running on it and then later on
ported to a Syroslogics 90 megahertz CPU 2 that was 2.6. That's the 2.6 kernel so it's about 2003.
So quite a few long-term contributors around Xanadux. I think the only one that is missing
is Carlos from a completely mistaken but Carlos doesn't seem to be around so. Okay, cool.
Yeah, yeah. Let's start with a little bit of history. Given the fact that Linux has just turned
26, I think, there's a little bit of history about this whole thing. What do you consider the most
the most important achievement over this period of time and anything goes here. It doesn't have
to be technical. It can be, it realizes change can be the fact that ARM entered the architecture
mix in whatever was it, 94 or something. Anything goes. The floor is yours.
First off, I think it's 30 years, then it started in 91. Thank you.
Obviously, the fact that it's running at all as a major contribution project this size.
I mean, I read a piece about a couple of years back that this is the only operating system running
right from tiny embedded systems running up to mainframes. So I reckon the architecture
choice would come to mind. But again, people may have different views on this.
I would say it was the opposite. For the first 10 years or so of Linux existence,
it was largely in X86 PC thing and there was this NetBSD that was the portable operating system
that claimed to be running on 100 different architectures or something like that.
And then around 2000 early-ish, I ran the numbers and realized that actually Linux
by now ran on more systems than NetBSD. Well, that that trajectory hasn't changed. Linux runs
on pretty much everything. But it didn't start out with a claim to be this portable operating system.
It was very much written for X86. We still have code in the kernel that assumes
that the FSGS segment stuff that we inherited from the 386 is still around 30 years later.
It's gone now. Oh, it's gone. So it lived for a long time and I think it was even used
in other architectures or at least the names were being used for things that were clearly not
the same registers. The difference is more, it make it work at you. The first part of Linux
happened because someone at deck decided to send Linux a machine and Linux then had the
spare machine sitting around and got it working. And most ports afterwards were pretty much
okay, we have this extra machine and let's just copy the X86 or power PC or whatever
arch directory and fix all the bugs that were this particular architecture happens to be different
from the one we copied. And then that is very interesting. So that is very interesting that
you say it's X86 legacy because from the when device tree hit that was very, very clear that
people didn't understand the difference between an ARM embedded system where all the peripherals
are part of the actual computer and it's a comprehensive system where they expect it to be
PC like where everything was self describing a i.e a PCI express bus or a USB bus.
Yeah, in fact if I could say something it's to me what's really interesting it's not only the
portability across multiple architecture that is stunning it's also the different
use cases that say it works because we may have tiny embedded system not so tiny embedded
system like mobiles for super computers which might even be the same architecture sort of at
the end but they are very different use cases and links runs on all of them and it's all the same
care different configurations you are but it's all the same care now. Do we have any Linux RT people
on the call? I used to be one. When I was told that what I was doing was bullshit it was
the percentage was real time is bullshit and I was doing real time scheduling back then I'm doing
virtualization now so now I mean on this very common I found this very interesting because
if you take a look at Linux history the z architecture of 390 x as an s390 x support started as a
hobby project in bubbling in where some IBM engineers but it simply took the kernel and
in their spare time so the log goes anyway see if they could make this work as a I think initially
as a logical partition on the main system and then marketing it came along a few years later
in the rest of history right up to Linux 1 or what is called Linux 1 these days and so I fully
concur with the fact that yeah let's take the arc tree and let's see if we can get this running
on our beloved particular platform hardware platform. I think on the mainframe
took a while for marketing to take over but one of the what was he principle engineer or something
like that fairly quickly saw this Linux thing and thought that was a good idea
and gave or not precise the management support he was sort of on the technical side but he was
the sort of person that no manager really wanted to pick a fight with so effectively the
the small garlic village that tried to run Linux on the mainframe fairly early on had support
which helped a lot and then when IBM realized how many systems they managed to sell because Linux
ran on it that also helped quite a lot. The broad extent you think that the portability of Linux is
actually attributable to to Unix because I mean even Dennis Ritchie and Ken Thompson
ported the first Unix system on to IBM mainframe. It's like being a really simple operating system
Unix compared to you know like VMS or VM36370 that kind of stuff like a file system and everything
is part and parcel of the operating system. On the device that I did the porting it was for
the Xenodox project. We actually uploaded busy box via initial RAM disk over a serial link
believed or not using a tool called handheld reverse engineering tool which you ran on the
pda under wince gave it the two files the in the Linux kernel and the initial RAM disk
and slowly over time I worked out how to put ssh and x11 r3 into the initial RAM disk
in order to get graphical things so we could actually do some exploring and actually run applications
on it and then finally we reverse engineered the the nandflash drive from where they put things
onto the onto the device itself but yeah busy box was key for that not so much whether it was
when it was actually Linux or a posix subsystem itself. No interesting observation and I think
this is what makes the panel pretty interesting because you all seem to come from different
background. Since I don't have interesting stories that goes that back in the past to be
interesting but I might have a question if I look at the landscape right now. Do you think
is there any other space in software that might not be covered by Linux development or
whatever brought to the development development mentality Linux or is it more interesting the
space which is opening like with things like a risk five and open open software hardware basically
and open power. Don't forget open power. Open power. Nobody stood using this.
Yes me. I'm developing an open power processor. That's a very good point. I think you know we're
going to see more and more specific hardware architectures with accelerators and GPUs and stuff
like that. And yes you know we'll operating systems evolved to actually provide support for this.
Well the labours project was I'm heading is designed as a hybrid 3D GPU VPU CPU
you know single in a single processor it's actually extended the open power instruction set.
Interesting observation there because you clearly have the counter
revolution for one of a better expression the shape of Android right. If you take a look
at the at your tip of the Android spec there's more likely that there's more likely than not
an arm compatible architecture like Snapdragon or something with blobs put on top of this
put on top of the architecture in terms of SOC support as a system on the chip and
system it's just let me finish here as a system on the chip but apart from the kind of
proprietary SSE implementations this is pretty much AOS AOS P is an Android open
a source project. There's not much deviation going on here never mind innovation and
any thoughts on this. Well I think I have you started no.
If I can make an observation and that's maybe the perspective from someone who was
worked for also middle-sized business that when you choose to make a product with something you
get you get the BSP and what I've noticed is that is that basically maybe the last 10 years you
get an Android BSP which is designed to work in ways that optimize the GPU space for doing
nasty things in user space and distributing binary blobs and basically making the current
interfaces empty layers that just drives the right values to register basically I've seen that
from camera architecture there is really weird stuff out there and I think that one of the
damages that Android made it's giving a lot of space for doing this kind of things which
have brought many people to work with very old current version or very cripple current version
there was a statistic from Greg that says like a quadcom current is like four million
lines of code different from the main line one and I think one of the important things that might
happen is that give possibility to people that works with devices to work with a kernel which is
close to main line so they can contribute back instead of being old in that very best and I think
an Android has a tiny tiny it's been a force is the driving force is that I don't know if my
perspective is correct any thoughts you just reminded me so you know Intel sold the PXA
processor to Marvel that was the PXA series was ARM did a deal with Intel where they said
will give you a license for a hundred thousand pounds because we're really out of money
I think but they promised to provide modifications to that to the chip back but they when they
looked at it they found that the ARM 11 was so bad that they started from scratch and that was
and sold to Marvel who then bless them when they released the finally released the learning
kernel modifications that they've been keeping in house it was a tarball of seven zip files
where they had the developers had just abandoned the previous one they hadn't done a single commit
ever it any of the things they just taken whatever the latest learning kernel tree was
and it made entry modifications of no commits at all it's stunning so yeah it starts
some of the the disparate stuff of debt of old kernels and dating back comes down to the
to the the the the manufacturers of the processors not knowing what the hell they're doing
and goodness now the narrow started up
but this wasn't the cause for the advisory system was it
no no um that that's another story which I looked on with dismay from having um
uh from the reverse engineering of you know nine smartphones
and none of them shared any hardware and I'm looking at this is easy thing and thinking
what the hell is going on everybody's loving device tree but when you've got a hundred thousand
peripherals none of which are different and only you're only only going to going to have a seed
one then it's kernel device device driver for it because that chip is a one off for that product
and it's going to go end of life there's no freaking point in having a device tree
before before we continue maybe for the one for the few listeners who are not familiar
with the Linux kernel with Linux kernel source content source distributions maybe we should
explain what the device tree system or what the device tree is more than happy to do this
myself understand the other volunteers okay essentially it's a type of parameterization
for hardware descriptions inside the kernel as people already pointed out the little scums
from an inter background but the idea is and this is basically what was just described
with the advent of more and more SOC architectures and other stuff the ecosystem became
quite complex let's put it this way and the device tree was the idea to parameterize
the description of said devices to some extent and this is what exactly what you see basically
with the with the majority of arm architectures smart phones embedded systems you name them
more often than not these kernels would have a an associated device tree explaining or the
defining rather describing the hardware architecture the hardware ecosystem running on
so meaning that you don't have to hard code these these parameters into the kernel but rather
supply them as a config file for one of the better expression but sorry do go ahead just a
just a clarification for all listeners yeah where the assumption is that it's going to be shared
across multiple devices correct but if if I mean I don't know if you were the the the HTC
universal was basically a micro laptop with an embedded 3g modem in it so imagine your
um uh your your your laptop in size only five inches in size we think of course back in 2004
that was just amazing nobody had it anything like it but um it they ran out of pins on the processor
that GPIO pins on the processor bear in mind it's got 110 GPIO pins they hand hat to you
I had a high tech corporation had to do their own custom ASIC we called it ASIC 3 it had 64 more
GPIO pins plus some SPI and iSquedC and STMC ports and things and they ran out of those GPIO pins
and had to use the Ericsson 3G ROM communicating over the serial bus to use 16 more pins this is a
phone that had seven speakers an audio parts um and but it was a custom one off it absolutely
no point whatsoever in having device to it for something where you've only there's only ever one
chip yeah custom unique chip across the whole thing uh Martin do you want to take the next question?
oh yes sorry um yeah so we talked about uh that we've had the history and before we move on to
future opinion it's how you guys see it's um just wondering why did uh you all start doing
bits of kernel development and what did you do before that?
what was the motivation really behind contributing to an open source project
who would like to go first? I've told quite a lot so
it was the proverbial itch to scratch um okay I had a problem the uh only decent solution for the
problem was to write kernel code so okay the kernel is just software I uh was a student at the time
I knew how to program software how hard could it be um turn out to be fairly hard but eventually you
go through with it excellent you'd be doing it ever since yes yeah okay is is that how are we all
started? if I have to mention one motivation is that well whatever reason you get into that the
quantity of code which is available and the average how deep is that? I think when you try
when you start managing all that that gets really exciting and if you ask me why I've started I
cannot tell probably it was a university project but once you get that I think I got stuck into this
it's it's like reading a lot of books every day and that's if you if you enjoy this kind of
things of course it kind of pleasure and pain but I think the the diversity of Linux
is what makes it interesting at least I'm mostly considered the subsystems which I'm into I
cannot tell the real code part maybe that's nester but I think that that's why it's still funny
at least yeah I mean it was kind of the same for me I would say I was studying at the
university computer engineering but even since the beginning of studying university maybe even
before I always wanted to write system software let's say and so the chance came then during the
phd or actually doing some scale development and yeah that area sorry you're a little bit
quiet could you perhaps talk a little bit louder yes I'll try I think you're still very
faint faint somewhere just turn it up a notch and please repeat us because I could really understand it
okay yes I was saying that it was okay sorry go ahead sure it was kind of the same it's kind
of the same for me at what Yaku said the the the amount of stuff that it's the area of very
different things it's what keeps me still excited about doing it I always wanted to write
system software that was staying and the chance to actually act on the Linux schedule it was
the schedule back then came during the phd but then yeah still find it exciting and they
had even if I changed the systems okay excellent and I think all in the end well how did
you start and why did you start more importantly what was your motivation to start doing in the school
I have a bit of unusual path I guess I studied the computer graphics in university so like 3D rendering
video processing is sort of stuff and I couldn't find a job and I did this Google summer of code
saying on the summer project and yeah I did pretty well I think and once I was done with with
my studies so my mentor who worked at the time sent me an email saying no one like we don't have
any good application this year would you like to apply again so since I was finished I was done
with my studies I couldn't apply I wasn't understood anymore so I said yeah I'm not a student anymore
but I'm looking for a job and I couldn't find anything you know computer graphics related around my
area so he said oh yeah I'll you know just send your CV over and it will take your look and the rest
is history as they say so I got the the job that's with me working on the SMB SMB stuff
mostly standby in the beginning since that's what I knew but you know how so we have a lot of
books to work on and eventually we had kernel bugs related to CIPS which is the name of the module
doing the kernel clients for SMB and so little by little I started working on kernel bugs and
now that's mostly what I work on I don't really touch the server side anymore so much
so yeah interesting with with 25 to 30 years in the making given the fact that Linux is already
apparently on a planet called Mars these days where do you see this going
you're asking me or no I'm asking this depending in general and of course you're more than welcome
to take this to take this question oh I forgot to I forgot to say the previous question
so I love that you are working on the CIPS stuff I'm already on because I was the person who did
the network reverse engineering of the anti-domain protocol back in the 96 to 2000 I think your name
is familiar somehow I think yeah yeah you must know James McDonough
we're James was 22 years ago now so my name is getting quite quite rusty I'll not
on that one I got into the Linux in general because of the yawning chasm between the windows and the
and the and the unix world and sambal was the the strategic choke point for that but after
the things didn't really go well because it's quite difficult I don't know if anybody else has
done any reverse engineering of device drivers and and things don't prove reverse engineering
of peripherals but imagine that you are looking at some network traffic and you're literally
just copying and pasting the you implement the client and send it at the remote server so
then then you get a response back and you copy that and you put that into the server and then
you get a client to talk to your server and then you go into the next packet so it's a bootstrap
mechanism and you have no idea what the hell the traffic is doing and then you present that to
you're the rest of the team and they ask you okay so now it's time for a review what does this do
I don't know I think so what does this packet do I don't know it's reverse engineered all I know
is that after 10,000 network packets it just works so it didn't go down very well and I had to
leave the team but then later I got into reverse engineering of smartphones win smartphones for
exactly the same reason that I could see that there was going to be with with the win smartphones
there was going to be exactly the same divide between people who had windows windows see smartphones
and anything else so and then 2006 Andrew came out so I stopped doing that cool
so just to go ahead just to complete the list of how do we get into kernel hacking or kernel
development for my part I used to work for Hillary Packard a lot of test and measurement stuff and
I also tried to reverse engineer a driver for a GPIV or HPAV board from Windows NT
it was extremely difficult because you know it was going by a PLX PCI bridge and then to an FPGA
which had to have the code loaded into it and eventually I managed to get information from
the manufacturer as to what the code is to put on to the FPGA and once I got my driver working
on my decided to contribute it back to the to the community and that's how I ended up becoming
a maintainer for the GPIV at least up I then bought myself a USB to a oscilloscope only to find
that the USB TNC driver from Linux didn't support the IEEE for 8.2 standard so I implemented the
missing code and contributed back again and learned what it was to try and get past the kernel police
that sounds yeah that sounds like Glasgow by the way yes
as a next question how difficult is it to get past the kernel police
anybody having the quality of your code
I wanted to answer the last question regarding how do you get into kernel development
oh my case I started using Open Solaris back in the day and they had very good
documentation but I have a cheap MP3 player they have an F32 partition so that that I couldn't
mount that partition on the Open Solaris so looking at the kernel I guess I realized that
the block size was calculated in the kernel so I changed that to be able to mount the device
so it was more for necessity to go into the kernel development but after that I went to the
I had I had a job that was more sort of a trial by fire because I have little knowledge I was
as a student and my assignment was to port or integrate the avi patches into the kernel that was
to that for that at that time to be able to run cobalt micro focus cobalt from Linux
to cellular Unix into the Red Hat version that was 4.0 so that was a learning experience and
I'm really frustrating experience using gdv s2rs just to find out which code was missing on the
avi patch to make this cobalt finally work and those years yeah Linux didn't have bpf ebf
understand but sorry has the so the trace so yeah I have experience on solaris at that time and
and I was kind of frustrated that I couldn't use the trace and after 12 years now we have bpf and
all this observability that so I had on those those years but we're really good now regarding those
years okay I'm going back to my original question people where do you see this going
in terms of it's it's the biggest it's the biggest open source project in the planet you have
at least more than a thousand steady contributors there's a guy seriously affected or potentially
seriously affected by the bus vector sorry bus vector of course being if Linux is run
over by a bus of course you there are kernel kernel source commentators but ultimately he called
the shots if I'm a completely mistaken what goes into a kernel and what doesn't
needless to say he's very outspoken about certain commodity issues but that would peak my next
question but let's focus on on where do you see this going first anything goes that's the
endora question I guess well I think there's a a desire to start cleaning the kernel I get rid
of the the crafty stuff you know we talked about set a phase get a phase going you know so the hangover
from the 286 to 86 days they've just dumped a whole bunch of ymax code because there aren't very
many ymax users in fact there are none so I think you know where's is going to go we're going to
need to do a lot of cleanup in order to you know get Linux or keep Linux as agile to adapt to new
environments as it has been I think that more than technical issue that that might be many and I
think also the push for removing all the architectures it's maybe due to that it's how well that
that this thing will continue to scale because in in the last year there's been several talks and
presentation about how maintainers doesn't scale in example and the number of contributors keeps
growing but some subsystem are really there is people which is bottleneck and that's not
their fault it's just the structure which is is not scaling fast enough probably and now to make
it scale in a way which is consistent for something like it's what remains of a community because
it's it's huge and it's very hard to keep it together somehow how to scale and out to multiply
the responsibilities which means not to also maintain code but testing also and also how to make
that I think it's where past the professionalism time where it's it's it's it's harder to
contribute to the kernel as a single individual so most maintainer now I have a paycheck for someone
which has a lot of power in in Linux to take decisions and also I so I think scaling in a way which
is not letting grow organically it's a key to keep the community functioning now and be less
independent and to in respect to some pushes that has been in the last years probably
and that's of course as technical repercussion like what you said removing all the
architecture in the body in the body maintains that there might be uses but maintaining an
architecture it's a lot of job and now it's probably harder than than 20 years ago
yeah the Linux doesn't scale threats started I think in 2001 so scalability issues are nothing new
and in spite of the doom and doom in 2001 we survive and we're still around so I'm not that sceptical
so this is what you remember the the Cambridge Linux
Linux meeting where Linux asked all of the kernel maintainers the heads of the kernel
kernel maintainers to to to turn up and there were 36 people at the meeting 18 of them were
armed people and then I said what are you doing here you're armed people go away and come back
when there's only one representative with for the whole of arm and it's like oh my god thank
it's not recognizing that you know at the time there were a thousand licensees for the
arm architecture they're all different they happen to use the same instruction set or
actually they don't because you got arm 7 9 11 the weird Broadcom version arm hard arm 11 with a
hard float thing and then you got the cortex they're all completely different and very sadly
Russell King it's Russell still around as they are maintainer
he's still around whether he's still around us the arm in China I'm not so sure yeah because
when I when we submitted our team submitted the the patches upstream we had done we divided
down into separate files in a subtree below the main arm thing this is when there were only
200 files in the 100 files in the arm kernel device tree device tree directory
Russell asked us to submit it under the current structure which to put everything in a flat in
one directory and was so embarrassed by this because it would have dropped 250 because we have
the nine smartphones that we've been reverse engineering it would have dropped 250 files into that
directory and completely overwhelmed it and I was we were so embarrassed that we just didn't
didn't submit it we didn't take it further because they don't want to embarrass or overload Russell
that I think connect with the fact that for the perspective of being relatively new to the thing
and looking at a bit from the external as I feel myself and still wondering what I'm doing here
I think 20 all the story about people which is very close to Linus and that direct interfacing
with Linus work when the system most broke this motor and now we have people which is indeed
core maintainers and core developer of Linux which is direct access to the maintainers meeting
but there is there is a lot of work that needs to be done inside the subsystem and there has
been a push with the subsystem maintainers book sure maintainership like the RMKMS is doing
and so I'm not concerned about the the the fact that core developer could scale and access to Linus
and a functioning way of keeping it going I'm more concerned about the fact that a lot of people
has a lot of responsibilities and that it's it's bad for their health in session time because
there is people which is very overwhelmed and also it may be hinders a bit the the contribution
weight of some of some part of the kernel or make it less reliable and testing or blah blah and
I think growing in a number of contributors and but also people which has smaller responsibilities
a key for first scaling in a way which is sustainable and maybe you made maybe you see that
happens we've not concerned I think a problem there is always
code review in a way you have way more people that have written some code and send it upstream
wherever upstream is and now upstream you have far fewer people that get to take a look at that
code decide to merge it or more likely decide to say you could have done this a little bit differently
and review is not that much less work than writing the code in the first place depending on the
quality of the code it may actually be more work so you tend to have reviews that are overwhelmed
or then from the contributor point of view review is not happening or it's not happening fast enough
and I'm sending my patches and get no response and getting people to actually send good patches
where the review is very simple and eventually you build up trust and all this comes from Jens
Expo I will merge it because everything has been fine most of the time and if it isn't I
complain loudly once and he will learn getting these people is important but you get obviously far
more people that do one kind of patch in the lifetime it never come back or do a few things or
just don't have the quality or don't have the time they are willing to commit and the botanac tends
to be on the one run up the people that are reviewing code to maintain us and they can spend 90
hours a week and still not get done so there's an issue and I think this was very well
connected with the previous question how hard is to get past the current police I think the
real question is how hard is get to be the current police look at you sometime for contribution
because there is shortage of people doing reviews and that plays a role something like
I think it's still the imposter syndrome sometime it's happened and the very beginning when you start
contributing look at patches and say yeah it's okay for me but who am I to send a review by tag
which is instead very helpful for the community to know that someone are looking to that and maybe
and there's been a discussion recently I guess if it's not worth to make it the review tags
reach it like say yeah I'm looking at this part I have no really idea what this part does but that
part I look into that and it's fine to me and that might make it easier for people to feel like
not embarrassed to say yes I've reviewed that I take it the authority to say I've reviewed that
which I think plays a role in the fact that people may be reached code but does it really feels
like saying that that's okay for me I take responsibility to say so because who am I to say so
it is there a particular reason why something like Garrett is not being used I mean will it
is it just the sheer size that the whole thing would fall over or require a massive super computer to run
you mean Garrett the the code review system the patch review system yeah I think because that's
opened a never-ending discussion about why people would not never give up the best review by email
and I'm totally I do love that I do love that absolutely but we've seen several several time we've
seen some system moving to get left here I can't ask you just I have used Garrett for other
projects the luminous issues in Garrett now a free svvv is using a regulator and often this V just
use send the patches to the tech list and you're done but yeah mostly think that the current
way of doing things today is fine maybe and just just do it but I have used all the systems that
for those for those karnosan I think is the review by email is pretty good yeah I agree with
you Carlos and the other thing is that you know there are so many trees that different patches are
going to go into depending on what area you're providing you're submitting a patch on it's not
like your patches are going into one place they'll go into sort of you know usb next and then into
usb and then into linux next and then into linus is tree so you know automated code review system
is not going to be able to deal with the diversity of trees and which tree feeds which are a tree
subject to which conditions etc etc it would fall over we'll acquire a super computer to run
yeah well if if nothing else if you wanted to make changes you have to acknowledge that it's
distributed and you need the ability to make local changes while still being compatible with the
interfaces to the unchanged part of the infrastructure so basically feel free to use Garrett in your
subsystem but then send patches via email or whatever or send full requests to the next
sprung up in in the tree structure yeah sorry go ahead I mean if if Garrett ends up being a good
thing and everyone is just raving about it it will spread and if it's if people don't like it that
much it will not spread it's very organic well that's the usual evolution called the
the survival of the fittest right which you by the way see outside the kernel in the
in the ecosystem as well and changing tack a little bit not to show how many of you have followed
the last years linux plumber conference because something very interesting happened there
a few people made a proposal to introduce a second program in language in the kernel called rust
now when I review the slides of the of the presentation I came across a remark from linux himself
saying that and I'm over simplifying things but entering c++ won't happen let's put it this way
there's a very interesting mail on the little kernel mailing list outlining the rational behind
this and as usual linux was quite explicit on this but linux actually blessed subsequently
the potential use of rust in the kernel as a safe alternative let's put it this way to see
so given the fact that this is a major change or can be a major change let's put it this way
after about almost 30 years of programming in c what are your views on the introduction
of this new program in language given the fact that this is mostly aimed at the moment if I
understand this correctly a driver development and that's kind of instrumental sub systems like
schedules and so forth I think it's an interesting attempt but it's still an editorial state I think
at this point there's so many stuff to to need it to make it work that I need some likely to see
anything interesting and you know for the at least two or three years I don't know couldn't
conquer more at the moment it only runs on 64 bit inter architectures you need a special c-lang
version for this you need a nightly installation of the rust compiler and and a few other bits
let's put it this way always gonna need a nightly version of the rest of my life but I consider this
to be the first step as you rightly said and given the fact that it took almost 30 years because
until now pretty much the kernel has been done in the semiconductor and c and that's where it stops
I think that's an interesting interesting developer let's put this by especially given the
traits that rust has in comparison to plain c I think it was it's surprising how Linus was
convinced to emerge this I didn't expect it to go that smoothly well yeah me too at least a
certain extent then the entirely personal opinion that I've grown about this is that
I mean at some point I was to say I mean kind of a lot of people are pushing not not
necessarily from inside the head of the community but even outside and what happens outside in
I don't know matters to certainly really so multiple examples of that so I rode opinion that
since there is a lot of attention of these there is a lot of motion these then at some point
probably fighting toward against something like this would just have been not only perceived as
bad but also required a lot of effort while letting it in in this way and waiting for seeing whether
people will ever be able to do anything useful with it which perhaps yes but perhaps no
was the least resistant path maybe I don't buy I don't buy that at all that's not how Linus states
if he thinks that the code is crap while the design is ugly or use any other curse words
he will say so very publicly and if half the current community disagrees with him but he still
believes that he's right he will not back down so this is not based on popular opinion this is
Linus is at least somewhat intrigued by what rust offers memory security in the set type safety
and things yeah whether he's entirely convinced I'm not sure but he's at least intrigued
yeah I think it's strange because you know he's been very adamant about how he dislikes
C++ because it's very complex and other things but rust is also very complex I think compared to
the simplicity of C it's it's not that easy like there's a book to know as a as an exercise to
so the book is about implementing linked list it's a whole book dedicated to different ways
of implementing linked list in rust because it's not that straightforward and actually to do it
properly you know with the language so yeah it's really surprising I think for me
that's the first reason why there's a component in the standard library coming with rust to
implementing lists and but I get your point the thing is the learning curve and I've mastered
that much of a few years back and I wouldn't consider myself for us expert by no means
it's quite steep full disclosure I see was the third language I learned even before starting
university and she has basically been been following me let's play this way for for to bet
expression and so I'm not saying hunting me but C has been basically with me for my for the
better heart for the better part rather of my professional life and I still regard this as one of
the well let's put it this way not complicated languages but you see rust has one big advantage even
Vince the grass compiler to generate code for you code for you you're rather there in terms of
there's not much left in terms of final QA of the code whereas in C once you kind of
convince the compiler to generate code then the real fun starts and give the fact that I'm talking
to currently developers I don't have to explain what an oops is so this is my take on the situation
I mean I agree with you but at the same time I mean C++ also had some arguments you know
some ways to do check compile checks and things like it's not as advanced as rust but still had
some features for that and the latest solid was you know too complex where he couldn't I remember one
of his argument was when you write C you know that a few lines you write you have you have an idea
of how they would compile to you have an idea of the instruction getting generated and such
and for us it's like C++ you don't really know one line could generate a lot of things you have
not really there's no guarantee on that so I think it's very complex language
yeah but I think it's really more complex than rust because of the
processing of operate we have a loading language I've known of people who
why is my code running slow it's C++ is because they did so many operator overloads and somebody
helped them single step through it and they were horrified what they produced
I think what I think is rust allowed in the kernel and that produces more people that could
contribute or maybe more drivers that I think that is when when should we do it for for all
means more drivers maybe more laptops are supported and more more devices are supported
I think rust is a good language I learn first C then list then C++ then go then I'm learning rust I
think it's a good language is yeah it's more obviously more competency but I think choosing
between C++ and rust I choose rust rust I think one of the beauties of C is actually if you
if you're using it long enough you pretty much have a good idea of what the assembly looks like
once you code in a statement and see that is not the case as you just said with C++ rust I think it's
halfway in between let's put it this way complex or what complex I think the safety argument can
make the real difference compared to C++ because it might they might be more or less complex
one of each other but one guarantees you safety for some for most of the of the the meaning that
you can now you can assign to safety the other one doesn't guarantee you that so that's a win
exactly you just have to take a look at the at the at the ownership concept because I'm doing rust
and I think we'll figure out two technical depth and these metrics I would reckon are come into
factor come into come into play a play certain role here because essentially it's talking about
how many million lots of code these days the kernel itself about 20 million last time I check
but it's been a while okay so reckon it hasn't well the last couple of of releases have seen a
certain vanishing of subsystems but I reckon it's right to say it's still a complex beast and
factors or metrics like technical depth and especially if you refactor power portions of such a
large significant code base then any little tiny help you can get from a from a from a from a
development ecosystem like like rust and carbon all the rest of it I reckon do factor in big time
well it's interesting as you mentioned cargo there that's one of my biggest concerns is that
whatever is behind hard cargo gets hacked when I did a very comprehensive review of
the the requirements for secure code distribution I was it took about a month and I found I came up
with 17 separate and distinct requirements to do the full chain of trust and it is concerning me
that rust developers are heavily reliant on cargo which downloads code from completely random
locations that they've never even checked where crates that I would be a go to resource needs to
say you can reconfigure this yep but what if crates.io got gets hacked this happened with
or some location gets replaced like Arch Linux had a package that was replaced by a GitHub
user developer had closed their account a hacker had registered that same exact same username
and created the exact same package and managed to get get a Trojan uploaded into into us Linux
that is of course the danger with centralized package distribution
infrastructures but at the same time it has happened to part why and I think it took it
it took about what seven hours for these I don't know how many I pass of eyes to spot this
and to eliminate this grant as it wouldn't happen with Debian because they they use the
what a webed a chain of trust to to to do GPG signing of the things it would never happen with Debian
well for free just submit a poor request for for cargo for cargo or for for great
oh here no jokes I hear you absolutely Debian has had other security at night may as that
shall not be named so I won't hold up Debian as a final example okay guys okay final final
question are reckon because we we have to close the show at some stage Linux itself Linux of
course is a bit of a of a colorful character let's put it this way and especially if you take a
look at something called the Linux kernel main Linux you'd see quite a few runs you'd see quite
few examples of very strong language that split this way there was this hiatus thing where he went
actually out of I would say out of business but what's the bottom for into a temporary time let's
put it this way I think a few years back came back there were rumors on the street that actually
went to the therapy he admittedly did say about himself that he's that is coming from a dysfunction
background but then he sees nothing wrong with being abrasive outspoken never mind insulting it
at times what is your view on this hmm if I if I may so is anybody seen misbusters is anybody
else a fan of mythbusters I love it I watch every every eight months I would watch all 250 episodes
of a period on a cycle and one of the episodes they had what is the effect of swearing on pain
because you know if somebody hurts themselves they swear right so they they studied
scientists did a scientific control study of whether if you swore with where your hand was in
an ice bucket which is a unknown way to to to to induce pain whether you will be able to tolerate
pain more if you were swearing and it turned out that you can and I think that's sort of people
misunderstanding why people are swearing on the mailing list it's because they are relieving stress
and that's a yo to to to be intolerant of people because they are doing so is completely
misunderstanding the neurophysiological effects of the the extraordinary complex job that they're
doing I fully appreciate that concern on the other side they're basic social protocols which
you violate when you become that abrasive that insulting because that's normally not how society
is work isn't there a bit of responsibility is around that about all the custom chatting that
happens around allliness as as called this guy on the mailing list and you go there there's
four onig say here's the middle finger to it to someone isn't that a bit responsibility to
it made too much gossip around that what we care is actually working I am trying to play
devil's expert advocate yeah but no I'm in general not the specific I'm in general about
the initial characterization of that here is the background the feelings and that opens
totally different argument and the definition of people there were quite a few developers who left
the ecosystem because of his behavior if my results correct on the other side you also have
developers that have left or not joined various new projects because quite frankly the culture
doesn't fit absolutely without taking sides you have a certain culture in a certain project and
if you are okay with that culture free free to join that project contribute have fun
if you don't like that culture go find a different project
I think I'm one perspective that sorry but I've been talking about it with a friend
a few days ago so I'm sorry to interrupt you but I understand what you say but it told me you
have to consider people liking different cultures when being schooled in public it's like public
communication and so you have you have a developer for from a culture where that perceived being
schooled in public or being salty in a way which is it's not a fending for me something like
that hands his career especially in Japan or exactly I would think about Asia
there were similar issues in the airline industries where exactly those
different cultures where our saving phase was a big thing had airplanes crash because the
co-pilot settings like who the weather looks bad which would have been translated to your flying
into a storm and you're going to kill us all but because culturally it was not acceptable to be so
frank the co-pilot tried to warn the pilot in also many words and the pilot didn't get it
and the plane crashed and 200 people died and there were I think multiple instances of that
so now it may be bad for your career but but it also has consequences to be more polite and
in the airline industry people actively went to a more confrontational it's paid as it's paid
culture is that a good thing or a bad thing it's harder to say for the colonel but
somebody else's career made me it not be the most important thing in the world
somebody else's life might be more important the quality of the product might be more important
I still feel your right it is important to recognize the responsibility of getting things wrong
especially with Linux now being used in embedded systems mission critical systems
frank but insulting is really crossing the line of things especially when you're communicating
by text you don't know there's no need to insult a thing so to over simplify things the more you
use rather but in the code get the code gets especially in embedded systems
I'm just I'm just over simplifying things here I think it is really important to understand that
when somebody is shouting like that that they are releasing stress and they're it's a way of coping
for them just off and spread okay we're wrong yes I'll fully see that angle
I mean this I get this when it's happening in like when you're talking with your voice but when
you're writing typing I I don't buy it I don't think this releases any stress by typing insults
especially after the fact maybe Linux has a voice to space system it does review like that
yeah I've done it I've done it many times really all right I mean he's he's not the only one right
maybe he's the worst but quite a few people are quite a pretty outspoken whenever I'll
fully start I don't read the LKM regularly but ever whenever now then I do take a look and
quite a few people quite a few contributors don't beat on the bushless British way especially
when they have a concerned voice okay any final thoughts before we wrap this up
I think that one was quite a quite an interesting thorny thorny topic I do wish I would like to see
that there was it was a how can I put it a conflict resolution series of procedures that people
would would felt safe to call on in in three separate projects that everybody was happy to to use
maybe it's more a diplomatic process you know voting or something I don't know
but the one I mentioned this I did it with JT and the open source this what open source voices
one there's a website called CRNHQ.org which has some great conflict resolution things and
there's also the Aldi Institute you know the guy from MASH who helps people to scientist to
communicate and these these procedures but the processes that the CRNHQ anybody can read them
they don't need voting it's it's about empathy and people acting in a role of mediator should they
choose to do so to to listen to what the other person is saying I think it's very interesting
I do wish that more free software projects actually use this include
for Linus's case and he's a personality that we've lived with with a very long time I mean he's not
he's not surprising the way he is and you know he has a model of the benevolent dictator and he's
not going to show empathy with something he thinks is a bad idea ever you know so you just got
a look where that all just you know find another project that's my opinion how do you
sorry to ask you a question no I appreciate we'll wrap up but how do you square that
when you see it as a duty or responsibility to fix something fix as in you have a bug in your code
or fix as in you have a bug in your culture a bug in a culture or a systemic systemic problem
that I think for example with the the whole reason why I got into sambal was to fix the
yearning chasm between the windows world and the and the and the unix and linux world where
people would get trapped running windows and couldn't convert over to running on a unix system
a unix fast system well if you follow the discussion about the introduction of CFCI I think was
mostly about that moving from a code of conflict like the Linus kernel that to a code of conduct
which is punitive and the other implications I think it's part of this of this process between
trying to adapt the culture to be and which not being insulting doesn't mean accept in bed code
but sometimes also saying like no this is wrong asking why did you do that it in reviews makes
difference and maybe people is sensitive I know and sometimes you have to the feeling that people
they think to seriously grow but I think when the introduction of code of conduct and removal
of code of conflict or all the day made that it arose around the proposal communication
standards in lavella it's part of this kind of evolution and the empathy that you've been talking
about probably okay we had a code of conflict and I mean the rules were already there but they
were not enforced like you have not to do these things and that and that was not enough so
probably shows that something was not totally okay with that that's a big for another time that
one is a whole dollroom ever so absolutely I know that it's not that the end of the right answer
for the last question let me share yes and there will be a continuation of this at results
you will you will get the right answer no worries okay Martin you would like to do the other
to wrap this up yeah that's like to say great contributions from everybody and a great bunch of
experience that you all have in in developing Linux and obviously there's many many systems
relying on your contribution so great job everyone and thank you from myself and likewise from
Chris I'm sure yes thank you very much for participating and thanks again for participating thank
you for your life thank you
well that was a very eclectic discussion don't you think yeah it's great insight into the
well into the the biggest and longest longest running not entirely sure probably the longest running
open-source project right so is there any other one running longer I can't remember
of course ebex ebex never mind ti yes our favorite ever so excellent indeed
and what I'm particularly interesting was the discussion at the very end about swearing and
lowering paying levels and all the rest of it yeah I agree that that's a bit of a common theme
amongst well especially if the for people who are considering contributing considering patches
right on on any project what are the the levels of you have to or the hurdles you have to overcome
to do this and why would you do one if you're just going to be shafted out to put it simply
and then but then only on the flip side you have the case of a recent episode which is yet to be
released where you know your code is public and by the way this is a rust episode coming up
your code is public and people will dig into it and people will find holes in it therefore which
is the great thing about open-source many so holds in rust okay not not in rust itself but in the
application of it I see I mean this is yeah this is the usual dilemma right because as as the
discussion turns out yes there may be indications of swearing lowering paying levels I get it
but on the other side that violence basically basic social protocols so when so you essentially
you're working a fine line between lowering your paying level and pissing off contributors as
in fellow contributors and that's exactly what happened because quite a few people actually left
the project right right there was some intel girl lady details will be in the show notes of course
who left for that very reason and I can at least recall one subsystem in here now defecting for
the same reason because he was simply not having it with regards to the behavior of certain
benevolent dictators for life as a speaker fails of the project as you put it it's a certain social
protocol was I mean just take a look at at a self-confessed artist called Richardson Stormman
hmm yeah it's if if people choose to break those protocols and they do so for a reason and as you
say maybe for their own reasons but then that from the contributor side you have to consider
is it worth this I mean okay you may not agree with it but what are you trying to achieve with your
contribution right so that is maybe the main thing that is important here rather than
whichever way people communicate you have to be beyond that right I mean this is this is the
reason why I suppose that many many many many projects now and you'll see this happening with events
as well now I have something called the Court of Conduct which explicitly outruled such behavior
right that's very sensible yeah yeah and I think someone made the point about
the fact that it's done via email right so you know if you swear in a moment when something happens
and its reaction to a certain event or a pain or whatever is then that makes sense but if you
do an email it's it's a thought process that you can easily reconsider or not apply and let's
put it that way plus the fact that either for example the Linux kernel mail-inness gets archived
so your rands your swervings your your your operas are there for eternity to everywhere for
everybody to read not great no I agree with that yeah yeah so yeah but I think we had a number of
people who well actually we had the people in both camps right indeed yeah and and most of them
most of the panel were on have been contributing for many many years right apart from a few relatively
new guys as in five five years I think some of that but um all the others have been doing it for
many many years so they obviously you know as I said CB on that or or put up with it or whatever
you want to call it yeah indeed yes for a bigger bigger project right maybe they're just born
to suffer no I'm joking people this is not this is not this is not this is not how I see things
actually I see both sides but I think society and this is a broader philosophical statement now
so td has got to the states where it's in and of course that is a subject for debate for the
other whole different show not by swearing but rather by cooperating and facilitating and stuff
and social basic social protocols played a very important part in this part in this
indeed indeed and we are at every end right or any final passing remarks
well the final passing remarks apart from that that last topic that I found it yeah interesting to
me people that have been everything at that kind of level since since the year zero pretty much
yep pretty impressive and that brings us to the end of sad show season one episode 29
the big kernel panel and see you next time bye bye this is the Linux in-laws you come
for the knowledge but stay for the madness thank you for listening this podcast is licensed under
the latest version of the creative commons license type attribution share like credits for the
intro music go to blue zero stirs for the song summit market to twin flames for their
piece called the flow used for the second intros and finally to select your ground for the
songs we just use by the dark side you find these and other details licensed under cc hmando
or website dedicated to liberate the music industry from choking copyright legislation and
other crap concepts
the
You've been listening to Hacker Public Radio at HackerPublicRadio.org.
We are a community podcast network that releases shows every weekday, Monday through Friday.
Today's show, like all our shows, was contributed by an HBR listener like yourself.
If you ever thought of recording a podcast, then click on our contributing to find out
how easy it really is.
Hacker Public Radio was founded by the Digital Dove Pound and the Infonomicon Computer Club,
and is part of the binary revolution at binrev.com.
If you have comments on today's show, please email the host directly, leave a comment
on the website or record a follow-up episode yourself.
Unless otherwise stated, today's show is released on the creative comments, attribution,
share a like, 3.0 license.