A long-winded and probably inaccurate guide to the kernel development process (particularly regarding Baytrail)

Hi folks!

So I've been working on Miscellaneous Stuff this week - a little release criteria drafting here, a little dealing with NetworkManager package splits there, a little discovering significant systemd bugs, yadda yadda - and still trying to get a Fedlet 3.16 build that works for me. 3.16 kernels still won't light up the screen on my Venue 8 Pro, but one of my Intel contacts says the latest kernel in the fedlet repo works for him. The difference may possibly be the firmware (he has updated to the latest firmware from Dell, I have not because the upgrade requires Windows and I blew my Windows install away).

I got a mail from a very nice person asking how he could help with the Baytrail hardware support work, and as part of our discussion, I wrote up a rather lengthy description of the kernel development process - mainly intending to explain how he can find the work being done on Baytrail hardware, but en passant it sort of turned into a general description of how kernel development gets done. I figured since I wrote it, I may as well put it up here in case anyone else finds it useful.

Bear in mind this is just one idiot outsider monkey's interpretation of how things work, and it very likely contains errors. Corrections happily welcomed!

For me, anyway, learning how big projects are put together was mostly just a matter of experience, but you can find actual guides for some prominent projects, certainly for the kernel.

So, let's look at Linus' kernel tree:

Let's take graphics first. The kernel graphics drivers handle quite a lot of work these days, more than they used to; as a rule of thumb anything to do with actually initializing the card and setting a mode is going to be in the kernel, for e.g. So, all the kernel graphics drivers live in drivers/gpu . Just about all the ones you're likely to care about live in drivers/gpu/drm.

The three Intel drivers there are i810 (for very ancient hardware), gma500 (for the horrible Poulsbo chipset that shipped with some low-power hardware a few years ago), and i915 (for just about all the Intel graphics hardware you're very likely to care about). So all the stuff for Baytrail graphics is in i915.

The code for detecting the displays on our hardware, for instance, is mostly in the intel_dsi files there, since the displays in our tablets are connected via DSI. That code all landed in the mainstream kernel in 3.16.

One thing that isn't so easily documented (or at least not that I've found) is where all the stuff that winds up in Linus' tree actually comes from. The setup is that there are maintainers for various areas of responsibility in the kernel, who maintain their own kernel git repos with their own set of branches. Periodically they'll tag one of their branches for merging into upstream (or just note a particular commit ID on a particular branch), mail Linus to request the merge, and he'll review the merge request and (usually) pull it in. (understanding the kernel maintenance setup is quite handy for understanding git, as git was written explicitly to back how kernel development works.) then all the subsidiary trees get synced back against the upstream masters and start working on the next batch of changes.

So, the guy who's in charge of graphics drivers is Dave Airlie, and here's his repo. Stuff from his 'drm-next' branch gets merged up into the mainline kernel during the 'development' / 'merge' phase, between the release of kernel 3.xx and the release of 3.(xx+1)rc1. Stuff from his 'drm-fixes' branch gets merged into mainline during the 'stabilization' phase, between rc1 and release. So right now, Linus is merging from Dave's 'drm-fixes' branch - here's the last such merge, for instance. Those were the DRM fixes between 3.16rc1 and 3.16rc2.

Now Dave, in turn, basically subcontracts development of each driver, so the whole setup repeats with sub-SUB-repos pushing changes up to Dave. So actual development of the Intel drivers ultimately happens here. Again, you see the '-fixes' and '-next' split. I can't quite remember why they have both 'drm-intel' and 'for-linux' branches, but they're usually in sync anyway. So more or less, the stuff that'll ultimately wind up as bug fixes for 3.16, you'll see going into for-linux-next-fixes, from where it'll get pushed up to Dave, from where it'll get pushed up to Linus. You can see right now a juicy change as the last commit to that tree.

'vlv' is one of the keywords you'll want to look out for in dealing with our hardware: it's short for "Valleyview", and is the codename usually used to refer to our stuff. You'll see 'baytrail' sometimes, but usually 'vlv'.

The other piece of the kernel development puzzle is the mailing lists. The big hairy one is lkml, the general kernel development list. You can find its archives in a wonderfully Web 1.0 format here. People all have different methods of interacting with it; you can of course subscribe directly to it, make sure you have a good email setup though, it's a high volume list. Some people like to read it through a gateway called GMANE that converts it into a Usenet newsgroup. Slightly confusingly, GMANE has a web interface, so you can also find LKML there, though that's an even worse interface than the lkml.org one in my opinion.

If all you're interested in is patches, there is an awesome shortcut I only found out about myself recently, called Patchwork. Patchwork basically monitors the mailing lists where kernel development discussion happens, sucks out all the patches, and gives you an interface where you can browse and query the mails with patches attached, and conveniently download them in either bare patch form or 'mbox' form (with mail headers attached, as many git commands expect).

There's a list for Intel graphics development, and that's where all the patches for our graphics hardware get posted for review. So you can find them in Patchwork here. By default it shows all patches sent to the list that haven't yet been committed to a tree Patchwork knows form a part of the whole kernel development setup. You can change the query in various ways by clicking "Filters". So we can search for 'vlv' patches and find, for e.g., this one - and finally we more or less have an entire kernel development workflow! Patches are mailed to this list, reviewed and discussed, committed from there to the drm-intel repo (either 'fixes' or 'next', depending on where we are in the cycle and the nature of the patch), then up to Dave then up to Linus and finally out to the world.

Several of the lists are interesting to follow to watch Baytrail dev work. Some patches just get sent straight to LKML. 'alsa-devel' is the list where all sound development happens (for our hardware and all other sound hardware; very high traffic). 'linux-acpi' is for, not surprisingly, ACPI; 'linux-pm' is for power management (you'd think there'd be an overlap with ACPI there, I'm not quite sure how that works, I just watch both).

So, you asked about sound. Sound used to be a bit different from everything else, as ALSA was technically an independent project, but these days it works more or less like everything else. Patch review on alsa-devel. There's an ALSA git repo but I believe it's not used for the kernel drivers any more. Instead, there's a repo that's approximately the equivalent of Dave Airlie's DRM repo, owned by Takashi Iwai, who has overall responsibility for audio.

Changes either go directly to tiwai, or to various subsidiary repos for different drivers and then up to tiwai. AIUI, the sound hardware in the Baytrail tablets is part of the 'ASoC' subproject (that's another keyword to look out for). tiwai pulls ASoC work up from Mark Brown's repo. Mark in turn pulls stuff for various specific chipsets from other folks at Intel. It seems that a lot of the work on our hardware comes from Liam Girdwood. I don't know precisely how they handle their merges, but I believe that's the general flow of work. I think some work on our stuff goes straight into Mark's tree or comes from somewhere other than Liam's, but I'm not entirely sure of the details.