Introduction
I'm starting this topic in hope of slowly populating it with some cool Windows Internals info and tricks that aren't widely known. Every so often I'll come in and post a new update with a new cool bit of functionality I've reverse engineered or discovered.
Required knowledge
A minimum of intermediate C and general systems programming should be enough to get you by in here. I aim to explain things reasonably thoroughly and simply so hopefully you shouldn't have too tough a time deciphering my mad rants. If you have any questions or ideas, please reply to the thread. Keep it on-topic though!
Warranty
I offer no warranty for this information. It's all pulled out of late night debugging sessions and my own intuition, so there may well be mistakes and confusion. I may have also used tarot cards and a ouija board as technical references at one point. Don't use this stuff in your KMDs. Don't quote this stuff to sound clever in front of your driver developer friends. In fact, don't use this stuff at all. It's brain food designed to help you understand how Windows works inside, nothing more. If you use anything I write in here and your computer halts and catches fire, a wild badger cockblocks you at Mardi Gras, or nazi aliens rape your uncle then it is entirely your own fault for not listening to me when I said you should never use this stuff. Mkay!? Mkay. Now to the serious stuff.
Process and Thread IDs - What are they and how are they allocated?
This question had been bugging me for a month or two until a couple of days ago when I tackled the problem head on and found out the answer. I had sat looking at my task manager and noticed that whilst new processes IDs were usually larger than old ones, the amount they increased was non-uniform and sometimes they did make backward leaps. So, what internal mechanism is used to allocate new pIDs and tIDs in Windows, and what do process IDs really mean in the context of the kernel?
First I tried asking a few people in the know about this kind of thing. My first port of call was Dark_Byte, the creator of Cheat Engine. I talk to him online on a regular basis and he's pretty much an encyclopedia of low level programming. Unfortunately the problem stumped him, too, since he'd never really looked into it. He also posed some questions of his own and made the point that even after a process exits you can query its exit status via its process ID, so the IDs must remain cached somewhere and there must be some protocol as to when they can be disposed of or re-used. During our conversation I also suggested that process and thread IDs may actually come from the same pool, hence the seemingly non-uniform process IDs.
I then asked Mark Russinovich (the Microsoft Sysinternals guy who wrote the Windows Internals book) who explained that pIDs and tIDs are simply indices in certain object tables in the kernel, but couldn't go into the details. Damn the limitations of working for Microsoft! On a side note, it was REALLY cool to actually get a response from him, since he's practically my idol.
Since asking people didn't really give me a decent answer, I had to go my own route. After reading the leaked NT4 kernel source code (you should be able to find this on ThePirateBay), doing a little reversing on Win7 and a few hours playing with my results in WinDbg I can confidently say that I know the flow of how pIDs and tIDs are created.
The kernel needs to be able to generate a sequence of process and thread IDs that are unique across the whole system. To efficiently and safely do this, the kernel creates a pool of IDs that can be used for both processes and threads. This pool is exported in the kernel as a HANDLE_TABLE object called PspCidTable. During Phase0 startup of the system, the PspInitPhase0 function is called. This function creates a HANDLE_TABLE object using ExCreateHandleTable, which automatically populates the table with 65536 entires. Each entry is a 16-bit unsigned integer (at least it is on a 32-bit OS) stored inside a list item object that is part of a doubly linked list. Both process and thread IDs come from the PspCidTable pool.
When a new ID is needed one can be gained using ExCreateHandle, which removes a handle from the PspCidTable pool and makes it active. The logic for disposal of these IDs is somewhat obscure, and seems to be somewhat like a garbage collector. When no handles in object handle tables reference the process ID, all references to it are disposed of and it can be dropped or re-used.
If the PspCidTable reaches a critically low level (i.e. there are very few remaining usable pIDs) it automatically expands to fill with unused IDs that were disposed of previously.
And that's pretty much it for now. If I have any updates to this, I will post them at a later date. I'll also soon be posting more cool facts about Windows Internals that aren't really covered anywhere else.
