The way I have always reverse engineered an undocumented network protocol is to code a client and server at the same time.
Here's my process for example:
1. Run official client / server programs (if available, otherwise, just the client)
2. Note which ports they connect to / open to listen on
3. Write a quick program to listen on the server port.
4. Redirect the client to your server (lots of ways to do this, hosts file being one)
5. Run the client... see if it sends any data first. If it doesn't send data, the server probably initiates the hand shake. If it does send data, then ..
6. Start writing the client. If the client starts the initial hand shake, send the data to the server and see what you get back.
7. Add this response into your server and run the official client again.
8. Repeat the last few steps as necessary.
Once you start to grasp how the programs communicate with each other, you can start to understand how the actual communication works, then start writing a protocol spec.