[Tutorial@Python] Network communication

[flowjob]

Hi,
I decided to make an contribution to Evilzone, and so I'm writing this tutorial about network communication in Python.

Requirements:

• Basic knowledge at Python
• Python installed on your computer (Python Homepage)

1.0 Network communication

The network communication can be divided into several layers.
Illustration1.0.0 shows a picture of a highly simplified OSI-layers model,wich shows the hierarchy of the different protocols:


^{Illustration1.0.0 ('Leitung' means cable)}

Here the Protocols and their tasks:
FTP:        file transfer processing
SMTP:     Sending mails
POP3:     Get mails
IMAP4:    Get mails
Telnet:    Terminal Emulation
HTTP:      Sending textfiles (websites for example)
TCP:        Basic connection oriented network protocol
UDP:       Basic connectionless network protocol
IP:          Connectionless networkprotcol

1.1.0 Socket-Module
The socket module is included in the standartlibrary like all modules I'll teach you now.

But first a little bit about IPs and Ports:

IP stands for Internet Protocol and looks like this:
'127.0.0.1'
Every part of the IP stands for a byte (so it can be a value between 0 and 255) so one IP is 4 bytes big.

There are two IPs:

• Local IP
• Rooter IP

The local ip is the IP of your computer (I'll show you later how to get it)
The rooter ip is the IP of your rooter (You can check it at http://www.ip-adress.com)

Then you have exactly 65536 ( 99999 ) Ports, but a lot of them are reserved for protocols (for example port 21 for FTP/TCP protocol)

Now you can see it as with Tin can telephones: The IP is your can and the string is a protocol, and every hole for a string is a port. But here you can talk into each Tin cantelephone connection with one can.

1.1.1 Client/Server Relation
There is everytime a server and a client (both sides are server and client)
The Server is the one who provides the data (Evilzone.org for example), and the Client is the one who wants the data to work with it (You for example).

You can connect to a server with its domain (Google.com) or its IP (74.125.79.106).
Try it: Open a new tab and type '74.125.79.106' and press enter. You'll get to google.com.

1.1.2 UDP

The UDP protocol was made as a alternative to the TCP protocol to send human language.
UDP is faster than TCP but if you didn't recieve some bits or bytes you'll never get them, except you load everything again.

Let's make a little script to send a message to a server:

Code: (python) [Select]

import socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

ip = raw_input("Enter Server IP: ")
msg = raw_input("Message: ")

s.sendto(msg,(ip,50000))
s.close()
First is a socket instance is created. You  have to pass it to parameters: the used address-type and the used network portocol. 'AF_INET' stands for Internet/IPv4 and 'SOCK_DGRAM' for the UDP protocol. As the second you say python the ip to connect to and as the third you say python what message should be send.
Then 's.sendto()' sends the message to the given ip.With 's.close()' you end the connection to the server.

But you always need a server too:

Code: (python) [Select]

import socket
s = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)

try:
    s.bind(("",500))
   while True:
        data,address = s.recvfrom(1024)
        print "[%s] %s" % (address[0],data)
finally:
    s.close()
We create a socket instace first here too. In the 'try/finally' the instance is bound to a address ( "" stands for every ip-address) and a Port (here 50000). In the 'while True' loop it waits to recieve data. The Server-script can only be closed by a keyboardinterrupt or by finishing the task.Then it will end the connection in the 'finally' by 's.close()'


To be continued
     

[flowjob]

1.1.3 TCP
TCP (Transmission Control Protocol) isn't a competitive product to UDP, but rather an improvement. For example if you don't recvieve some bits or bytes will ,instead of UDP, TCP send them again. It's more reliable than UDP,but needs more time to send something.

Let's make 2 little scripts.
First the Server:

Code: (python) [Select]

import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("", 50000))
s.listen(1)

try:
    while True:
        connection, address = s.accept()
        while True:
            data = connection.recv(1024)
            if not data:
                connection.close()
                break

             print "[%s] %s" % (address[0], data)
             msg = raw_input("Answer: ")
             connection.send(msg)
finally:
    s.close()
The only difference in the script between TCP and UDP is the second parameter at 'socket.socket()'. After we made the 's' instance we have to bind it to an address and a port again. 'bind' expects a tupel with the ip and the port. "" stands like in the upper scripts for every ip. 'listen' listens for a connection request. 'listen' should have a parameter of a minimum of 1! In the 'while True' loop python waits for a connection request and with 'accept()' we accept the request. The next 'while TRue' loop  has two parts:
'connection.recieve' waits for a message from the Client. If it get's a empty string then the Client ended the connection. If that happens the script will jump back into the first 'while True' loop.
If the message isn't empty it will print and the server can enter a message to send back.

Now try to write the Client script!



Here one solution how the script could look like:

Code: (python) [Select]

import socket
ip = raw_input("IP-Address: ")
s = socket.socket(socket.AF_INET, socket.SOCKSTREAM)
s.connect((ip, 50000)

try:
    while True:
        msg = raw_input("Message: ")
        s.send(msg)
        answer = s.recv(1024)
        print "[%s] %s" % (ip,answer)
finally:
    s.close()
The mostimportant thing in this script is to write FIRST the message-input and message-sending and at SECOND the 'recv()'. Otherwise both sides Server and Client would wait in a loop for a message which would never come!

I won't explain this script now,because it's all like the upper scripts.

The biggest porblem in this chat-application is that you can't write one message after another. This problem could be solved with the 'threading' or the 'Thread' module.

1.1.4 Blocking Sockets

If you create a socket instance it's in the blocking mode. This means that every function call waits till his task is done. The 'recv' funtion for example would block the hole script till a message is recieved. Sometimes this is needed but if you have more than one socket or you use threading then this would be a problem.
To solve this problem you can put a socket in the non-blocking mode:

Code: (python) [Select]

s.setblocking(0)
A parameter with 1 would set the socket in the blocking-mode again.

But:

• 'recv'  and 'recvfrom' print a socket.error-exception when recieving no data
• If the puffer is full then 'send' and 'sendto' will print an error exception and data may won't send.
• 'connect' doesn't wait till the connection is made and 'send' would send a message wich may won't recieved

You can use 'connect_ex()' wich will give you a return-value of 0 if the connection is build.

[flowjob]

1.1.5 socket-commands

Here now the rest of the socket commands of the socket module:

Code: (python) [Select]

socket.getfqdn([name])Will give you the full domainname of the domain 'name'.

Code: (python) [Select]

socket.gethostbyname(hostname)Will give you the IPv4 of the host 'hostname' as string

Code: (python) [Select]

socket.gethostname()Will give you your hostname as a string. With 'socket.gethostbyname(socket.gethostname())' you'll get your IP.

Code: (python) [Select]

socket.getservbyport(port)Will give you the server type of the port 'port'. 'socket.getservbyport(21)' Will print 'ftp'.

Code: (python) [Select]

socket.setdefaulttimeout(timeout)Will set a timeout for the maximum waiting time (to recieve data for example). 'timeout' is in seconds.With socket.getdefaulttimeout() you'll get your set timeout.

With the instance 's':

Code: (python) [Select]

s.getpeername()Will print you the used port and the connected IP

Code: (python) [Select]

s.sendall(msg)Will try to send the message 'msg' till the whole string is recieved at the other side or an exception is thrown.

Code: (python) [Select]

s.setblocking(flag)If 'flag' is 0 then the socketis put into the non-blocking mode. Otherwise it's set into the blocking-mode.




Next part will come soon

[Stackprotector]

Woohoo,   +1  i will check it out soon, thanks !

[Z3R0]

From a programming perspective, yes it is a nice write-up; however, from a networking perspective, it is not. In your OSI-model diagram, layers 5 and 6 are missing, and leitung should just be replaced with, "cable," as not everyone understands German. IP is not technically an independent protocol, but rather a group of various protocols that help with Layer-3 functionality in general. There's also the part where you say this:

Then you have about 99999 Ports, but a lot of them are reserved for protocols (for example port 21 for FTP/TCP protocol)

There are exactly 65536 ports, but port 0 is unusable, similar in the manner that you will never seen an IP address with the 4th octet set as 0. Also, I'm really sorry, but the Tin Can analogy was terrible.

I'm sorry to rip you up like this, but if you want to be taken seriously you really have to be careful about the accuracy of the details you put in your work. I understand English isn't your primary language, but just so you know, what I've talked to you about so far aren't the only misleading objects in your paper. Other than that, it was very nicely organized, and the general topic was very easy to understand the way you wrote it.

[xzid]

@Area_13: I object to your brainfuck sig, its the hello world from wikipedia.

Code: [Select]

++++++++++[>+++++++>++++++++++>+++>+<<<<-]>++.>+.+++++++..+++.>++.<<+++++++++++++++.>.+++.------.--------.>+.>.

Here you go:

Code: [Select]

++++[->++++<]>      16
[
    -
    >++++           64
    >+++++++        112
    >++++++         96
    >+++            48
    <<<<
]
>+.                 65      'A'
>++.                114     'r'
>+++++.             101     'e'
----.               97      'a'
--.                 95      '_'
>+.                 49      '1'
++.                 51      '3'


Code: [Select]

++++[->++++<]>[->++++>+++++++>++++++>+++<<<<]>+.>++.>+++++.----.--.>+.++.

on topic: I like raw sockets, in C.

[flowjob]

@m0rph
I'm sorry about my mistakes and I'll try to have no mistakes in my next part

@xzid
Yes I have the code from wikipedia, 'cause it's a funny language,but i don't really now how to program in BF
And a thanks to the BF code.Will use it right away

[flowjob]

1.1.6 Network-Byte-Order

As you know every OS reads/writes strings in a different byte-order. As long as you use only ASCII chars there will be no problem,but if you want to send a message with special chars like 'ä','{','['  or other there may be a problem if the server/client computer is using a different OS. In order to solve this problem the network-byte-order was defined.
Now you only have to convert the binary from your host-byte-order into the network-byte-order before sending, and when recieving a message from network-byte-order to host-byte-order before printing it to the computer.

You can convert the byte-order in python with the socket module this way:

Code: (python) [Select]

socket.htonl(x)Converts a 32-bit-number from the host-byte-order into the network-byte-order.

Code: (python) [Select]

socket.ntohl(x)Converts a 32-bit-number from the network-byte-order into the host-byte-order.

Code: (python) [Select]

socket.htons(x)Converts a 16-bit-number from the host-byte-order into the network-byte-order.

Code: (python) [Select]

socket.ntohs(x)Converts a 16-bit-number from the network-byte-order into the host-byte-order.

You may ask yourself how to use this with text as it only allows intereger.

Convert the text into interegers:

Code: (python) [Select]

ord('x')Converts the letter 'x' into a decimal intereger.

Code: (python) [Select]

chr(x)Converts the decimal intereger 'x' into a letter.

To convert a whole string into decimal you can write functiones like these:

Code: (python) [Select]

def strtoint(textstring):
    strg = list(textstring)
    dec = []
    for i in range(len(strg)):
        dec.append(ord(str(strg[i])))
    dec = '000'.join(dec)
    return dec

def inttostr(intstring):
    dec = str.split(intstring[::-1],000)       #reverse to split int correctly
    dec = str.split(' '.join(dec)[::-1],' ')     #reverse int back and make it to a list
    strg = []
    for i in range(len(dec)):
        strg.append(chr(dec[i]))
    strg = ''.join(strg)
    return strg


Next part will come soon

[Stackprotector]

Thanks bro ,   but can you write something on threading?

[flowjob]

I'll make one when I have finished the one about network communication (two parts left). 

[Stackprotector]

i mean threading in combination with networking.

[flowjob]

Need no additionally knowledge about any threading module.
The following two parts are about threading-like handling of servers/clients

[flowjob]

1.1.7 Multiplexing Servers - select module

As most servers are used to handle more than one client I'll show you a way how to without threading: Multiplexing servers

You also COULD set all sockets into the non-blocking mode and manage all by hand,but firstly this wouldn't be a very good solution,because blocking sockets have the advantage to wait till an event happens (e.g. recieving a message),and secondly it would need much more main memory.

With the select module you can manage more than one socket in blocking mode without threading.

Here is a Script for a server to recieve and print Messages:

Code: (python) [Select]

import socket
import select

server = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
server.bind(("",50000))
server.listens(5)

clients = []

try:
    while True:
        read,write,oob = select.select([server] + clients,[],[])
        for sock in read:
            if sock is server:
                client,addr = server.accept()
                clients.append(client)
                print '+++ Connected to &s' % addr[0]
            else:
                massage = sock.recv(1024)
                ip = sock.getpeername()[0]
                if message:
                    print '[%s] %s' % (ip,message)
                else:
                    print '+++ Closed connection to %s' % ip
                    sock.close()
                    clients.remove(sock)
finally:
    for c in clients:
        c.close()
    server.close()The list 'clients' will contain all clients. The following try/finally is to close the connection correctly. In the 'while True' loop we create an select instance. 'select.select' needs 4 lists: The first is a list wich contains all socket to try to recieve messages,the second to write messages,the third to recieve out-of-band data (urgent data) amd the last one to set a timeout to wake up python if nothing happend (empty list = no timeout).
I won't explain the remaining script further because you can read the explanation at one of the upper server-scripts.


The client script is the same like the upper client script because multiplex servers have no influence onto the client.

[Stackprotector]

Just suggesting things to write about http://ilab.cs.byu.edu/python/threadingmodule.html

[flowjob]

1.2.0 SocketServer module

As you saw it's really complexe to make an simple server with the socket module.
Because of that there's an easier module to create simple server for chatting for exaple:
The SocketServer module.

As before we'll make an chat script,but this time with the SocketServer module:

Code: (python) [Select]

import SocketServer

class ChatRequestHandler(SocketServer.BaseRequestHandler):
    def handle(self):
    addr = self.client_Address[0]
    print '+++ Connected to %s' % addr
    while True:
        s = self.request.recv(1024)
        if s:
            print '[%s] %s' % (addr,s)
        else:
            print '+++ Closed connection to %s' % addr
            breakAs you see it's much easier to write an chat programm with this module. First we create the class 'ChatRequestHandler' wich is derived from the basic class 'SocketServer.BaseRequestHandler'. Then we have to define the 'handle' function wich python will use to manage recieving messages.
Everytime a connection is made SocketServer will create a new instance with the function 'handle'. In this function the communication with the client is made. Additionally you can create the functions 'setup' and finish wich will be ran before or after the 'handle'.
In our handle a loop reads all the recieved message and print them.If the message is empty the connection will be closed.

Now we are done with the RequestHandler. Now we only have to make an server wich accepts the connection and starts the ChatRequestHandler:

Code: (python) [Select]

server = SocketServer.ThreadingTCPServer(("",50000),ChatRequestHandler)
server.serve_forever()With 'SocketServer.ThreadingTCPServer' we create the true server.
With 'server.serve_forever' python will accept an indefinite number of connections.



There are also some other classes instead of ThreadingTCPServer:

Code: (python) [Select]

SocketServer.TCPServer/SocketServer.UDPServerThis classes can only accept one connection at a time.

Code: (python) [Select]

SocketServer.ThreadingTCPServer / SocketServer.ThreadingUDPServerWith this classes you can make an indefinite number of connections because python does threading with them.

Code: (python) [Select]

SocketServer.ForkingTCPServer / SocketServer.ForkerUDPServerLike the upper one,but here python will handle every connection in a new task

The class SocketServer:

Code: (python) [Select]

s.socketCreates a new socket instance.

Code: (python) [Select]

s.fileno()Prints the file descriptor of the Server-Socket.

Code: (python) [Select]

s.handle_request()Tells the server to accept and handle only one connection request at a time.

Code: (python) [Select]

s.serve_forever()Tells the Server to accpet and handle an indefinite number of connections.

The BaseRequestHandler class

Code: (python) [Select]

rh.requestWith 'request' you can get info of a client.

Code: (python) [Select]

rh.client_addressPrints the IP and the used port for the connection.

Code: (python) [Select]

rh.handle()This function manages the sending/recieving of data

Code: (python) [Select]

rh.setup()This function,if it's used,will be called before 'handle'.It can contain configuration or something else

Code: (python) [Select]

rh.finish()This function,if it's used, is called after 'handle'.It can contain a 'Goodbye' message or something else.

Please give me a feedback if/what you liked/didn't like

I hope my tutorial was helpful