short answer, it looks for patterns and common known code that are in programs. For example, things that are produced by MSF, have a similar code structure. The AV knows this, and catches programs that it see's have that code structure.
Long answer, if you really want to know how it works, get something you know is infected, and clean it and scrub it until you find the code that the AV detects, change it around so that it still works but looks different, and by then you will fully understand the general principles.