I want to reverse engineer software
What it means in simple terms is the process of "unbuilding" something to study how it was put together in the first place.
Why reverse engineer?
Answer: Because you can.
It comes down to an issue of power and control. Every computer enthusiast is a control-freak. We love the details. We love being able to figure things out. We love to be able to wrap our heads around a system and be able to predict its every move, and more, be able to direct its every move.
Furthermore, software that you do not have source code to is usually the most interesting kind of software. Sometimes you may be curious as to how a particular security feature works, or if the copy protection is really "unbreakable", and sometimes you just want to know how a particular feature is implemented.
What do you need ?
* Tools for the job (hex editor,debugger,disassembler)
* Information about the target and possible protections
* Basic knowledge like assembly, win32, PE header, import table
Basic terms in Assembly
Registers:
Registers are variables which are stored in your processor. The processor uses these variables for basic mathematical and logical operations. The mostly used registers are: eax, ebx, ecx and edx. Sometimes you will see edi, esi, esp, ebp. There are three types of registers: 32Bit registers, 16Bit registers and 8Bit registers. The 32Bit registers start with e, such as eax. There are 16Bit equivalents of these registers. The only Difference between the two types is the veriable size. These registers are: ax, bx, cx, dx, di, si, sp, bp. There are also 8 bit registers. The 8Bit registers are: al, ah, bl, bh, cl, ch, dl, dh. l - means the lower 8 bits of the 16Bit register. h - means the higher 8 bits of the 16Bit register. Here the l stands for the lower and h for the higher 8 bits of a 16 bit register.
Flags:
Flags are Boolean variables (get 0 or 1 values). Flags are used by the processor for internal logical and mathematical operations, in order to get the result of the operation. The most important flag is the Zero Flag, which can get zero or non-zero (1) values.
Code Flow
When you are analyzing a piece of code, you must understand that the processor is actually quite stupid, and all it does is to simply follow the basic instructions, line by line. It does anything the code tells it to do, and cannot do anything that is not written in the code. This is why you have to think like the processor when you're analyzing a piece of code, and to act like it. You have to do everything the processor does, you have to compare registers and variables, execute jumps and calls, calculate Basic mathematical operations, store and load register values and addresses, and so on... The processor has an instruction pointer especially for this, which is also called IP (it has nothing to do with IP addresses in the Internet Protocol, trust me). Using the instruction pointer, the processor points to the instruction that is about to be executed. The processor also has and executes instructions which change the code flow.These instructions can be function calls, any other routine calls, jumps, conditional jumps, which depend on the zero flag, negative conditional jumps...
Where do i start and what do I learn first?
--->>path 1: the quick way
1 ) google some reversing videos that train you in the RCE basis.
2 ) check http://www.crackmes.de to find some visual solution submitted for crackmes.
3 ) Try to repeat the shown steps, and reproduce them with other crackmes. This will bring you quickly to practice.
4 ) Learn 'on-demand' what you need to know over the time by using info found on the internet.
--->> path 2: the good way
1 ) read tutorials about PE Format, win32 api, AntiDebug, AntiDump, import tables, etc
2 ) Now you know the Win32 basis and then select an Assembler Ebook/tutorial and start learning a bit of assembler.the more, the better.
3 ) learn to work with Ollydbg and other tools
4 ) read tutorials on packers, protectors, and cryptography
5 ) Do the exercises at http://www.binary-auditing.com/ and http://www.crackmes.de
6 ) Look for Lena's reversing for newbies tutorials (They're aimed at beginners and come packaged with some useful tools and test targets. Through following the tuts you'll get a better idea as to exactly what you need to know)
7 ) you can then use google to gather further information for your study.
Knowledge is everything! So read read read and study hard
A few helpful books are Assembly Language Step By Step and Reversing: Secrets of Reverse Engineering.
Here are some good links:
http://www.tuts4you.com/download.php?list.17
http://www.openrce.org/articles/
http://www.crackmes.de/
http://www.reddit.com/r/reverseengineering