Wiki: Functions

WARNING: Wiki content is an archive, no promise about quality!

Please choose a tutorial page:

The previous section about the stack has shown how to call a standard function with parameters. This section will go over some other "calling conventions" besides the standard.

A "calling convention" is the way in which a function is called. The standard convention, __cdecl, is what has been used up until now. Some other common ones are __stdcall, __fastcall, and __thiscall.

A less common declaration used when writing hacks is __declspec(naked).

__cdecl

__cdecl is the default calling convention on most C compilers. The properties are as follows:

  • The caller places all the parameters on the stack
  • The caller removes the parameters from the stack (often by adding the total size added to the stack pointer)

Throughout previous sections, __cdecl has been the calling convention used. However, here is an example to help illustrate it:

  push param3
  push param2
  push param1   ; Parameters are pushed onto the stack
  call func     ; The function is called
  add esp, 0Ch  ; Parameters are removed from the stack
  ...
 func:
  ...
  ret

__stdcall

__stdcall is another common calling convention. The properties of __stdcall are:

  • The caller places parameters on the stack
  • The called function removes the parameters from the stack, often by using the return instruction with a parameter equal to the number of parameters, "ret xx"

Here's an example of a __stdcall function (note that if no parameters are passed, __stdcall is indistinguishable from __cdecl.

  push param3
  push param2
  push param1   ; Parameters are pushed onto the stack
  call func     ; The function is called
  ... 
 func:
  ...
  ret 0c;       ; The function cleans up the stack

The most useful part about __stdcall is that it tells a reverse engineer how many parameters are passed to any given function. In cases where no examples of the function being called may be found (possibly because it's an exported .dll function), it is easier to check the return than to enumerate local variables (of course, IDA looks after that automatically if that's an option).

__fastcall

__fastcall is the final common calling convention seen. All implementations of __fastcall pass parameters in registers, although Microsoft and Borland, for example, use different registers. Here are the properties of Microsoft's __fastcall implementation:

  • First two parameters are passed in ecx and edx, respectively
  • Third parameter and on are passed on the stack, as usual
  • Functions clean up their own stack, if necessary

Recognizing a __fastcall function is easy: look for ecx and edx being used without being initialized in a function.

A __fastcall with no parameters is identical to __cdecl and __stdcall with no parameters, and a __fastcall with a single parameter looks like __thiscall.

Here are some __fastcall examples:

  mov ecx, 7
  call func
  ...
 func:
  ...
  ret

  mov ecx, 7
  mov edx, 8
  call func
  ...
 func:
  ...
  ret

  mov ecx, 7
  mov edx, 8
  push param4
  push param3
  call func
  ...
 func:
  ...
  ret 8       ; Note that the function cleans up the stack.

__thiscall

Seen only in object-oriented programming, __thiscall is very similar to __stdcall, except that a pointer to the class whose member is being called is passed in ecx.

  • ecx is assigned a pointer to the class whose member is being called
  • The parameters are placed on the stack, the same as ''__stdcall'
  • The function cleans itself up, the same as __stdcall

Here is an example of __thiscall:

  push param3
  push param2
  push param1
  mov ecx, this
  call func
  ...
 func:
  ...
  ret 12

__declspec(naked)

__declspec(naked), a Visual Studio-specific convention, can't really be identified in assembly, since it's identical to __cdecl once it reaches assembly. However, the special property of this convention is that the compiler will generate no code in a function. This allows the program, in a __asm{} block, to write everything from preserving registers to allocating local variables and returning. This is useful when patching a jump in the middle of code, since it prevents the function from changing registers without the programmer's knowledge.

This C function:

void __declspec(naked) test()
{
}

Would translate to this in assembly:

Since no code is generated.

Questions

Feel free to edit this section and post questions, I'll do my best to answer them. But you may need to contact me to let me know that a question exists.