Linux x64 Calling Convention: Stack Frame

TL; DR
In 64-bit Linux system, function arguments of type integer/pointers are passed to the callee function in the following way:
Arguments 1-6 are passed via registers RDI, RSI, RDX, RCX, R8, R9 respectively;
Arguments 7 and above are pushed on to the stack.
Once inside the callee function:
Arguments 1-6 are accessed via registers RDI, RSI, RDX, RCX, R8, R9 before they are modified or via  offsets from the RBP register like so: rbp - $offset. For example, if the first argument passed to the callee is int (4 bytes) and there are no local variables defined in the function, we could access it via rbp - 0x4; 
It's worth noting, that:
if the 1st argument was 8 bytes (for example, long int), we'd access it via rbp - 0x8;
if the callee function had 1 local variable defined that is smaller or equal to 16 bytes, the first argument of type int would be accessed via rbp - (0x10 + 0x4) or simply rbp - 0x14;
if the callee function had more than 16 bytes reserved for local variables, we'd access the first argument of type int via rbp - 0x24, which suggests that with every 16 bytes worth of local variables defined, the first argument is shifted by 0x10 bytes as shown here;
Argument 7 can be accessed via rbp + 0x10, argument 8 via rbp + 0x18 and so on.
Conclusions listed above are based on the code sample and screenshots provided in the below sections.
Code
This lab and conclusions are based on the following C program compiled on a 64-bit Linux machine:
stack.c
1
#include <stdio.h>
2
​
3
int test(int a, int b, int c, int d, int e, int f, int g, int h, int i)
4
{
5
    //int a2 = 0x555577;
6
    return 1;
7
}
8
​
9
int main(int argc, char *argv[])
10
{
11
    test(0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7, 0x8, 0x9);
12
    return 1;
13
}
14
​
15
// compile with gcc stack.c -o stack
Copied!
How Arguments Are Passed
Let's now see how arguments are passed from a caller to callee.
Below is a screenshot that shows where the 9 arguments 0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7, 0x8, 0x9  passed to the function test(int a, int b, int c, int d, int e, int f, int g, int h, int i) end up in registers and the stack:
Below is a table that complements the above screenshot and shows where arguments live in registers and on the stack and how they get there:
Argument #
Location
Variable
Value
Colour
1
RDI
a
0x1
Red
2
RSI
b
0x2
Red
3
RDX
c
0x3
Red
4
RCX
d
0x4
Red
5
R8
e
0x5
Orange
6
R9
f
0x6
Orange
7
RSP + 0x10
g
0x7
Lime
8
RSP + 0x18
h
0x8
Lime
9
RSP + 0x20
i
0x9
Lime
Same applies to arguments that are memory addresses/pointers.
Stack Inside test()
Below shows how function's test stack frame looks like on a 64-bit platform:
Stack frame x64 inside the function test()
Again, note the following:
Arguments 1 - 6 are moved through the registers edi, esi, edx, ecx, r8d, r9d (orange);
Arguments 7 - 9 are pushed to the stack via push (blue);
Accessing the 1st Argument & Local Variables
Until now, our test() function did not have any local variables defined, so let's see how the stack changes once we have some variables and how we can access them.
If the callee had a local variable defined, such as int a1 = 0x555577 (4 bytes, lime) as in our case shown below (lime), we'd access the first argument not via rbp - 0x4 as it was the case previously when the callee had no local variables, but via rbp - 0x14 (i.e it shifted by 0x10 bytes, red):
First argument (red) is now shifted by 0x10 on the stack and can be accessed via rbp - 0x14
Based on the above case, the test() function stack frame, would now look like this:
64-bit stack frame with 1 local variable defined inside the callee function
Note that the 1st argument, that we previously could access via rbp - 0x4 has been shifted up by 0x10 bytes and is now accessible via rbp - 0x14 whereas the local variable is now at rbp - 0x4 (where the 1st argument was when the function did not have a local variable defined) followed by 0x10 bytes of padding.
Following the same principle as outlined above, if the callee had more than 16 bytes of local variables defined (17 bytes in our case as shown in the below screenshot), we'd now access the first argument via rbp - 0x24 (i.e another 0x10 bytes shift from rbp - 0x14):
First argument is shifted by 0x10 once again and can be accessed via rbp - 0x24
Similarly, if the callee had more than 32 bytes of local variables defined (33 bytes in our case as shown in the below screenshot), we'd now access the first argument via rbp - 0x34 (i.e yet another 0x10 bytes  shift):
First argument is shifted by 0x10 once again and can be accessed via rbp - 0x34
...and so on.
State Inside main()
Below captures program's state once inside main():
RDI and RSI registers inside main() contain argument count and argument values
Note from the above screenshot:
Lime - RDI contains the the count of arguments our program was launched with (argc);
Orange - RSI contains the address to an array of arguments our program was run with (argv[]) and the first one (argv[0]), as expected, is always the full path to the program itself, which is /home/kali/labs/stack/stack in our case.
Also, if we check what's happening higher up at the stack, we will see that it contains the environment variables the program was started with:
Combining all the above knowledge, we can get a general view of the stack layout:
Stack layout for 64-bit program on 64-bit Linux system
References
Applied Reverse Engineering: The Stack - Reverse Engineering
Reverse Engineering
Applied Reverse Engineering: Accelerated Assembly [P1] - Reverse Engineering
Reverse Engineering

Windows x64 Calling Convention: Stack Frame

System Service Descriptor Table - SSDT

Last modified 1mo ago

Copy link

Contents

TL; DR

Code

How Arguments Are Passed

Stack Inside test()

Accessing the 1st Argument & Local Variables

State Inside main()

References