Why strace isn't showing system call being made in shellcode?

Question

I'm new to shellcoding. I have written assembly code :

section .text
global _start

_start:
 jmp end

start:
  ;open file
  pop ebx ; get address of filename
  xor eax,eax
  mov [ebx+3], al
  mov al,5
  xor ecx,ecx
  mov edx,777
  int 80h
  ;exit

  xor eax,eax
  mov al,1
  mov ebx,1
  int 80h

end:
   call start
   db "AAAA"

However when I check either the "sys_open" system call is being made or not using "strace" tool it doesn't show any system call related to file opening.

What is wrong with my shellcode???

"strace" output:

rakesh@rakesh-VirtualBox:~/shellcode$ strace ./a.out 



execve("./a.out", ["./a.out"], [/* 21 vars */]) = 0
brk(0)                                  = 0x6cf000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7c53efe000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=62357, ...}) = 0
mmap(NULL, 62357, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7c53eee000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\30\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1815224, ...}) = 0
mmap(NULL, 3929304, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7c5391e000
mprotect(0x7f7c53ad3000, 2097152, PROT_NONE) = 0
mmap(0x7f7c53cd3000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b5000) = 0x7f7c53cd3000
mmap(0x7f7c53cd9000, 17624, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7c53cd9000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7c53eed000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7c53eec000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7c53eeb000
arch_prctl(ARCH_SET_FS, 0x7f7c53eec700) = 0
mprotect(0x7f7c53cd3000, 16384, PROT_READ) = 0
mprotect(0x600000, 4096, PROT_READ)     = 0
mprotect(0x7f7c53f00000, 4096, PROT_READ) = 0
munmap(0x7f7c53eee000, 62357)           = 0
fstat(1, {st_mode=03260764276, st_size=140733642434881, ...}) = 3
write(1, "\242\350\303\32\377\177\0\0\0\0\0\0\0\0\0\0\252\350\303\32\377\177\0\0\276\350\303\32\377\177\0\0"..., 777 <unfinished ... exit status 1>

Answer 1

I got the solution. Actually by mistake I compiled the shellcode testing program as 64bit executable and that's why the shellcode was being executed as 64bit and in 64bit mode the syscall no 5 is for "fstat" and that's what strace tool was showing.

Answer 2

I'm editing this answer to clean up the confusion about -e open. -e only does filtering and will not add extra info to the strace log. The only case when strace is not logging open syscalls is when a forked subprocess is invoking them and the -f parameter is not set, which is not relevant in your case.

Answer 3

On the amd64 architecture, and even on newer x86 cpus, the system calls happen with the sysenter opcode and not with an int 80h any more. Your strace output clearly shows that you compiled your code into an amd64, despite that you used only x86 instructions and 32-bit registers (I can see that it uses 32+ bit pointers).

The differences are handled with a single-page shared lib, mapped by the kernel into the userspace (libvdso). Your app should only call this libvdso just as normal functions.

The most easy way to find their format if you compile a minimal C code statically and disassemble it (if you don't want to use even the glibc).