Automatic Compact Alphanumeric Encoding of Shellcode

Abstract

Shellcode is a machine code that is injected in the form of a string to exploit buffer overflows. It generally contains non-ASCII bytes as not all machine instructions have their encoding bytes in ASCII range. To mitigate shellcode injection, there are some filtering techniques that only accepts set of ASCII bytes as an input string. Alphanumeric shellcode helps attackers to bypass this filtering. It is a tedious task to generate alphanumeric shellcode manually from any arbitrary shellcode. There exist tools to automate this process. Existing tools to automatically generate alphanumeric shellcode works as follows : the bytes of shellcode are placed in encoded form and then recovered at runtime using selfmodifying code (also called as decoder). The alphanumeric shellcode generated using such tools has a larger size than original shellcode. The small shellcodes are useful as they fit into the small buffers and are more useful when there is a size restriction on input string. In this work, we present optimization techniques that focuses on optimizing the encoded form as well as decoding loop. One of the technique named as Dynamic Encoding, results into compact size of encodedshellcode, while another technique - Generating Alphanumeric Decoder - focuses onmaking decoding loop more compact. And by combining these two techniques,it produces more compact alphanumeric shellcode (decoder + encoded shellcode)than existing schemes for some shellcode with larger size (greater than 200 bytes).However, the size of output is bigger for some small shellcodes due to larger sizeof decoding loop.