Code obfuscation has become a vital tool to protect, for example, intellectual property against the prying eyes of competitors. Generally speaking, obfuscation makes program code more complex and thus less intelligible.
In our talk, we first give an overview of contemporary code obfuscation schemes. We focus on the design & architecture of virtual machines and discuss the weaknesses of well-established approaches: One being that they mostly rely on fixed instruction set architectures and weak obfuscation of their individual components. A variety of deobfuscation attacks including compiler optimizations, symbolic execution and program synthesis are highly efficient in deobfuscating individual VM components; they even allow us to automate the reconstruction of the underlying code which should be protected by the virtual machine. As a consequence, both academia and industry are currently working towards the next generation of virtual machines, aiming for resilience against such attacks.
We present the core design principles behind such next-gen virtual machines and highlight how they abuse inherent weaknesses of the analysis techniques. Following this, we introduce concrete methods that center around generating target-specific instruction set architectures and intertwined VM components. While some of these methods use theoretic underpinnings to withstand specific attacks, we show that their combination even has beneficial synergy effects.
We conclude the talk by pointing out that such techniques will shape the landscape of modern obfuscation in the next few years; further, we outline required advances in code deobfuscation research to tackle such virtual machines.
