Android Assembly and Disassembly

What started as an initiative to learn about Android development turned into a thought experiment about programming languages and a project to understand how applications are created for Android. Not from the perspective of a Java programmer writing applications in the SDK, but starting from the ground up with the file formats in use. As someone who writes tools to extract data from files in obscure formats, the idea that I am learning about Android this way, and even the Java language, seems quite appropriate.

First of all it's worth noting that other tools for understanding Android file formats already exist. smali has a corresponding disassembler called baksmali, and Androguard is a suite of tools for pulling apart Android packages. In the interest of learning about the file formats in use, I decided to write tools of my own to look at packages. I find that I learn more about something if I have a task where I need to apply my knowledge of it, and in writing software to decode file formats I learn more about those formats than I would by just using existing tools. My tools were also written with a different overall purpose than other tools in this area: I wanted to be able to read existing packages, but I also wanted to create new ones, but not necessarily ones that could include every possible feature allowed by the packaging format.

Inside the Box

Android packages are distributed as APK files with the .apk file name extension. As with many other package formats, these are simply archive files that contain files in a particular directory structure. Like the JAR files used for Java packages, APK files are ZIP archive files. Opening them up with the unzip reveals their structure.

  Length      Date    Time    Name
---------  ---------- -----   ----
     1864  2016-04-28 16:47   classes.dex
     1096  2016-04-28 16:47   resources.arsc
     1620  2016-04-28 16:47   AndroidManifest.xml
    13241  2016-04-28 16:47   res/drawable-hdpi/ic_launcher.png
     4559  2016-04-28 16:47   res/drawable-ldpi/ic_launcher.png
     6953  2016-04-28 16:47   res/drawable-mdpi/ic_launcher.png
      566  2016-04-28 16:47   META-INF/CERT.SF
     1270  2016-04-28 16:47   META-INF/CERT.RSA
      513  2016-04-28 16:47   META-INF/MANIFEST.MF
---------                     -------
    31682                     9 files

The thing that draws our attention is the META-INF directory that the archive contains. This is from Dalvik's Java heritage and its contents describe the files in the archive as well as providing a signature to ensure the integrity of the archive.

To be specific, the MANIFEST.MF text file lists the files and provides an SHA1 digest for each of them so that we can discover if they have been tampered with, the CERT.SF text file provides SHA1 digests for the entries in the MANIFEST.SF file, and the CERT.RSA binary file contains a signature for the CERT.SF file that can be used to verify that the contents of the CERT.SF file are correct and were signed on behalf of a specific person or organisation.

The AndroidManifest.xml and resources.arsc files are binary-encoded XML files that describe the application and its resources. Unlike the MANIFEST.MF file, the AndroidManifest.xml file describes the capabilities of the application rather than the files in the archive. The original file it is derived from is a normal XML file in a traditional Android project. The resources.arsc file describes the resources included in the package's res directory in a structure that presumably makes it easy to reference them.

Finally, the classes.dex file contains the description of the classes supplied by the application and includes references to any parts of the platform API that it uses. The bytecode expressing the application's logic is included in this file, with each sequence of bytecode attached to a method definition.

Creating these files isn't something developers normally have to think too deeply about. Traditionally the Java toolchain takes care of the process of creating them. The resources.arsc file will be created and a source code file containing a set of definitions will be generated so that application code can refer to specific resources. The compiler will create the classes.dex file from the source code, both developer-written and generated. The AndroidManifest.xml file is encoded from the original XML file, using the resources.arsc file as a reference for certain resources, like the application icon. Finally, the files in the META-INF directory are created with the jarsigner tool. Much of this involves ZIP archives: first an unsigned one, then a signed, final package file.

Although it is convenient to install a Java Development Kit (JDK) and the Android SDK, none of the above really requires Java at all as long as we can write files in the appropriate formats. Perhaps the most daunting part – the signing process – can be done completely with OpenSSL.

Let's consider a way of producing APK files without the usual toolchain. To make something like the above package we need to create classes.dex, resources.arsc and AndroidManifest.xml files, put some PNG files in the res directory, sign the package, then put it all in a ZIP archive. The first step is to create the classes.dex file.

Classes and Methods

DEX files are binary files containing a number of different types of information, each grouped into its own section. The file has to be put together in a certain way, with constraints described in the relevant document, but in general terms we can think of each section in a DEX file as a sequence of objects which we could express in Python, with lower level objects declared first and higher level objects later referring to them.

So, types refer to strings, prototypes and fields refer to type, methods refer to types, prototypes and strings, and classes refer to all of the others. Creating the basic structure of the file is straightforward once you know all the types and interfaces you need to declare.

The objects that contain the bytecode to be executed by the virtual machine are the Method objects. Classes merely refer to these and don't contain any bytecode of their own.

The above collections don't contain all the relevant data for the objects they describe, merely references to items that occur in a data section that follows them. Items of the same type are stored together in the data section and, though they may refer to items of a different type, sometimes those other items need to be defined after them. This means that the simplest way to serialise those items – first writing out those that others depend on – isn't always an option. One could imagine a cleaner format that includes all this information in a more accessible way.

Methods and Code

Methods are the key components in the DEX file. Classes defined by the developer provide implementations of them, and those called by the application need to be declared. The instructions used to implement the application's logic are also encoded as bytecode and conceptually stored within the method objects, though really they are encoded in code items and dumped in the data section of the file.

The bytecode itself is fairly simple to understand. Although the summary of the instructions talks about the registers as being general purpose containers of data, you need to use different instructions depending on whether you are manipulating primitive types or objects. Unfortunately, certain instructions need to use the lowest block of 16 registers, so careful management of those blocks is required. Inevitably, you run out of these registers, meaning that values have to be copied into and out of them, requiring more registers to be used and an even greater need to copy values around. Add to this limitations on how methods are called – ideally with registers in the first 16 and less than 6 arguments – and even more work is required to keep track of everything.

This document is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International license.