Chameth.com - posts like break-everything-fuzz-testing, obfuscating-kotlin-proguard but not debugging-beyond-the-debugger, finding-an-awkward-bug-with-claude-code, why-you-should-be-using-https

How to break everything by fuzz testing

2020-04-26T00:00:00Z

Fuzz testing is a bit like the infinite monkey theorem, but instead of Shakespeare you get crashes.

Fuzz testing, if you’re not aware, is a form of testing that uses procedurally generated random inputs to see how a program behaves. For instance, if you were fuzz testing a web page renderer you might generate a bunch of HTML - some valid, and some not - and make sure the rendering process didn’t unexpectedly crash.

Fuzz testing doesn’t readily lend itself to all types of software, but it particularly shines in cases where some kind of complex user input is accepted and processed in some way - like the aforementioned web page renderer. I was recently adding a library to parse EXIF data to images to an Internet-facing service and realised it was a perfect opportunity to do some fuzz testing. Even if I didn’t find any issues, I’d improve my confidence that the library was safe enough to expose to the Internet.

Breaking my EXIF library

I wrote a quick harness to run go-fuzz on the library, and gave it some pre-existing demo files as sample input. The way go-fuzz works is that it instruments your code and then mutates the inputs to try to improve the coverage. For example, if I had some sample data that had an EXIF tag with a value of 1 then go-fuzz might change it to a 2 and see if the code follows a different path. In most cases it won’t but when it does, they tend to be very interesting cases.

One of the first issues that go-fuzz found was that some values in a maker note field would cause the library to panic (i.e., crash). This happened because there was a check to see if the first six characters were “Nikon” and a null byte, without first checking to see if there were actually six characters available. This is a kind of bug that doesn’t happen much with “real” data - as the field is either not present or completed correctly - but could easily be exploited once this code is exposed to the Internet.

Another interesting bug that go-fuzz found was that if a tag had a particularly large count, the library would try to allocate an obscene amount of memory and die. There was already a check in the code that was meant to avoid this exact scenario, but go-fuzz managed to find a way around it. Each tag has a size (for example an integer tag takes a fixed number of bytes) and a count; the existing check multiplied the two together and made sure that the result wasn’t too large. For most cases this was fine but go-fuzz found a case where the count was so large that when multiplied by the size of the tag it overflowed the integer and became negative, thus passing the sanity check but then subsequently failing when it came around to actually allocating the memory.

The final bug of note that go-fuzz found was the most interesting. EXIF data is stored in IFDs (“Image File Directories”), and each IFD provides what is effectively a pointer (a byte offset) to the next one. The EXIF library already had a check to make sure that these didn’t loop, but it only checked the immediately preceding IFD - so if IFD 1 linked to IFD 2, it would catch IFD 2 linking back to IFD 1 and break the loop with an error. Go-fuzz found that having three interlinked IFDs had the same issue, though, and the guard code wasn’t triggered. This created an infinite loop, maxing out a CPU core until the process was eventually killed - one of the worst kind of bugs you could have in an Internet-facing service which doesn’t deal with private data! The fix for this was fairly straightforward - I just made the library keep a record of the previously visited IFDs and bail out if it found a loop.

Breaking my IDE

When go-fuzz detects an issue it outputs not only the details of the problem (the stack trace, error message, and so forth) but also the input that generated the problem. This is useful for reproducing and making sure the issue is fixed, but it also makes it really easy to write a test to ensure that the behaviour never regresses in the future.

As I was working through fixing the bugs that go-fuzz found, I dutifully added new tests where needed. After adding the sample input with looping IFDs to the project, I switched to IDEA to write a test to use it. I clicked on the input file to copy the file name, and then the entire IDE hung and had to be restarted. Uh oh! When I restarted IDEA, it immediately began indexing the project and again hung. It turns out IDEA parses EXIF data (presumably, even if it does nothing else with the data, to get the rotation property for images), and the library they use - an independent one written on Java - had the same bug as the Go library I was using.

In order to stop IDEA from indexing the file and becoming unusable I renamed it from a ‘.tif’ extension to ‘.dat’, and everything went back to normal. I thought I’d best report the bug to JetBrains, though, so they could put a proper fix in.

Breaking YouTrack

JetBrains use their own issue tracker called YouTrack for reporting bugs in IDEA. I dutifully went over and described the problem, attaching the log files from the IDE, a description of how the file was malformed, and carefully selected the .dat version of the file to upload so that it wouldn’t cause anyone else the same immediate problem.

After trying to upload the file I got a strange error back. Uh oh! I submitted the IDEA issue as it stood, unable to see if the attachments had even uploaded, and went and wrote up an issue for YouTrack itself about the error message. While I was doing that, YouTrack seemed to slow down and become really annoying to use. I had a sinking feeling the exact same thing was probably happening as with IDEA and my library - but this time YouTrack had content-sniffed the file instead of relying on the file extension. In hindsight, I should’ve put the file in a passworded archive to ensure no automated tools got hold of it. I marked the issue as a security problem as in a service like YouTrack it presents a denial-of-service opportunity¹ (remember when I said it was one of the worst kinds of bugs you could have in an Internet-facing service?…)

Shortly after I raised my YouTrack ticket, a notice appeared at the top of the page saying they were investigating the current performance issues. Uh oh! I was holding out hope that this was unrelated to me uploading the buggy dat file, but the timing all seemed a bit suspect. I shot support an e-mail saying I think I might be the root cause for their performance issues and linked to the ticket. In the time it took me to e-mail them, the entire site had been put into maintenance mode. I got an e-mail back a few hours later confirming the outage was in fact all my fault, as I’d feared. Within the space of days the JetBrains security team had fixed the issue in YouTrack, which was a pretty nice turnaround.

So if you were trying to access YouTrack at the start of March and couldn’t - I’m sorry, I didn’t mean to! Also, if you’re building an Internet-facing service that takes user input you should really consider running a fuzz tester against it!

“We have a problem”. “Remember, there are no such things as problems, only opportunities”. “Well then we have a DDoS opportunity.” – @J4vv4d ↩︎

Obfuscating Kotlin code with ProGuard

2019-10-21T00:00:00Z

Kotlin + Proguard = fun

Obfuscating code is the process of modifying source code or build output in order to make it harder for humans to understand. It’s often employed as a tactic to deter reverse engineering of commercial applications or libraries when you have no choice but to ship binaries or byte code. For Android apps, ProGuard is part of the default toolchain and obfuscation is usually only a config switch away.

I was recently working on an Android library written in Kotlin that my client wanted obfuscated to try and protect some of their trade secrets that were included. Not a problem, I thought: it’s just a few lines of ProGuard config and we’re away. Four hours and lots of hair pulling later I finally got it working…

If at first you don’t succeed…

At first it seemed like ProGuard was refusing to obfuscate any class with a keep rule. With a simple test class:

class Test {

    private val secret = 123
    var name = "Chris"

    fun greet() {
        println("Hi $name! Enter a number: ")
        readLine()?.let { guess(it.toInt()) }
    }

    private fun guess(attempt: Int) = println(
        if (attempt == secret) {
            "Correct!"
        } else {
            "Nope!"
        }
    )

}

And a ProGuard rule of:

-keep public class Test {
    public void greet();
}

I expected that the Test class and the greet method would remain, but both fields and the guess method would be obfuscated. When I built the project and opened the class from Android Studio’s APK inspector I was disappointed:

public final class Test public constructor() {
    public final var name: kotlin.String /* compiled code */

    private final val secret: kotlin.Int /* compiled code */

    public final fun greet(): kotlin.Unit { /* compiled code */ }

    private final fun guess(attempt: kotlin.Int): kotlin.Unit { /* compiled code */ }
}

Having your secret sauce in a field labelled “secret” isn’t exactly the level of obfuscation I was hoping for. ProGuard has lots of knobs that you can twist to affect what it keeps and what it renames, but all the incantations of -keep, -keepmembernames, -allowobfuscation, and so on, that I could come up with either resulted in the class completely vanishing (because it wasn’t kept) or showing up with all its symbols intact.

There are lots of useful Stack Overflow posts describing how to obfuscate a single method, or keep a single method and obfuscate the rest, but nothing I tried seemed to make a difference. I’m not the biggest fan of ProGuard, but I’ve used it enough before to know that it’s not usually that hard to make it submit to your demands. Obviously something else was going on.

… Maybe you’re solving the wrong problem

My next thought was that perhaps Android Studio was doing something clever like reading the ProGuard mapping file and automatically deobfuscating the output for me. Looking at the mapping file it seems that ProGuard has indeed decided to rename some things:

Test -> Test:
    int secret -> a
    java.lang.String name -> b
    void greet() -> greet
    void <init>() -> <init>

The obvious solution is to look at the class file in something less smart than Android Studio. A couple of unzips later and I could do a quick test to see if the original names were still present:

$ strings Test.class | grep secret
secret

The problem is evidently not with Android Studio, as secret shouldn’t end up in the class file at all: it should have been entirely replaced with a like the mapping file says. The output from javap -p doesn’t show any hint of the original names, however:

$ javap -p Test
public final class Test {
  private final int a;
  private java.lang.String b;
  public final void greet();
  public Test();
}

But, given the names show up in strings, they must be kicking around somewhere. None of the various outputs from javap helped until I hit -verbose. Right at the end of the class is:

RuntimeVisibleAnnotations:
  0: #58(#84=[I#2,I#2,I#4],#69=[I#2,I#1,I#3],#81=I#2,#70=[s#40],#71=[s#55,s#39,s#43,s#85,s#39,s#72,s#42,s#91,s#47,s#90,s#39,s#73,s#39,s#74,s#67,s#83])
    kotlin.Metadata(
      mv=[1,1,15]
      bv=[1,0,3]
      k=1
      d1=["\u0000\"\n\u0002\u0018\u0002\n\u0002\u0010\u0000\n\u0002\b\u0002\n\u0002\u0010\u000e\n\u0002\b\u0005\n\u0002\u0010\b\n\u0000\n\u0002\u0010\u0002\n\u0002\b\u0003\u0018\u00002\u00020\u0001B\u0005¢\u0006\u0002\u0010\u0002J\u0006\u0010\u000b\u001a\u00020\fJ\u0010\u0010\r\u001a\u00020\f2\u0006\u0010\u000e\u001a\u00020\nH\u0002R\u001a\u0010\u0003\u001a\u00020\u0004X\u0086\u000e¢\u0006\u000e\n\u0000\u001a\u0004\b\u0005\u0010\u0006\"\u0004\b\u0007\u0010\bR\u000e\u0010\t\u001a\u00020\nX\u0082D¢\u0006\u0002\n\u0000¨\u0006\u000f"]
      d2=["LTest;","","()V","name","","getName","()Ljava/lang/String;","setName","(Ljava/lang/String;)V","secret","","greet","","guess","attempt","lib_release"]
    )

There’s a Kotlin annotation containing all of the symbols we were trying to obfuscate away! Kotlin apparently uses this annotation for reflection and for keeping track of various language features that don’t have a direct mapping in Java bytecode (such as members with internal access). Sure enough, switching back to Android Studio and making it “decompile” the Kotlin code into Java shows the annotation:

@Metadata(
   mv = {1, 1, 15},
   bv = {1, 0, 3},
   k = 1,
   d1 = {"\u0000\"\n\u0002\u0018\u0002\n\u0002\u0010\u0000\n\u0002\b\u0002\n\u0002\u0010\u000e\n\u0002\b\u0005\n\u0002\u0010\b\n\u0000\n\u0002\u0010\u0002\n\u0002\b\u0003\u0018\u00002\u00020\u0001B\u0005¢\u0006\u0002\u0010\u0002J\u0006\u0010\u000b\u001a\u00020\fJ\u0010\u0010\r\u001a\u00020\f2\u0006\u0010\u000e\u001a\u00020\nH\u0002R\u001a\u0010\u0003\u001a\u00020\u0004X\u0086\u000e¢\u0006\u000e\n\u0000\u001a\u0004\b\u0005\u0010\u0006\"\u0004\b\u0007\u0010\bR\u000e\u0010\t\u001a\u00020\nX\u0082D¢\u0006\u0002\n\u0000¨\u0006\u000f"},
   d2 = {"LTest;", "", "()V", "name", "", "getName", "()Ljava/lang/String;", "setName", "(Ljava/lang/String;)V", "secret", "", "greet", "", "guess", "attempt", "lib_release"}
)
public final class Test {

Solving the right problem

Now I had an idea of what was happening I was able to find a couple of other reports of people having the same issue. There’s an open feature request for ProGuard to support Kotlin’s metadata annotation but it’s not yet supported.

As this was a library with a fairly straight forward interface I reasoned I could probably get ProGuard to strip out the metadata annotation. If I stopped Kotlin reflection working in the process then that would actually be a small bonus. Unfortunately other people with the same idea had reported back that they were unsuccessful: even when not using the default ProGuard config, somehow the annotations are kept.

Adding a -printconfiguration instruction to my configuration lets me see the full configuration being passed to ProGuard, and the reason for keeping quickly becomes obvious:

-keepattributes *Annotation*,*Annotation*

This appears to be added by the Android build plugin before it invokes ProGuard, and there’s no obvious way to disable it. ProGuard doesn’t offer a way to reverse this instruction, either, but fortunately the build plugin seems to concatenate all of the -keepattribute values together and puts our user-supplied ones first. Adding a negative filter:

-keepattributes !*Annotation*

Results in the following in the printed configuration:

-keepattributes !*Annotation*,*Annotation*,*Annotation*

The negative filter prevents any subsequent filters from matching. Recompiling and looking at the class file again looks a lot more sensible:

import kotlin.io.ConsoleKt;

public final class Test {
   private final int a = 123;
   private String b = "Chris";

   public final void greet() {
      String var1 = "Hi " + this.b + "! Enter a number: ";
      System.out.println(var1);
      String var10000 = ConsoleKt.readLine();
      if (var10000 != null) {
         var1 = var10000;
         int var3 = Integer.parseInt(var1);
         var1 = var3 == 123 ? "Correct!" : "Nope!";
         System.out.println(var1);
      }
   }
}

And that’s the story of how I spent half a day making a one line change.