Chameth.com - posts like obfuscating-kotlin-proguard, utility-user-stories but not debugging-beyond-the-debugger, why-you-should-be-using-https

On the utility of user stories

2021-10-16T00:00:00Z

User stories are a staple of most agile methodologies. You’d be hard-pressed to find an experienced software developer that’s not come across them at some point in their career. In case you haven’t, they look something like this:

As a frequent customer,
I want to be able to browse my previous orders,
So that I can quickly re-order products.

They provide a persona (in this case “a frequent customer”), a goal (“browse my previous orders”) and a reason (“so that I can quickly re-order products”). This fictitious user story would probably rank among one of the better ones I’ve seen. More typically you end up with something like:

As a user,
I want to be able to login,
So that I can browse while logged in.

This doesn’t really provide a persona or any proper reasoning. It’s just a straight-forward task pretending to be a user story. If this is written in an issue then it provides no extra information over one that simply says “Allow users to login”. In fact, because it’s expressed so awkwardly I’d argue that it’s worse.

This kind of task-disguised-as-a-user-story problem becomes more obvious when people try to write technical tasks in the same way:

As a developer,
I want to refactor the JobFactory,
So that I can work with it more efficiently in the future.

This just says “Refactor the JobFactory”. If you wrote that in a ticket you’d probably feel bad for not describing it very well, but somehow when it’s dressed up as a user story it feels more valuable.

Does the user really want that?

One thing that really irks me about user stories is that it lets you twist your business objectives into sounding like they’re the user’s idea: the story becomes a post-hoc justification for a task you decided was required.

Say you’re making a mobile app for an online book store, and your team for whatever reason has a target of increasing the number of users who view the daily book-of-the-day offer. Maybe you do some interviews and users tell you they forget to check in each day to see what the offer is. How can you funnel more users there?

As a mobile app user,
I want to receive a push notification whenever a new book-of-the-day is available,
So that I have the chance to buy the book.

Some users probably do want such a thing, but I’d argue the vast majority of them do not. Imagine if every app on your phone alerted you whenever there was a new deal, or popped up a dialog whenever you went near a physical shop they had vouchers for…

Obviously in the cold reality of capitalism businesses make money by doing things not strictly in the interests of users¹. When written up like this it becomes so painfully disingenuous, though. Despite writing a user story that starts with the words “As a user”, you’re not really putting yourself in the user’s shoes.

Who even is the user?

One of the big benefits of user stories comes from using personas to describe users. It’s also one of the things that’s rarely actually done, at least in my experience. Looking at Google Trends you can see the rise of searches for “user story” as agile slowly takes over the world, but the movement in searches about personas is very slight:

Graph showing relative search frequency for user stories against user personas

A lot of the time stories are just written with “As a user”, or have some adjectives tacked on to the start (“as a mobile user”, “as a logged-in user”). The best I’ve ever seen in the wild was specifying the class of user like in the example at the start: “as a frequent customer”, or “as a visually impaired user”.

The problem with using these classes is it requires you to come up with them when you’re writing the story. Maybe one day you think about visually impaired users, but the next you don’t. Maybe Bob thinks about certain classes of users, but Alice concentrates on different ones.

The ideal way to solve this is to come up with personas that all the team understand. For example:

Kiera is addicted to books. She reads across genres, and often buys books to gift to her friends and family. She has piles and piles of books to read, but that doesn’t stop her ordering more if she sees a good deal. She likes receiving new books almost as much as reading them, and opts for the fastest delivery available.
Sharon is a slow, methodical reader. She buys one book at a time when she’s close to finishing her current one, and gifts or resells her old books. She almost exclusively reads Science Fiction, and will generally read complete series from start to finish. She’s short-sighted and often struggles when using mobile phones or computers.

You’d probably want 2-4 personas that collectively represent most facets of your userbase. They can be a lot more fleshed out than these - if you search for example user personas you’ll find many beautifully presented examples that have complete backgrounds including hobbies, education levels, and so on. Even with this minimal level of detail, though, I’d argue they’re more useful than just writing things in a standard user story form.

Thinking back to the story about push notifications, writing it to use one of these two personas forces you to think about the trade-offs involved. Kiera probably would like a notification, but it would annoy and possibly confuse Sharon. This then leads you down the path of considering how to accommodate both types of user — maybe adding it as an option, or doing some fancy machine learning, etc. The conversation is now focused around the users, rather than steamrolling over them to reach a business objective.

You’re holding it wrong

The problems I’ve described are not a problem with user stories per se, but rather common issues with how they’re used. But there’s only so many times you can tell people they’re holding a tool wrong before you have to accept that maybe the tool was badly designed.

Considering features from the perspective of multiple personas is the single best thing you can possibly do to ensure you’re providing value to your users. You don’t even need to write things in the typical user story style to benefit from this.

Tacking “As a user,” to the start of all your JIRA tickets isn’t being agile, and isn’t good for users, even if it lets you tick a box somewhere. In some cases this lip-service to users is actively detrimental to them. We should be valuing users and personas over stilted templates and check-box exercises (remind you of anything?).

As a user, I want to pay more money for the things that I buy, so that the company’s CEO can afford to go to space. ↩︎

Obfuscating Kotlin code with ProGuard

2019-10-21T00:00:00Z

Kotlin + Proguard = fun

Obfuscating code is the process of modifying source code or build output in order to make it harder for humans to understand. It’s often employed as a tactic to deter reverse engineering of commercial applications or libraries when you have no choice but to ship binaries or byte code. For Android apps, ProGuard is part of the default toolchain and obfuscation is usually only a config switch away.

I was recently working on an Android library written in Kotlin that my client wanted obfuscated to try and protect some of their trade secrets that were included. Not a problem, I thought: it’s just a few lines of ProGuard config and we’re away. Four hours and lots of hair pulling later I finally got it working…

If at first you don’t succeed…

At first it seemed like ProGuard was refusing to obfuscate any class with a keep rule. With a simple test class:

class Test {

    private val secret = 123
    var name = "Chris"

    fun greet() {
        println("Hi $name! Enter a number: ")
        readLine()?.let { guess(it.toInt()) }
    }

    private fun guess(attempt: Int) = println(
        if (attempt == secret) {
            "Correct!"
        } else {
            "Nope!"
        }
    )

}

And a ProGuard rule of:

-keep public class Test {
    public void greet();
}

I expected that the Test class and the greet method would remain, but both fields and the guess method would be obfuscated. When I built the project and opened the class from Android Studio’s APK inspector I was disappointed:

public final class Test public constructor() {
    public final var name: kotlin.String /* compiled code */

    private final val secret: kotlin.Int /* compiled code */

    public final fun greet(): kotlin.Unit { /* compiled code */ }

    private final fun guess(attempt: kotlin.Int): kotlin.Unit { /* compiled code */ }
}

Having your secret sauce in a field labelled “secret” isn’t exactly the level of obfuscation I was hoping for. ProGuard has lots of knobs that you can twist to affect what it keeps and what it renames, but all the incantations of -keep, -keepmembernames, -allowobfuscation, and so on, that I could come up with either resulted in the class completely vanishing (because it wasn’t kept) or showing up with all its symbols intact.

There are lots of useful Stack Overflow posts describing how to obfuscate a single method, or keep a single method and obfuscate the rest, but nothing I tried seemed to make a difference. I’m not the biggest fan of ProGuard, but I’ve used it enough before to know that it’s not usually that hard to make it submit to your demands. Obviously something else was going on.

… Maybe you’re solving the wrong problem

My next thought was that perhaps Android Studio was doing something clever like reading the ProGuard mapping file and automatically deobfuscating the output for me. Looking at the mapping file it seems that ProGuard has indeed decided to rename some things:

Test -> Test:
    int secret -> a
    java.lang.String name -> b
    void greet() -> greet
    void <init>() -> <init>

The obvious solution is to look at the class file in something less smart than Android Studio. A couple of unzips later and I could do a quick test to see if the original names were still present:

$ strings Test.class | grep secret
secret

The problem is evidently not with Android Studio, as secret shouldn’t end up in the class file at all: it should have been entirely replaced with a like the mapping file says. The output from javap -p doesn’t show any hint of the original names, however:

$ javap -p Test
public final class Test {
  private final int a;
  private java.lang.String b;
  public final void greet();
  public Test();
}

But, given the names show up in strings, they must be kicking around somewhere. None of the various outputs from javap helped until I hit -verbose. Right at the end of the class is:

RuntimeVisibleAnnotations:
  0: #58(#84=[I#2,I#2,I#4],#69=[I#2,I#1,I#3],#81=I#2,#70=[s#40],#71=[s#55,s#39,s#43,s#85,s#39,s#72,s#42,s#91,s#47,s#90,s#39,s#73,s#39,s#74,s#67,s#83])
    kotlin.Metadata(
      mv=[1,1,15]
      bv=[1,0,3]
      k=1
      d1=["\u0000\"\n\u0002\u0018\u0002\n\u0002\u0010\u0000\n\u0002\b\u0002\n\u0002\u0010\u000e\n\u0002\b\u0005\n\u0002\u0010\b\n\u0000\n\u0002\u0010\u0002\n\u0002\b\u0003\u0018\u00002\u00020\u0001B\u0005¢\u0006\u0002\u0010\u0002J\u0006\u0010\u000b\u001a\u00020\fJ\u0010\u0010\r\u001a\u00020\f2\u0006\u0010\u000e\u001a\u00020\nH\u0002R\u001a\u0010\u0003\u001a\u00020\u0004X\u0086\u000e¢\u0006\u000e\n\u0000\u001a\u0004\b\u0005\u0010\u0006\"\u0004\b\u0007\u0010\bR\u000e\u0010\t\u001a\u00020\nX\u0082D¢\u0006\u0002\n\u0000¨\u0006\u000f"]
      d2=["LTest;","","()V","name","","getName","()Ljava/lang/String;","setName","(Ljava/lang/String;)V","secret","","greet","","guess","attempt","lib_release"]
    )

There’s a Kotlin annotation containing all of the symbols we were trying to obfuscate away! Kotlin apparently uses this annotation for reflection and for keeping track of various language features that don’t have a direct mapping in Java bytecode (such as members with internal access). Sure enough, switching back to Android Studio and making it “decompile” the Kotlin code into Java shows the annotation:

@Metadata(
   mv = {1, 1, 15},
   bv = {1, 0, 3},
   k = 1,
   d1 = {"\u0000\"\n\u0002\u0018\u0002\n\u0002\u0010\u0000\n\u0002\b\u0002\n\u0002\u0010\u000e\n\u0002\b\u0005\n\u0002\u0010\b\n\u0000\n\u0002\u0010\u0002\n\u0002\b\u0003\u0018\u00002\u00020\u0001B\u0005¢\u0006\u0002\u0010\u0002J\u0006\u0010\u000b\u001a\u00020\fJ\u0010\u0010\r\u001a\u00020\f2\u0006\u0010\u000e\u001a\u00020\nH\u0002R\u001a\u0010\u0003\u001a\u00020\u0004X\u0086\u000e¢\u0006\u000e\n\u0000\u001a\u0004\b\u0005\u0010\u0006\"\u0004\b\u0007\u0010\bR\u000e\u0010\t\u001a\u00020\nX\u0082D¢\u0006\u0002\n\u0000¨\u0006\u000f"},
   d2 = {"LTest;", "", "()V", "name", "", "getName", "()Ljava/lang/String;", "setName", "(Ljava/lang/String;)V", "secret", "", "greet", "", "guess", "attempt", "lib_release"}
)
public final class Test {

Solving the right problem

Now I had an idea of what was happening I was able to find a couple of other reports of people having the same issue. There’s an open feature request for ProGuard to support Kotlin’s metadata annotation but it’s not yet supported.

As this was a library with a fairly straight forward interface I reasoned I could probably get ProGuard to strip out the metadata annotation. If I stopped Kotlin reflection working in the process then that would actually be a small bonus. Unfortunately other people with the same idea had reported back that they were unsuccessful: even when not using the default ProGuard config, somehow the annotations are kept.

Adding a -printconfiguration instruction to my configuration lets me see the full configuration being passed to ProGuard, and the reason for keeping quickly becomes obvious:

-keepattributes *Annotation*,*Annotation*

This appears to be added by the Android build plugin before it invokes ProGuard, and there’s no obvious way to disable it. ProGuard doesn’t offer a way to reverse this instruction, either, but fortunately the build plugin seems to concatenate all of the -keepattribute values together and puts our user-supplied ones first. Adding a negative filter:

-keepattributes !*Annotation*

Results in the following in the printed configuration:

-keepattributes !*Annotation*,*Annotation*,*Annotation*

The negative filter prevents any subsequent filters from matching. Recompiling and looking at the class file again looks a lot more sensible:

import kotlin.io.ConsoleKt;

public final class Test {
   private final int a = 123;
   private String b = "Chris";

   public final void greet() {
      String var1 = "Hi " + this.b + "! Enter a number: ";
      System.out.println(var1);
      String var10000 = ConsoleKt.readLine();
      if (var10000 != null) {
         var1 = var10000;
         int var3 = Integer.parseInt(var1);
         var1 = var3 == 123 ? "Correct!" : "Nope!";
         System.out.println(var1);
      }
   }
}

And that’s the story of how I spent half a day making a one line change.