Chameth.com - posts like android-espresso-spoon, g15-ram-upgrade, obfuscating-kotlin-proguard but not docker-automatic-nginx-proxy, why-you-should-be-using-https

Upgrading the RAM in a Dell G15 laptop

2023-07-29T00:00:00Z

The Dell G15

I currently use a Dell G15 laptop for work. It has served me well for a little over a year, but recently it has been struggling a little with my day-to-day workload. It came with 32GB of RAM — the highest possible specification at the time¹ — but that is apparently no longer enough for me.

For a recent project, I was working on a Rust library used in an Android app. That meant running the usual glut of Android tools (Android Studio, an emulator and at least one Gradle daemon) alongside a normal IDE (IntelliJ IDEA). Throw in a web browser and a couple of electron apps, and I often managed to use all 32GB.

When you start swapping memory out to an encrypted disk — even an SSD — it doesn’t make for great performance. At first, I tried to work around this by enabling the Linux out-of-memory (OOM) killer, but it turns out that it’s not too good with Electron apps: it will kill the large browser process, but then the small Electron wrapper will just respawn it.

Can it be upgraded or not?

The obvious solution to not having enough RAM is to add more RAM. A quick look in the manual showed this might not be possible, though. The manual includes the following “Memory specifications” table:

Description	Values
Memory slots	Two SODIMM slots
Memory type	DDR5
Memory speed	4800
Maximum memory configuration	32GB
Minimum memory configuration	8GB
Memory size per slot	8GB or 16GB
Memory configurations supported	8 GB, 1 x 8 GB, DDR5, 4800 MHz 16 GB, 1 x 16 GB, DDR5, 4800 MHz 16 GB, 2 x 8 GB, DDR5, 4800 MHz, dual-channel 32 GB, 2 x 16 GB, DDR5, 4800 MHz, dual-channel

That unambiguously says that an upgrade from 32GB is not possible. I gave up.

Later though, I was complaining about memory issues to a friend, and he pointed out a Dell forum thread where a couple of people claim to have successfully installed dual-channel 32GB modules. Since the alternative was getting an entire new PC after only a year, I decided to give it a go.

The upgrade attempt

I ordered a pair of Crucial 32GB DDR5-4800 SODIMMs, and after they turned up dismantled the laptop. The G15 comes apart pretty normally: there are uncovered screws on the bottom holding the lower part of the case on. With those removed and some gentle prying, it pops off, and you get access to the battery, GPU and motherboard.

The first thing I saw was this:

The G15 motherboard adamantly proclaiming that it doesn’t want 32GB SODIMMs

Not one but two labels that indicate it will only accept 8GB or 16GB modules. Oh well, what’s the worst that can happen?

Oops?

I dutifully installed the new modules, reconnected the battery and put the case back together. I pressed the power button, and… nothing. None of the usual garish lights that immediately come on, no screen output, just a dead laptop. After reading some more of the user manual, I found that there is a tiny status LED on the side next to the ethernet port. Forcing the laptop to power off and back on again, the status LED blinked a distress code at me: 2 amber blinks, 4 white blinks. The manual says that is — unsurprisingly — a memory fault.

I figured at this point that the manual and labels on the motherboard were probably right. I took the laptop apart again, reinstalled the original 2x16GB modules, reassembled it, and pressed the power button. It didn’t boot. I don’t spend a lot of time fiddling inside computers, but I’ve done it enough that I’m reasonably confident I can’t entirely break a computer while swapping some RAM modules. I took to Googling², and found an interesting article that said Dell laptops don’t like to boot after RAM changes unless you clear the CMOS by popping out the battery for 15 minutes.

I opened the laptop up, and looked around for the CMOS battery. There wasn’t one. Turns out they don’t exist any more. I left the main battery disconnected for a while to see if it would help, and it didn’t.

Unexpected success

I started to get worried: if I couldn’t fix this, I wouldn’t be able to work until I got a new PC, and that wasn’t really in my budget at the minute. I sat reading old forum threads and help guides³, none of which were actually useful. Out of nowhere, though, the laptop booted up.

Naturally, I immediately shut the laptop down again, opened it up, and switched back to the new RAM modules. Then I turned it on again and sat waiting. After about 15 minutes of it looking totally dead, it turned on and showed a BIOS warning about the hardware configuration being changed. It then booted perfectly normally, and all 64GB of RAM was visible and usable.

My theory is that the forum threads were right: Dell laptops are funny about RAM upgrades. But somehow in removing the physical CMOS battery, they’ve kept the same “you have to wait 15 minutes” behaviour just without any indication that’s what’s happening. Regardless, I now have enough RAM even for the greediest of IDEs and Electron apps.

Bizarrely, the maximum spec has decreased to 16GB since then. ↩︎
In the generic sense. I use Kagi these days. ↩︎
On my phone because, y’know, the laptop was busted. ↩︎

Obfuscating Kotlin code with ProGuard

2019-10-21T00:00:00Z

Kotlin + Proguard = fun

Obfuscating code is the process of modifying source code or build output in order to make it harder for humans to understand. It’s often employed as a tactic to deter reverse engineering of commercial applications or libraries when you have no choice but to ship binaries or byte code. For Android apps, ProGuard is part of the default toolchain and obfuscation is usually only a config switch away.

I was recently working on an Android library written in Kotlin that my client wanted obfuscated to try and protect some of their trade secrets that were included. Not a problem, I thought: it’s just a few lines of ProGuard config and we’re away. Four hours and lots of hair pulling later I finally got it working…

If at first you don’t succeed…

At first it seemed like ProGuard was refusing to obfuscate any class with a keep rule. With a simple test class:

class Test {

    private val secret = 123
    var name = "Chris"

    fun greet() {
        println("Hi $name! Enter a number: ")
        readLine()?.let { guess(it.toInt()) }
    }

    private fun guess(attempt: Int) = println(
        if (attempt == secret) {
            "Correct!"
        } else {
            "Nope!"
        }
    )

}

And a ProGuard rule of:

-keep public class Test {
    public void greet();
}

I expected that the Test class and the greet method would remain, but both fields and the guess method would be obfuscated. When I built the project and opened the class from Android Studio’s APK inspector I was disappointed:

public final class Test public constructor() {
    public final var name: kotlin.String /* compiled code */

    private final val secret: kotlin.Int /* compiled code */

    public final fun greet(): kotlin.Unit { /* compiled code */ }

    private final fun guess(attempt: kotlin.Int): kotlin.Unit { /* compiled code */ }
}

Having your secret sauce in a field labelled “secret” isn’t exactly the level of obfuscation I was hoping for. ProGuard has lots of knobs that you can twist to affect what it keeps and what it renames, but all the incantations of -keep, -keepmembernames, -allowobfuscation, and so on, that I could come up with either resulted in the class completely vanishing (because it wasn’t kept) or showing up with all its symbols intact.

There are lots of useful Stack Overflow posts describing how to obfuscate a single method, or keep a single method and obfuscate the rest, but nothing I tried seemed to make a difference. I’m not the biggest fan of ProGuard, but I’ve used it enough before to know that it’s not usually that hard to make it submit to your demands. Obviously something else was going on.

… Maybe you’re solving the wrong problem

My next thought was that perhaps Android Studio was doing something clever like reading the ProGuard mapping file and automatically deobfuscating the output for me. Looking at the mapping file it seems that ProGuard has indeed decided to rename some things:

Test -> Test:
    int secret -> a
    java.lang.String name -> b
    void greet() -> greet
    void <init>() -> <init>

The obvious solution is to look at the class file in something less smart than Android Studio. A couple of unzips later and I could do a quick test to see if the original names were still present:

$ strings Test.class | grep secret
secret

The problem is evidently not with Android Studio, as secret shouldn’t end up in the class file at all: it should have been entirely replaced with a like the mapping file says. The output from javap -p doesn’t show any hint of the original names, however:

$ javap -p Test
public final class Test {
  private final int a;
  private java.lang.String b;
  public final void greet();
  public Test();
}

But, given the names show up in strings, they must be kicking around somewhere. None of the various outputs from javap helped until I hit -verbose. Right at the end of the class is:

RuntimeVisibleAnnotations:
  0: #58(#84=[I#2,I#2,I#4],#69=[I#2,I#1,I#3],#81=I#2,#70=[s#40],#71=[s#55,s#39,s#43,s#85,s#39,s#72,s#42,s#91,s#47,s#90,s#39,s#73,s#39,s#74,s#67,s#83])
    kotlin.Metadata(
      mv=[1,1,15]
      bv=[1,0,3]
      k=1
      d1=["\u0000\"\n\u0002\u0018\u0002\n\u0002\u0010\u0000\n\u0002\b\u0002\n\u0002\u0010\u000e\n\u0002\b\u0005\n\u0002\u0010\b\n\u0000\n\u0002\u0010\u0002\n\u0002\b\u0003\u0018\u00002\u00020\u0001B\u0005¢\u0006\u0002\u0010\u0002J\u0006\u0010\u000b\u001a\u00020\fJ\u0010\u0010\r\u001a\u00020\f2\u0006\u0010\u000e\u001a\u00020\nH\u0002R\u001a\u0010\u0003\u001a\u00020\u0004X\u0086\u000e¢\u0006\u000e\n\u0000\u001a\u0004\b\u0005\u0010\u0006\"\u0004\b\u0007\u0010\bR\u000e\u0010\t\u001a\u00020\nX\u0082D¢\u0006\u0002\n\u0000¨\u0006\u000f"]
      d2=["LTest;","","()V","name","","getName","()Ljava/lang/String;","setName","(Ljava/lang/String;)V","secret","","greet","","guess","attempt","lib_release"]
    )

There’s a Kotlin annotation containing all of the symbols we were trying to obfuscate away! Kotlin apparently uses this annotation for reflection and for keeping track of various language features that don’t have a direct mapping in Java bytecode (such as members with internal access). Sure enough, switching back to Android Studio and making it “decompile” the Kotlin code into Java shows the annotation:

@Metadata(
   mv = {1, 1, 15},
   bv = {1, 0, 3},
   k = 1,
   d1 = {"\u0000\"\n\u0002\u0018\u0002\n\u0002\u0010\u0000\n\u0002\b\u0002\n\u0002\u0010\u000e\n\u0002\b\u0005\n\u0002\u0010\b\n\u0000\n\u0002\u0010\u0002\n\u0002\b\u0003\u0018\u00002\u00020\u0001B\u0005¢\u0006\u0002\u0010\u0002J\u0006\u0010\u000b\u001a\u00020\fJ\u0010\u0010\r\u001a\u00020\f2\u0006\u0010\u000e\u001a\u00020\nH\u0002R\u001a\u0010\u0003\u001a\u00020\u0004X\u0086\u000e¢\u0006\u000e\n\u0000\u001a\u0004\b\u0005\u0010\u0006\"\u0004\b\u0007\u0010\bR\u000e\u0010\t\u001a\u00020\nX\u0082D¢\u0006\u0002\n\u0000¨\u0006\u000f"},
   d2 = {"LTest;", "", "()V", "name", "", "getName", "()Ljava/lang/String;", "setName", "(Ljava/lang/String;)V", "secret", "", "greet", "", "guess", "attempt", "lib_release"}
)
public final class Test {

Solving the right problem

Now I had an idea of what was happening I was able to find a couple of other reports of people having the same issue. There’s an open feature request for ProGuard to support Kotlin’s metadata annotation but it’s not yet supported.

As this was a library with a fairly straight forward interface I reasoned I could probably get ProGuard to strip out the metadata annotation. If I stopped Kotlin reflection working in the process then that would actually be a small bonus. Unfortunately other people with the same idea had reported back that they were unsuccessful: even when not using the default ProGuard config, somehow the annotations are kept.

Adding a -printconfiguration instruction to my configuration lets me see the full configuration being passed to ProGuard, and the reason for keeping quickly becomes obvious:

-keepattributes *Annotation*,*Annotation*

This appears to be added by the Android build plugin before it invokes ProGuard, and there’s no obvious way to disable it. ProGuard doesn’t offer a way to reverse this instruction, either, but fortunately the build plugin seems to concatenate all of the -keepattribute values together and puts our user-supplied ones first. Adding a negative filter:

-keepattributes !*Annotation*

Results in the following in the printed configuration:

-keepattributes !*Annotation*,*Annotation*,*Annotation*

The negative filter prevents any subsequent filters from matching. Recompiling and looking at the class file again looks a lot more sensible:

import kotlin.io.ConsoleKt;

public final class Test {
   private final int a = 123;
   private String b = "Chris";

   public final void greet() {
      String var1 = "Hi " + this.b + "! Enter a number: ";
      System.out.println(var1);
      String var10000 = ConsoleKt.readLine();
      if (var10000 != null) {
         var1 = var10000;
         int var3 = Integer.parseInt(var1);
         var1 = var3 == 123 ? "Correct!" : "Nope!";
         System.out.println(var1);
      }
   }
}

And that’s the story of how I spent half a day making a one line change.

Android testing with Espresso and Spoon

2017-05-16T00:00:00Z

I’ve been spending some time recently setting up automated testing for our collection of Android apps and libraries at work. We have a mixture of unit tests, integration tests, and UI tests for most projects, and getting them all to run reliably and automatically has posed some interesting challenges.

Running tests on multiple devices using Spoon

Spoon is a tool developed by Square that handles distributing instrumentation tests to multiple connected devices, aggregating the results, and making reports.

As part of our continuous integration we build both application and test APKs, and these are pushed to the build server as build artefacts. A separate build job then pulls these artefacts down to a Mac Mini we have in the office, and executes Spoon with a few arguments:

java -jar spoon-runner.jar \
    --apk application.apk \
    --test-apk applicationTests.apk \
    --fail-on-failure \
    --fail-if-no-device-connected

Spoon finds all devices, deploys both APKs on them, and then begins the instrumentation tests. We use two physical devices and an emulator to cover the form factors and API versions that are important to us; if any test fails on any of those devices, Spoon will return an error code and the build will fail.

For library projects, you only have a single APK containing both the tests and the library itself. The current version of Spoon requires both --apk and --test-apk to be specified, so we simply pass in the same APK to both. It looks like future versions of Spoon will be more flexible in this regard.

Spoon produces HTML reports, showing the status of each test run on each device. We have the report output folder collected as a build artefact, so the reports can be seen right from the build server:

Spoon output summary, showing results of 171 tests run on 3 devices

Flake-free UI testing with Espresso

Espresso is an Android library that provides an API for interacting with and making assertions about the UI of Android applications. Espresso has a very simple interface, and does lots of clever things under the hood to ensure that your tests only execute code when the UI is idle (and hence stable). You shouldn’t ever need to make your code sleep or wait.

In order for Espresso’s magic to work, it needs to know whenever some background activity is going on that it needs to wait for. By default, it hooks in to Android’s AsyncTask executor so it can wait for those to finish. In our apps, there were a few cases where we used an explicit Thread to do some background work, which caused tests to fail intermittently (depending on whether the thread performed its UI update before or after Espresso executed the test code). Rewriting these cases to use an AsyncTask enabled Espresso to figure out what was happening and the tests started passing reliably.

Another, similar, problem occurred where we were using RxJava to load some data. There are two possible ways to deal with this… Espresso has a concept of an idling resource, which provides a way of telling Espresso when a resource is busy so it can belay interacting with or testing the UI until the resource is finished. In our case, the code in question was going to be rewritten soon, so we went for a quicker and dirtier option: force RxJava to use the same executor as AsyncTask.

To do this, we added a simple test utility class that registers an RxJavaPlugin that overrides the Schedulers used by Rx:

/**
 * Hooks in to the RxJava plugins API to force Rx work to be scheduled on the
 * AsyncTask's thread pool executor. This is a quick and dirty hack to make
 * Espresso aware of when Rx is doing work (and wait for it).
 */
public final class RxSchedulerHook {

    private static final RxJavaSchedulersHook javaHook =
            new RxJavaTestSchedulerHook();

    private RxSchedulerHook() {
        // Should not be insantiated
    }

    public static void registerHooksForTesting() {
        if (RxJavaPlugins.getInstance().getSchedulersHook() != javaHook) {
            RxJavaPlugins.getInstance().reset();
            RxJavaPlugins.getInstance().registerSchedulersHook(javaHook);
        }
    }

    private static class RxJavaTestSchedulerHook extends RxJavaSchedulersHook {
        @Override
        public Scheduler getComputationScheduler() {
            return Schedulers.from(AsyncTask.THREAD_POOL_EXECUTOR);
        }

        @Override
        public Scheduler getIOScheduler() {
            return Schedulers.from(AsyncTask.THREAD_POOL_EXECUTOR);
        }

        @Override
        public Scheduler getNewThreadScheduler() {
            return Schedulers.from(AsyncTask.THREAD_POOL_EXECUTOR);
        }
    }
}

With the hook registered Rx does all of its work on the same thread pool as AsyncTask, which Espresso already knows about. It’s not the best long-term solution, but it means we don’t have to spend time integrating IdlingResource for code that doesn’t have long to live. With the hook in place, the tests that were flaking because of Rx started passing reliably as well.

Getting automatic screenshots of failures

Spoon provides a client library to, among other things, take a screenshot of the device. Espresso provides a hook that can be used to change how errors are handled. Putting the two together is very simple:

final FailureHandler defaultHandler =
        new DefaultFailureHandler(
                InstrumentationRegistry.getTargetContext());

Espresso.setFailureHandler(new FailureHandler() {
    @Override
    public void handle(Throwable throwable, Matcher<View> matcher) {
        try {
            Spoon.screenshot(
                    getActivity(),
                    "espresso-failure",
                    description.getClassName(),
                    description.getMethodName());
        } catch (Exception ex) {
            Log.e(TAG, "Error capturing screenshot", ex);
        }
        defaultHandler.handle(throwable, matcher);
    }
});

In our new error handler we simply ask Spoon to take a screenshot, then call Espresso’s original handler so that it can output its debugging information and fail the test. The Spoon runner automatically picks up the screenshot and adds it to the report:

Spoon output details, showing a screenshot captured of the failure

Having the screenshot, error message and logs all presented in a clean UI makes debugging failures much, much easier than searching through a huge build log to try and find the exception.

Filtering tests based on device capabilities

Some tests won’t work on every device you throw at them. We had two problems: some tests require a higher API version than some of our devices, and some of the UIs under test were designed to run only on certain resolutions.

The main cause for our dependence on newer API versions was the use of WireMock, a brilliant library for stubbing out web services. WireMock requires API 19, while our physical devices tend to run versions older than that. Stopping these tests running is simply a case of applying an annotation to the class:

@SdkSuppress(minSdkVersion=19)

Screen resolution is a bit more complicated. One of our apps is designed for a specific tablet device (and will never be used on anything else), and trying to render the UI on smaller screens results in items overlapping, important parts ending up below the fold, and other problems.

We’d still like all the other tests for that app to run on all of the devices, though, so we can test them on a variety of API versions and in other conditions. We just need to suppress the UI tests. To do this, we subclassed the Android ActivityTestRule and overrode the apply method:

@Override
public Statement apply(Statement base, final Description description) {
    if (!canRunUiTests(InstrumentationRegistry.getContext())) {
        // If we can't run UI tests, then return a statement that does nothing
        // at all.  With normal JUnit tests we'd just throw an assumption
        // failure and the test would be ignored, but that makes the Android
        // runner angry.
        return new Statement() {
            @Override
            public void evaluate() throws Throwable {
                // Do nothing
            }
        };
    }

    return super.apply(base, description);
}

/**
 * Checks that the screen size is close enough to that of our tablet device.
 */
@TargetApi(Build.VERSION_CODES.HONEYCOMB_MR2)
private boolean canRunUiTests(Context context) {
    if (Build.VERSION.SDK_INT < Build.VERSION_CODES.HONEYCOMB_MR2) {
        return false;
    }
    int screenWidth = dpToPx(context,
            context.getResources().getConfiguration().screenWidthDp);
    return screenWidth >= 600;
}

private int dpToPx(Context context, int dp) {
    DisplayMetrics displayMetrics = context.getResources().getDisplayMetrics();
    return Math.round(dp * displayMetrics.density);
}

When the rule is run, we check if the screen width meets a minimum number of pixels. If it doesn’t, an empty Statement is returned that turns the test into a no-op. These show up as passed rather than skipped in the output, but there doesn’t seem to be a nice way to signal to the Android JUnit runner that the test is being ignored programatically.