Chameth.com - posts like android-espresso-spoon, shoring-up-sshd but not debugging-beyond-the-debugger, g15-ram-upgrade, why-you-should-be-using-https

Android testing with Espresso and Spoon

2017-05-16T00:00:00Z

I’ve been spending some time recently setting up automated testing for our collection of Android apps and libraries at work. We have a mixture of unit tests, integration tests, and UI tests for most projects, and getting them all to run reliably and automatically has posed some interesting challenges.

Running tests on multiple devices using Spoon

Spoon is a tool developed by Square that handles distributing instrumentation tests to multiple connected devices, aggregating the results, and making reports.

As part of our continuous integration we build both application and test APKs, and these are pushed to the build server as build artefacts. A separate build job then pulls these artefacts down to a Mac Mini we have in the office, and executes Spoon with a few arguments:

java -jar spoon-runner.jar \
    --apk application.apk \
    --test-apk applicationTests.apk \
    --fail-on-failure \
    --fail-if-no-device-connected

Spoon finds all devices, deploys both APKs on them, and then begins the instrumentation tests. We use two physical devices and an emulator to cover the form factors and API versions that are important to us; if any test fails on any of those devices, Spoon will return an error code and the build will fail.

For library projects, you only have a single APK containing both the tests and the library itself. The current version of Spoon requires both --apk and --test-apk to be specified, so we simply pass in the same APK to both. It looks like future versions of Spoon will be more flexible in this regard.

Spoon produces HTML reports, showing the status of each test run on each device. We have the report output folder collected as a build artefact, so the reports can be seen right from the build server:

Spoon output summary, showing results of 171 tests run on 3 devices

Flake-free UI testing with Espresso

Espresso is an Android library that provides an API for interacting with and making assertions about the UI of Android applications. Espresso has a very simple interface, and does lots of clever things under the hood to ensure that your tests only execute code when the UI is idle (and hence stable). You shouldn’t ever need to make your code sleep or wait.

In order for Espresso’s magic to work, it needs to know whenever some background activity is going on that it needs to wait for. By default, it hooks in to Android’s AsyncTask executor so it can wait for those to finish. In our apps, there were a few cases where we used an explicit Thread to do some background work, which caused tests to fail intermittently (depending on whether the thread performed its UI update before or after Espresso executed the test code). Rewriting these cases to use an AsyncTask enabled Espresso to figure out what was happening and the tests started passing reliably.

Another, similar, problem occurred where we were using RxJava to load some data. There are two possible ways to deal with this… Espresso has a concept of an idling resource, which provides a way of telling Espresso when a resource is busy so it can belay interacting with or testing the UI until the resource is finished. In our case, the code in question was going to be rewritten soon, so we went for a quicker and dirtier option: force RxJava to use the same executor as AsyncTask.

To do this, we added a simple test utility class that registers an RxJavaPlugin that overrides the Schedulers used by Rx:

/**
 * Hooks in to the RxJava plugins API to force Rx work to be scheduled on the
 * AsyncTask's thread pool executor. This is a quick and dirty hack to make
 * Espresso aware of when Rx is doing work (and wait for it).
 */
public final class RxSchedulerHook {

    private static final RxJavaSchedulersHook javaHook =
            new RxJavaTestSchedulerHook();

    private RxSchedulerHook() {
        // Should not be insantiated
    }

    public static void registerHooksForTesting() {
        if (RxJavaPlugins.getInstance().getSchedulersHook() != javaHook) {
            RxJavaPlugins.getInstance().reset();
            RxJavaPlugins.getInstance().registerSchedulersHook(javaHook);
        }
    }

    private static class RxJavaTestSchedulerHook extends RxJavaSchedulersHook {
        @Override
        public Scheduler getComputationScheduler() {
            return Schedulers.from(AsyncTask.THREAD_POOL_EXECUTOR);
        }

        @Override
        public Scheduler getIOScheduler() {
            return Schedulers.from(AsyncTask.THREAD_POOL_EXECUTOR);
        }

        @Override
        public Scheduler getNewThreadScheduler() {
            return Schedulers.from(AsyncTask.THREAD_POOL_EXECUTOR);
        }
    }
}

With the hook registered Rx does all of its work on the same thread pool as AsyncTask, which Espresso already knows about. It’s not the best long-term solution, but it means we don’t have to spend time integrating IdlingResource for code that doesn’t have long to live. With the hook in place, the tests that were flaking because of Rx started passing reliably as well.

Getting automatic screenshots of failures

Spoon provides a client library to, among other things, take a screenshot of the device. Espresso provides a hook that can be used to change how errors are handled. Putting the two together is very simple:

final FailureHandler defaultHandler =
        new DefaultFailureHandler(
                InstrumentationRegistry.getTargetContext());

Espresso.setFailureHandler(new FailureHandler() {
    @Override
    public void handle(Throwable throwable, Matcher<View> matcher) {
        try {
            Spoon.screenshot(
                    getActivity(),
                    "espresso-failure",
                    description.getClassName(),
                    description.getMethodName());
        } catch (Exception ex) {
            Log.e(TAG, "Error capturing screenshot", ex);
        }
        defaultHandler.handle(throwable, matcher);
    }
});

In our new error handler we simply ask Spoon to take a screenshot, then call Espresso’s original handler so that it can output its debugging information and fail the test. The Spoon runner automatically picks up the screenshot and adds it to the report:

Spoon output details, showing a screenshot captured of the failure

Having the screenshot, error message and logs all presented in a clean UI makes debugging failures much, much easier than searching through a huge build log to try and find the exception.

Filtering tests based on device capabilities

Some tests won’t work on every device you throw at them. We had two problems: some tests require a higher API version than some of our devices, and some of the UIs under test were designed to run only on certain resolutions.

The main cause for our dependence on newer API versions was the use of WireMock, a brilliant library for stubbing out web services. WireMock requires API 19, while our physical devices tend to run versions older than that. Stopping these tests running is simply a case of applying an annotation to the class:

@SdkSuppress(minSdkVersion=19)

Screen resolution is a bit more complicated. One of our apps is designed for a specific tablet device (and will never be used on anything else), and trying to render the UI on smaller screens results in items overlapping, important parts ending up below the fold, and other problems.

We’d still like all the other tests for that app to run on all of the devices, though, so we can test them on a variety of API versions and in other conditions. We just need to suppress the UI tests. To do this, we subclassed the Android ActivityTestRule and overrode the apply method:

@Override
public Statement apply(Statement base, final Description description) {
    if (!canRunUiTests(InstrumentationRegistry.getContext())) {
        // If we can't run UI tests, then return a statement that does nothing
        // at all.  With normal JUnit tests we'd just throw an assumption
        // failure and the test would be ignored, but that makes the Android
        // runner angry.
        return new Statement() {
            @Override
            public void evaluate() throws Throwable {
                // Do nothing
            }
        };
    }

    return super.apply(base, description);
}

/**
 * Checks that the screen size is close enough to that of our tablet device.
 */
@TargetApi(Build.VERSION_CODES.HONEYCOMB_MR2)
private boolean canRunUiTests(Context context) {
    if (Build.VERSION.SDK_INT < Build.VERSION_CODES.HONEYCOMB_MR2) {
        return false;
    }
    int screenWidth = dpToPx(context,
            context.getResources().getConfiguration().screenWidthDp);
    return screenWidth >= 600;
}

private int dpToPx(Context context, int dp) {
    DisplayMetrics displayMetrics = context.getResources().getDisplayMetrics();
    return Math.round(dp * displayMetrics.density);
}

When the rule is run, we check if the screen width meets a minimum number of pixels. If it doesn’t, an empty Statement is returned that turns the test into a no-op. These show up as passed rather than skipped in the output, but there doesn’t seem to be a nice way to signal to the Android JUnit runner that the test is being ignored programatically.

Shoring up SSHd configuration

2016-10-18T00:00:00Z

I recently came across a useful tool on GitHub called ssh-audit. It’s a small Python script that connects to an SSH server, gathers a bunch of information, and then highlights any problems it has detected. The problems it reports range from potentially weak algorithms right up to know remote code execution vulnerabilities.

This is the kind of output you get when running ssh-audit. In this particular example, I’m looking at GitHub’s SSH server and have filtered the output to just warnings and failures:

Output of ssh-audit pointing at GitHub’s SSH servers

GitHub’s a bit of a special case, as they’re trying to cope with scores of developers pushing code: they can’t disable weaker algorithms without also stopping lots of people being able to use their service. Still, from the output you can see that ssh-audit has spotted a known vulnerability (CVE-2016-076) and has a lot to say about the various types of supported algorithms.

Background: crypto algorithms used by SSH

Establishing an SSH connection is a moderately complex endeavour, and various parts involve the use of a number of different cryptographic algorithms:

The first such algorithm is the key exchange algorithm. This is the process by which the client and the server agree on a shared key that will be used later. Next comes the host-key algorithm; this is how the server proves its identity to the client. Most SSH users will be familiar with warnings like the following:

$ ssh server.example.com
The authenticity of host 'server.example.com (11.22.33.444)' can't be established.
ED25519 key fingerprint is SHA256:rPVMho1fhEkJqvgce/8iAl353dX5QkGT9F3uCFndsa.
Are you sure you want to continue connecting (yes/no)?

The warning means that the SSH client doesn’t recognise the server’s key, and is asking the user to confirm it. If the key changes later, the SSH client will refuse to connect. In the warning above you can see the algorithm used by the server was ED25519.

Next up is the encryption algorithm, which handles actually encrypting the data sent over the connection. Finally comes the message authentication code algorithm, commonly referred to as ‘mac’. The mac algorithm is effectively responsible for signing each message as a proof that it came from the other party.

Following the recommendations

ssh-audit’s recommendations are pretty easy to follow. It points and shouts at a particular algorithm, and you configure SSHd to not allow it. This is a snippet from my new SSHd config, which gets no complaints from ssh-audit:

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

KexAlgorithms curve25519-sha256@libssh.org
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com

What’s more interesting is the reasoning behind some of the algorithms removed. The ecdh-sha2-nistp series of key exchange algorithms are subject to a sidechannel attack described in a paper in 2014. Some people are also concerned about the involvement of NIST, and the potential for backdoors. Various other key exchange algorithms use too small a number of bits in the key exchange (e.g. diffie-hellman-group1-sha1, which uses 1024). Others still use known-bad hash algorithms (e.g. diffie-hellman-group14-sha1, which uses an acceptable 2048 bit modulus, but relies on SHA1 hashes). ssh-audit only treats the use of SHA1 as a warning, but there’s no compelling reason to keep it around if you’re using remotely modern clients to connect. Similarly the host-key DSA algorithm uses a 1024 bit key, so should be disabled.

Many of the rejected encryption algorithms use basically-broken algorithms (3des-cbc and arcfour for example). Some of the remaining are block ciphers with small block sizes, which makes them weak (e.g. blockfish-cbc uses a block size of 64 bits).

Many of these concerns also apply to mac algorithms (e.g. eliminating hmac-md5, hmac-sha1-etm@openssh.com, etc, as they use weak hash algos). Of particular note, OpenSSH supports the hmac-ripemd160 and hmac-ripemd160-etm@openssh.com algorithms. RIPEMD160 isn’t that common but, like SHA1, is considered to be weak. One other concern with mac algorithms is the order in which the encryption and mac attachment are performed. Encrypt-then-mac is the preferred way of doing it (i.e., the message is encrypted, then a MAC of the ciphertext is attached). The default used in SSH is encrypt-and-mac, where the mac of the plaintext is attached after encryption. Attaching the plaintext mac potentially leaks information (a mac is designed to provide integrity, not confidentiality, after all). The encrypt-then-mac algorithms are indicated by the -etm suffix.

Other changes

In addition to the ssh-audit inspired changes, I took the time to review the rest of my standard SSH configuration. The config touches on a few areas; I’m only going to highlight a couple of them:

  PubkeyAuthentication yes
  RhostsRSAAuthentication no
  HostbasedAuthentication no
  ChallengeResponseAuthentication no
  PasswordAuthentication no

Here all authentication methods other than public key are disabled. A decent key (used in combination with good crypto algorithms!) is drastically harder to brute force than a very good password. It’s also less prone to accidentally being copied into the wrong place, provided to the wrong server, etc.

- UsePrivilegeSeparation yes
+ UsePrivilegeSeparation sandbox

Switching UsePrivilegeSeparation from ‘yes’ to ‘sandbox’ tells OpenSSH to employ kernel sandbox mechanisms on the unprivileged process. This adds another layer of defence in case there’s a severe exploit in OpenSSH itself.

An unexpected side effect

After reconfiguring OpenSSH, all of my servers stopped reporting SSH brute force attempts. Every day prior to the change saw hundreds of connections and, after rate limiting and automatic banning blocked a fair chunk, about two dozen unsuccessful login attempts. With the new algorithm selections in place, there were still hundreds of connections, but no failed login attempts at all. A closer look at the logs showed this:

fatal: Unable to negotiate with 1.2.3.4 port 55025:
  no matching key exchange method found. Their offer:
    diffie-hellman-group14-sha1,
    diffie-hellman-group-exchange-sha1,
    diffie-hellman-group1-sha1

Apparently not a single one of the clients trying to bruteforce their way in supported the one key exchange algorithm I now allow. I guess at some point they’ll be updated with a modern crypto stack, but until then it’s going to be oddly peaceful…