Chameth.com - posts like debugging-beyond-the-debugger, g15-ram-upgrade, shoring-up-sshd, travel-toolkit, why-you-should-be-using-https

Building a travel toolkit

2025-01-02T00:00:00Z

The container for my toolkit, a UGREEN “travel accessories” bag

For a while now, whenever I’ve traveled anywhere I’ve dumped a selection of tools into my backpack just in case I need them. Originally this was mostly focused around being able to open and fix my laptop if anything went wrong, but over time has slowly expanded to include other useful tools.

Repeatedly packing these tools and then returning them to where they belonged was tedious, though. I couldn’t just leave them in the backpack as I often needed them for non-travelling purposes, and I didn’t want to just buy duplicate tools that spent most of their life a few meters away from each other, so I ignored the problem and just dealt with the tedium.

Eventually I acquired enough tools that — while not quite duplicates — I wouldn’t miss often if they were kept in the backpack. Building on these I’ve created a little travel toolkit that contains everything I might want (and more) while away.

The base

The biggest challenge I had before was finding a screwdriver set. I have an iFixit Mako set that did the in-and-out-of-the-bag routine for several years. Nothing else covered the range of drivers I wanted for generic electronic tinkering and more chunky work. I finally found one in the form of a Wera Tool-Check PLUS TR-Edition. “TR” stands for “Tool Rebel” and is Wera’s “fan club” brand. Its signature features over the normal Tool-Check PLUS are a fancy fabric case and a 152mm bit extender; but it also comes with a much different set of bits, which is why I picked it up.

Here’s a comparison

Type	Standard version	Tool Rebel edition
Phillips	2x PH1 3x PH2 1x PH3	1x PH00 1x PH0 1x PH1 1x PH2 1x PH3
Pozidrive	None	1x PZ1 3x PZ2
Slotted	1x 5.5mm	1x 4mm 1x 4.5mm 2x 5.5mm 1x 6.5mm
Torx	1x T10 1x T15 1x T20 1x T25 1x T30 1x T40	1x T10 1x T15 1x T20 1x T25 1x T30 1x T40
Security Torx	1x T10 1x T15 1x T20 1x T25 1x T30	1x T25
Hex	1x H3 1x H4 1x H5 1x H6 1x H8	1x H1.5 1x H2 1x H2.5 1x H3 1x H4 1x H6 1x H8

There are only five bits present in the standard version that are missing in the TR edition: 5mm hex (pretty uncommon) and four sizes of security torx (rare). In exchange for those and the removal of a bunch of duplicates, you get two extra phillips sizes (0 and 00, both very useful for electronics and spectacles), some pozidrive bits (extremely common on random hardware/furniture/etc), smaller hex bits (for electronics, 3D printers, and so on), and a few other bits and pieces.

Overall it’s a vastly superior selection for my needs. And while it doesn’t have the full range of bits provided by the iFixit kit, it does cover almost everything I’m ever actually likely to need while travelling, and it’s probably around a third of the size.

The other core part of the toolkit is the bag. I had a UGREEN “travel accessory” bag hanging around unused, and noticed it was almost the exact right size for the Tool-Check. Now, how many other tools could I stuff in there?

All the tools

The array of tools that make up the travel toolkit

The kit can be broken down into a few sections:

Screwdriving

The aforementioned Wera Tool-Check
Its 152mm bit extender

I’ve talked enough about this already, I think, so I’ll spare you any additional commentary on screwdrivers and bits.

Splitting, gripping and cutting

Engineer tweezers
Engineer mini snips
Engineer mini screw removal pliers
Knipex mini waterpump pliers
Two iFixit spudgers (one metal, one plastic)
iFixit jimmy

I have bigger or slightly different versions of these that stay in my normal toolkit. The tweezers and snips are useful in a variety of situations, and the spudgers are handy for getting inside laptops and other devices. The pliers are less useful, but they’re one of those things where when you do need them there aren’t any good alternatives.

Sewing

Two needles
Some black thread
Set of mini clips

You never know when you’re going to rip a seam in something. I find the clips a lot less fiddly than using pins for keeping material in place when I’m trying to sew.

Computering

Retractable ethernet cable
RJ45 coupler
USB-C to RJ45 adapter
Dual USB-A and USB-C thumb drive, with an Arch ISO on it

Sometimes Wi-Fi just doesn’t cut it, or you need to fiddle with a router that doesn’t expose its admin UI over Wi-Fi. The USB-C adapter is for use with my phone, as it’s often easier to move that get the laptop where it needs to be.

I can boot from the thumb drive into an Arch environment, and from there reinstall my laptop from scratch if needed. There’s enough space left over so it can be used to transfer files as well.

Miscellaneousing

Fabric tape measure
Spirit level
Sharpie
Superglue
Spare batteries: 2 CR2032s, 2 AA, 2 AAA

I’m almost certain I won’t use the spirit level, but it’s tiny and I had a spare so it went in. Everything else is just in the “generally handy to have” category.

Putting it all together

The nicely-organised side of the toolkit

The Tool-Check PLUS fits almost perfectly into one side of the UGREEN bag, leaving a little wiggle room along the side for the longer parts like spudgers. The middle pockets can’t fit much in, but keep the batteries and sewing bits out of the way. Everything else is in a big zip-up compartment in the other side, which is slightly messy but still easy enough to dig through to find whatever I need.

The complete kit weighs in at around 1.2KG, so it’s not exactly lightweight, but it goes in a bag with a heavy laptop and all of its peripherals so it’s not as noticeable as it might otherwise be.

It remains to be seen how much use I’ll actually get out of any of this. Maybe I’ll put a bit of paper in so that I can track how often I use each tool…

Upgrading the RAM in a Dell G15 laptop

2023-07-29T00:00:00Z

The Dell G15

I currently use a Dell G15 laptop for work. It has served me well for a little over a year, but recently it has been struggling a little with my day-to-day workload. It came with 32GB of RAM — the highest possible specification at the time¹ — but that is apparently no longer enough for me.

For a recent project, I was working on a Rust library used in an Android app. That meant running the usual glut of Android tools (Android Studio, an emulator and at least one Gradle daemon) alongside a normal IDE (IntelliJ IDEA). Throw in a web browser and a couple of electron apps, and I often managed to use all 32GB.

When you start swapping memory out to an encrypted disk — even an SSD — it doesn’t make for great performance. At first, I tried to work around this by enabling the Linux out-of-memory (OOM) killer, but it turns out that it’s not too good with Electron apps: it will kill the large browser process, but then the small Electron wrapper will just respawn it.

Can it be upgraded or not?

The obvious solution to not having enough RAM is to add more RAM. A quick look in the manual showed this might not be possible, though. The manual includes the following “Memory specifications” table:

Description	Values
Memory slots	Two SODIMM slots
Memory type	DDR5
Memory speed	4800
Maximum memory configuration	32GB
Minimum memory configuration	8GB
Memory size per slot	8GB or 16GB
Memory configurations supported	8 GB, 1 x 8 GB, DDR5, 4800 MHz 16 GB, 1 x 16 GB, DDR5, 4800 MHz 16 GB, 2 x 8 GB, DDR5, 4800 MHz, dual-channel 32 GB, 2 x 16 GB, DDR5, 4800 MHz, dual-channel

That unambiguously says that an upgrade from 32GB is not possible. I gave up.

Later though, I was complaining about memory issues to a friend, and he pointed out a Dell forum thread where a couple of people claim to have successfully installed dual-channel 32GB modules. Since the alternative was getting an entire new PC after only a year, I decided to give it a go.

The upgrade attempt

I ordered a pair of Crucial 32GB DDR5-4800 SODIMMs, and after they turned up dismantled the laptop. The G15 comes apart pretty normally: there are uncovered screws on the bottom holding the lower part of the case on. With those removed and some gentle prying, it pops off, and you get access to the battery, GPU and motherboard.

The first thing I saw was this:

The G15 motherboard adamantly proclaiming that it doesn’t want 32GB SODIMMs

Not one but two labels that indicate it will only accept 8GB or 16GB modules. Oh well, what’s the worst that can happen?

Oops?

I dutifully installed the new modules, reconnected the battery and put the case back together. I pressed the power button, and… nothing. None of the usual garish lights that immediately come on, no screen output, just a dead laptop. After reading some more of the user manual, I found that there is a tiny status LED on the side next to the ethernet port. Forcing the laptop to power off and back on again, the status LED blinked a distress code at me: 2 amber blinks, 4 white blinks. The manual says that is — unsurprisingly — a memory fault.

I figured at this point that the manual and labels on the motherboard were probably right. I took the laptop apart again, reinstalled the original 2x16GB modules, reassembled it, and pressed the power button. It didn’t boot. I don’t spend a lot of time fiddling inside computers, but I’ve done it enough that I’m reasonably confident I can’t entirely break a computer while swapping some RAM modules. I took to Googling², and found an interesting article that said Dell laptops don’t like to boot after RAM changes unless you clear the CMOS by popping out the battery for 15 minutes.

I opened the laptop up, and looked around for the CMOS battery. There wasn’t one. Turns out they don’t exist any more. I left the main battery disconnected for a while to see if it would help, and it didn’t.

Unexpected success

I started to get worried: if I couldn’t fix this, I wouldn’t be able to work until I got a new PC, and that wasn’t really in my budget at the minute. I sat reading old forum threads and help guides³, none of which were actually useful. Out of nowhere, though, the laptop booted up.

Naturally, I immediately shut the laptop down again, opened it up, and switched back to the new RAM modules. Then I turned it on again and sat waiting. After about 15 minutes of it looking totally dead, it turned on and showed a BIOS warning about the hardware configuration being changed. It then booted perfectly normally, and all 64GB of RAM was visible and usable.

My theory is that the forum threads were right: Dell laptops are funny about RAM upgrades. But somehow in removing the physical CMOS battery, they’ve kept the same “you have to wait 15 minutes” behaviour just without any indication that’s what’s happening. Regardless, I now have enough RAM even for the greediest of IDEs and Electron apps.

Bizarrely, the maximum spec has decreased to 16GB since then. ↩︎
In the generic sense. I use Kagi these days. ↩︎
On my phone because, y’know, the laptop was busted. ↩︎

Debugging beyond the debugger

2019-05-08T00:00:00Z

Real-life debugging tools

Most programming — and sysadmin — problems can be debugged in a fairly straight forward manner using logs, print statements, educated guesses, or an actual debugger. Sometimes, though, the problem is more elusive. There’s a wider box of tricks that can be employed in these cases but I’ve not managed to find a nice overview of them, so here’s mine. I’m mainly focusing on Linux and similar systems, but there tend to be alternatives available for other Operating Systems or VMs if you seek them out.

Networking

tcpdump

tcpdump prints out descriptions of packets on a network interface. You can apply filters to limit which packets are displayed, chose to dump the entire content of the packet, and so forth.

Typical usage might look something like:

# tcpdump -nSi eth0 port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:03:35.577781 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [S], seq 2815779044, win 64800, options [mss 1440,sackOK,TS val 2378811665 ecr 0,nop,wscale 7], length 0
16:03:35.586853 IP6 2001:db8::2.80 > 2001:db8::1.54742: Flags [S.], seq 1522609102, ack 2815779045, win 28560, options [mss 1440,sackOK,TS val 3063610173 ecr 2378811665,nop,wscale 7], length 0
16:03:35.586877 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [.], ack 1522609103, win 507, options [nop,nop,TS val 2378811674 ecr 3063610173], length 0
16:03:35.620678 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [P.], seq 2815779045:2815779399, ack 1522609103, win 507, options [nop,nop,TS val 2378811708 ecr 3063610173], length 354: HTTP: GET / HTTP/1.1

Here you can see the start of a plaintext HTTP request: the three-way handshake as the TCP connection is established followed by a GET request. Even if the data is encrypted as it will be in most cases, it’s often useful to see the “shape” of the transmissions: did the client start sending data when it connected, did the server ever respond, etc.

Daniel Miessler has a good tutorial on tcpdump if you’re not familiar with it and don’t want to jump straight into the man page.

… with Docker

Docker sets up separate network namespaces for each container. To see the traffic across the interfaces of a single container you can nsenter the container’s network namespace:

# nsenter -t $(docker inspect --format '{{.State.Pid}}' my_container) -n tcpdump -nS port 80

This retrieves the PID for the container, and tells nsenter to enter the network (-n) namespace from the given target (-t) PID, and then run the given command (in this case tcpdump ...).

openssl s_client / s_server

When a connection is using TLS it’s often useful to try connecting to the server and see what certificate it presents, algorithms it negotiates, and so forth. OpenSSL offers two useful subcommands which can help with this: s_client for connecting as a client, and s_server for listening to connections.

For example, using s_client to connect to google.com on the standard HTTPS port shows us details about the server cert and its verification status:

$ openssl s_client -connect google.com:443
CONNECTED(00000003)
depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
verify return:1
depth=1 C = US, O = Google Trust Services, CN = Google Internet Authority G3
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com
   i:C = US, O = Google Trust Services, CN = Google Internet Authority G3
 1 s:C = US, O = Google Trust Services, CN = Google Internet Authority G3
   i:OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
---
# ...

Whereas connecting to my webserver and providing an unknown host in the SNI field results in an SSL alert 112 (“The server name sent was not recognized”) and no server certificate is sent:

$ openssl s_client -connect chameth.com:443 -servername example.com
CONNECTED(00000003)
140384831313024:error:14094458:SSL routines:ssl3_read_bytes:tlsv1 unrecognized name:../ssl/record/rec_layer_s3.c:1536:SSL alert number 112
---
no peer certificate available
---
# ...

Often if you hit this kind of alert in an application the exact error will be lost somewhere in the many layers between the SSL library and the logs, so being able to directly connect and test can help diagnose a lot of issues.

Once a connection is established you can read and write plain text and it will be encrypted and decrypted automatically.

Java apps

If a Java app is involved in the connection, you can enable a lot of built-in debugging with a simple JVM property: javax.net.debug. You can tweak what exactly gets logged, but the easiest thing to do is just set the property to all and you’ll see information about certificate chains, verification, and packet dumps:

$ java -Djavax.net.debug=all -jar ....
# ...
found key for : duke
chain [0] = [
[
  Version: V1
  Subject: CN=Duke, OU=Java Software, O="Sun Microsystems, Inc.",
  L=Cupertino, ST=CA, C=US
# ...

More information about Java’s debugging options is available on docs.oracle.com.

Thread and core dumps

Higher-level languages frequently provide an interactive way to dump the current execution state of all of their threads (a “thread dump”). This is useful to spot deadlocks, some types of race conditions, and as a quick and dirty method of investigating hangs or excessive CPU usage.

With both Java and Go applications you can send a QUIT signal to have a thread dump printed out; Go applications will quit after doing so, Java ones will carry on running. At most terminals you can hit Ctrl and \ to send a QUIT signal.

For Java you can also use the jstack tool from the JDK to dump threads by PID; this can be useful if the application is running in the background or has redirected sysout:

$ jstack 8321
Attaching to process ID 8321, please wait...
Debugger attached successfully.
Client compiler detected.

Thread t@5: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=-1107318896 (Interpreted frame)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Interpreted frame)

# ...

A core dump provides more complete information about the state of a process, but is often more complex to interpret. The gcore utility from GDB will create a core dump of a process with a given PID. You can then generally load the core file using your normal debugger, depending on the language in question.

System calls

strace is the swiss army knife for seeing what a process is doing. It details each system call made by a program (you can filter them down, of course). For example:

$ strace -e read curl https://google.com/
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \236\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P!\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200l\2\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20Q\0\0\0\0\0\0"..., 832) = 832
# ...
read(3, "\0\0\0\0\0\0\0\4\25\345\366\302\273sE6\365wI\225\321|\3435Z\362\216\372\215\251aO"..., 253) = 253
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
read(3, "\27\3\3\0!", 5)                = 5
# ...

Brendan Gregg has a nice guide on strace and alternatives.

… with docker

When the application is running in docker you can usually just strace it from the host with the correct PID (from e.g. docker inspect --format '{{.State.Pid}}' my_container). Sometimes you may need to trace the startup of an application though, which is a bit trickier. Instead you can run a new container using the same PID namespace as your target, and the permissions needed to strace:

$ docker run --rm -it --pid=container:my_container \
  --net=container:my_container \
  --cap-add sys_admin \
  --cap-add sys_ptrace \
  alpine

From within the new container you can install strace, and trace any running program within the target container using strace -p as normal. To start a new program you need access to the target container’s filesystem, which you can get to via /proc/1/root (PID 1 being the main process that docker started in the target container).

Files

Sometimes the problem might relate to file access. There are a couple of straight forward — but nonetheless useful — tools which might help here. inotifywait uses the Linux inotify subsystem to watch files or directories for operations. For example:

$ inotifywait -mr site/content
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ OPEN 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ CLOSE_WRITE,CLOSE 2019-05-08-debugging-beyond-the-debugger.md
# ...

Here the -m switch makes inotifywait monitor the files forever (instead of exiting on the first modification, which is the normal behaviour) and r makes it recurse into the directory and monitor each file and subdirectory in there.

If you want to see what processes currently have a file open, fuser is the go-to tool. For example:

$ fuser -v /
                     USER PID ACCESS COMMAND
/:                   root     kernel mount /
                     chris      2961 .rc.. systemd
                     chris      2986 .r... gdm-x-session
                     chris      2994 .r... dbus-daemon
                     chris      3001 .r... gnome-session-b
# ...

Honourable mentions

These aren’t really debugging tools, but I feel it’s worth mentioning as they often feature somewhere along the debugging-of-weird-problems journey.

I’ve seen some weird and wonderful problems happen because a disk is full, so a quick df early on in the debugging process never hurts. Some apps may hang, some may corrupt their config, some may fall over and die; sometimes the manner in which they fail doesn’t obviously point to a disk space issue.

Another issue that comes up now and then — especially inside VMs or other environment that don’t have a decent amount of “noise” happening — is entropy exhaustion. A quick look at /proc/sys/kernel/random/entropy_avail should be enough to confirm that everything is ticking along nicely. If it’s exceedingly low then you may find that anything involving random number generation stalls (TLS connections for example).

Shoring up SSHd configuration

2016-10-18T00:00:00Z

I recently came across a useful tool on GitHub called ssh-audit. It’s a small Python script that connects to an SSH server, gathers a bunch of information, and then highlights any problems it has detected. The problems it reports range from potentially weak algorithms right up to know remote code execution vulnerabilities.

This is the kind of output you get when running ssh-audit. In this particular example, I’m looking at GitHub’s SSH server and have filtered the output to just warnings and failures:

Output of ssh-audit pointing at GitHub’s SSH servers

GitHub’s a bit of a special case, as they’re trying to cope with scores of developers pushing code: they can’t disable weaker algorithms without also stopping lots of people being able to use their service. Still, from the output you can see that ssh-audit has spotted a known vulnerability (CVE-2016-076) and has a lot to say about the various types of supported algorithms.

Background: crypto algorithms used by SSH

Establishing an SSH connection is a moderately complex endeavour, and various parts involve the use of a number of different cryptographic algorithms:

The first such algorithm is the key exchange algorithm. This is the process by which the client and the server agree on a shared key that will be used later. Next comes the host-key algorithm; this is how the server proves its identity to the client. Most SSH users will be familiar with warnings like the following:

$ ssh server.example.com
The authenticity of host 'server.example.com (11.22.33.444)' can't be established.
ED25519 key fingerprint is SHA256:rPVMho1fhEkJqvgce/8iAl353dX5QkGT9F3uCFndsa.
Are you sure you want to continue connecting (yes/no)?

The warning means that the SSH client doesn’t recognise the server’s key, and is asking the user to confirm it. If the key changes later, the SSH client will refuse to connect. In the warning above you can see the algorithm used by the server was ED25519.

Next up is the encryption algorithm, which handles actually encrypting the data sent over the connection. Finally comes the message authentication code algorithm, commonly referred to as ‘mac’. The mac algorithm is effectively responsible for signing each message as a proof that it came from the other party.

Following the recommendations

ssh-audit’s recommendations are pretty easy to follow. It points and shouts at a particular algorithm, and you configure SSHd to not allow it. This is a snippet from my new SSHd config, which gets no complaints from ssh-audit:

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

KexAlgorithms curve25519-sha256@libssh.org
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com

What’s more interesting is the reasoning behind some of the algorithms removed. The ecdh-sha2-nistp series of key exchange algorithms are subject to a sidechannel attack described in a paper in 2014. Some people are also concerned about the involvement of NIST, and the potential for backdoors. Various other key exchange algorithms use too small a number of bits in the key exchange (e.g. diffie-hellman-group1-sha1, which uses 1024). Others still use known-bad hash algorithms (e.g. diffie-hellman-group14-sha1, which uses an acceptable 2048 bit modulus, but relies on SHA1 hashes). ssh-audit only treats the use of SHA1 as a warning, but there’s no compelling reason to keep it around if you’re using remotely modern clients to connect. Similarly the host-key DSA algorithm uses a 1024 bit key, so should be disabled.

Many of the rejected encryption algorithms use basically-broken algorithms (3des-cbc and arcfour for example). Some of the remaining are block ciphers with small block sizes, which makes them weak (e.g. blockfish-cbc uses a block size of 64 bits).

Many of these concerns also apply to mac algorithms (e.g. eliminating hmac-md5, hmac-sha1-etm@openssh.com, etc, as they use weak hash algos). Of particular note, OpenSSH supports the hmac-ripemd160 and hmac-ripemd160-etm@openssh.com algorithms. RIPEMD160 isn’t that common but, like SHA1, is considered to be weak. One other concern with mac algorithms is the order in which the encryption and mac attachment are performed. Encrypt-then-mac is the preferred way of doing it (i.e., the message is encrypted, then a MAC of the ciphertext is attached). The default used in SSH is encrypt-and-mac, where the mac of the plaintext is attached after encryption. Attaching the plaintext mac potentially leaks information (a mac is designed to provide integrity, not confidentiality, after all). The encrypt-then-mac algorithms are indicated by the -etm suffix.

Other changes

In addition to the ssh-audit inspired changes, I took the time to review the rest of my standard SSH configuration. The config touches on a few areas; I’m only going to highlight a couple of them:

  PubkeyAuthentication yes
  RhostsRSAAuthentication no
  HostbasedAuthentication no
  ChallengeResponseAuthentication no
  PasswordAuthentication no

Here all authentication methods other than public key are disabled. A decent key (used in combination with good crypto algorithms!) is drastically harder to brute force than a very good password. It’s also less prone to accidentally being copied into the wrong place, provided to the wrong server, etc.

- UsePrivilegeSeparation yes
+ UsePrivilegeSeparation sandbox

Switching UsePrivilegeSeparation from ‘yes’ to ‘sandbox’ tells OpenSSH to employ kernel sandbox mechanisms on the unprivileged process. This adds another layer of defence in case there’s a severe exploit in OpenSSH itself.

An unexpected side effect

After reconfiguring OpenSSH, all of my servers stopped reporting SSH brute force attempts. Every day prior to the change saw hundreds of connections and, after rate limiting and automatic banning blocked a fair chunk, about two dozen unsuccessful login attempts. With the new algorithm selections in place, there were still hundreds of connections, but no failed login attempts at all. A closer look at the logs showed this:

fatal: Unable to negotiate with 1.2.3.4 port 55025:
  no matching key exchange method found. Their offer:
    diffie-hellman-group14-sha1,
    diffie-hellman-group-exchange-sha1,
    diffie-hellman-group1-sha1

Apparently not a single one of the clients trying to bruteforce their way in supported the one key exchange algorithm I now allow. I guess at some point they’ll be updated with a modern crypto stack, but until then it’s going to be oddly peaceful…

Why you should be using HTTPS

2016-06-17T00:00:00Z

The EFF’s HTTPS Everywhere logo

One of my favourite hobbyhorses recently has been the use of HTTPS, or lack thereof. HTTPS is the thing that makes the little padlock appear in your browser, and has existed for over 20 years. In the past, that little padlock was the exclusive preserve of banks and other ‘high security’ establishments; over time its use has gradually expanded to most (but not all) websites that handle user information, and the time is now right for it to become ubiquitous.

Why use HTTPS?

There are numerous advantages to using HTTPS, both for the users of a website and for the operator:

Privacy

The most obvious advantage is that HTTPS gives your users additional privacy. An insecure (HTTP) request can potentially be read by anyone on the same network, or the network operators, or anyone who happens to operate a network along the path between the user and the server.

Users on shared WiFi networks (such as those in coffee shops, hotels, or offices) are particularly vulnerable to passive sniffing by anyone else on that network. If the network is open (as is frequently the case) then anyone in radio range can see exactly what the user is up to.

Integrity

HTTPS also helps to maintain the integrity of your site. With a plain HTTP request, there’s nothing to stop anyone in between the server and the user from modifying the content of the request or the response. This is a frequent tactic used by annoying WiFi gateways (such as the ones you’d find in a hotel), dubious ISPs who want to serve you extra adverts, or just plain old nefarious attackers.

If you’re trying to convey some kind of information to users (and if you aren’t, why exactly are you running a website again?) it seems beneficial to both you and them if the information arrives as you intended, rather than in a modified form due to someone or something tampering with it.

Security

If your website has any kind of authentication, or session identifiers, it becomes extremely vulnerable to an attacker monitoring the traffic and stealing the credentials. This was starkly demonstrated in 2010 when Firesheep was released. This tool allowed anyone to quickly and automatically hijack social media accounts of anyone on the same network who was using HTTP to access them.

Even if your login pages are served over HTTPS, if you send a single session ID cookie over HTTP (such as a page you decided wasn’t particularly ‘important’) then an attacker can probably spoof the user’s session and gain full access to their account. Again, in the case of open WiFi networks that could be anyone in radio range.

Search engine rankings

Some search engines use HTTPS as a signal in their ranking algorithms. Google announced in 2004 that it was using the presence of HTTPS as a small positive signal, but that it may strengthen that signal over time as more and more websites switch to using a secure transport. It’s not unthinkable that at some point in the future there will be HTTPS-only search engines.

But… But… But…

There are lots of excuses for not implementing HTTPS. Most of them are either misguided or outdated.

It’s too expensive and/or complicated

In the past, getting HTTPS certificates was a pain. A number of free suppliers have existed for a while but the process for getting their certificates wasn’t particularly straight forward, and many imposed arbitrary restrictions on the certificate parameters. Even once you had the certificate, you had to fiddle about with your HTTP server configuration to make it work, remember to manually get a new certificate when the old one expired, and lots of other annoying busywork.

With the arrival of Let’s Encrypt, all that changed. You can retrieve and deploy a free HTTPS certificate with two or three commands. Renewal can be handled completely automatically with a single command executed by cron.

There’s no point; nothing on my site is sensitive

You might not think your content warrants privacy, but can you speak for everyone who accesses it? Even content that seems mundane to you — such as travel advice, or technical writing — could be used to build up a profile of a user. If an attacker is monitoring traffic in a coffee shop and sees a user looking at travel advice and weather forecasts for a foreign country, he could use that information to plan a burglary knowing that the user will be away. Similarly, some content which is perfectly mundane to you may actually be very sensitive in other countries with repressive governments. HTTPS makes it much harder for these people to snoop on traffic.

From another angle, if you’re offering any kind of information, instructions, or especially file downloads, there’s a severe risk to users if the content is modified on its way to them. An evil sysadmin could rewrite your travel advice to suggest visiting the local drug dealer’s hangout, or replace your download with a malware-infested version.

HTTPS is slower, uses more resources, etc

Back in 1995 this might have been a valid argument. Enabling HTTPS on a modern server will make an almost negligible difference to performance. If you also enable HTTP/2 (which most implementations only support over HTTPS), it’s likely to actually use fewer resources, and result in a faster, smoother experience for your users. HTTP/2 was designed to work with HTTPS, and designed with modern requirements and networking techniques in mind.

CloudFlare have an excellent demonstration of the benefits of HTTP/2, and it can show speed improvements of 2-3x in a typical environment. On top of being faster, HTTP/2 uses fewer connections which results in less resource overhead on both the server and the client.

So what are you waiting for?

If you run a website and aren’t using HTTPS, give it a try.