Chameth.com - posts like debugging-beyond-the-debugger, offline-gnupg-master-yubikey-subkeys

Debugging beyond the debugger

2019-05-08T00:00:00Z

Real-life debugging tools

Most programming — and sysadmin — problems can be debugged in a fairly straight forward manner using logs, print statements, educated guesses, or an actual debugger. Sometimes, though, the problem is more elusive. There’s a wider box of tricks that can be employed in these cases but I’ve not managed to find a nice overview of them, so here’s mine. I’m mainly focusing on Linux and similar systems, but there tend to be alternatives available for other Operating Systems or VMs if you seek them out.

Networking

tcpdump

tcpdump prints out descriptions of packets on a network interface. You can apply filters to limit which packets are displayed, chose to dump the entire content of the packet, and so forth.

Typical usage might look something like:

# tcpdump -nSi eth0 port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:03:35.577781 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [S], seq 2815779044, win 64800, options [mss 1440,sackOK,TS val 2378811665 ecr 0,nop,wscale 7], length 0
16:03:35.586853 IP6 2001:db8::2.80 > 2001:db8::1.54742: Flags [S.], seq 1522609102, ack 2815779045, win 28560, options [mss 1440,sackOK,TS val 3063610173 ecr 2378811665,nop,wscale 7], length 0
16:03:35.586877 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [.], ack 1522609103, win 507, options [nop,nop,TS val 2378811674 ecr 3063610173], length 0
16:03:35.620678 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [P.], seq 2815779045:2815779399, ack 1522609103, win 507, options [nop,nop,TS val 2378811708 ecr 3063610173], length 354: HTTP: GET / HTTP/1.1

Here you can see the start of a plaintext HTTP request: the three-way handshake as the TCP connection is established followed by a GET request. Even if the data is encrypted as it will be in most cases, it’s often useful to see the “shape” of the transmissions: did the client start sending data when it connected, did the server ever respond, etc.

Daniel Miessler has a good tutorial on tcpdump if you’re not familiar with it and don’t want to jump straight into the man page.

… with Docker

Docker sets up separate network namespaces for each container. To see the traffic across the interfaces of a single container you can nsenter the container’s network namespace:

# nsenter -t $(docker inspect --format '{{.State.Pid}}' my_container) -n tcpdump -nS port 80

This retrieves the PID for the container, and tells nsenter to enter the network (-n) namespace from the given target (-t) PID, and then run the given command (in this case tcpdump ...).

openssl s_client / s_server

When a connection is using TLS it’s often useful to try connecting to the server and see what certificate it presents, algorithms it negotiates, and so forth. OpenSSL offers two useful subcommands which can help with this: s_client for connecting as a client, and s_server for listening to connections.

For example, using s_client to connect to google.com on the standard HTTPS port shows us details about the server cert and its verification status:

$ openssl s_client -connect google.com:443
CONNECTED(00000003)
depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
verify return:1
depth=1 C = US, O = Google Trust Services, CN = Google Internet Authority G3
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com
   i:C = US, O = Google Trust Services, CN = Google Internet Authority G3
 1 s:C = US, O = Google Trust Services, CN = Google Internet Authority G3
   i:OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
---
# ...

Whereas connecting to my webserver and providing an unknown host in the SNI field results in an SSL alert 112 (“The server name sent was not recognized”) and no server certificate is sent:

$ openssl s_client -connect chameth.com:443 -servername example.com
CONNECTED(00000003)
140384831313024:error:14094458:SSL routines:ssl3_read_bytes:tlsv1 unrecognized name:../ssl/record/rec_layer_s3.c:1536:SSL alert number 112
---
no peer certificate available
---
# ...

Often if you hit this kind of alert in an application the exact error will be lost somewhere in the many layers between the SSL library and the logs, so being able to directly connect and test can help diagnose a lot of issues.

Once a connection is established you can read and write plain text and it will be encrypted and decrypted automatically.

Java apps

If a Java app is involved in the connection, you can enable a lot of built-in debugging with a simple JVM property: javax.net.debug. You can tweak what exactly gets logged, but the easiest thing to do is just set the property to all and you’ll see information about certificate chains, verification, and packet dumps:

$ java -Djavax.net.debug=all -jar ....
# ...
found key for : duke
chain [0] = [
[
  Version: V1
  Subject: CN=Duke, OU=Java Software, O="Sun Microsystems, Inc.",
  L=Cupertino, ST=CA, C=US
# ...

More information about Java’s debugging options is available on docs.oracle.com.

Thread and core dumps

Higher-level languages frequently provide an interactive way to dump the current execution state of all of their threads (a “thread dump”). This is useful to spot deadlocks, some types of race conditions, and as a quick and dirty method of investigating hangs or excessive CPU usage.

With both Java and Go applications you can send a QUIT signal to have a thread dump printed out; Go applications will quit after doing so, Java ones will carry on running. At most terminals you can hit Ctrl and \ to send a QUIT signal.

For Java you can also use the jstack tool from the JDK to dump threads by PID; this can be useful if the application is running in the background or has redirected sysout:

$ jstack 8321
Attaching to process ID 8321, please wait...
Debugger attached successfully.
Client compiler detected.

Thread t@5: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=-1107318896 (Interpreted frame)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Interpreted frame)

# ...

A core dump provides more complete information about the state of a process, but is often more complex to interpret. The gcore utility from GDB will create a core dump of a process with a given PID. You can then generally load the core file using your normal debugger, depending on the language in question.

System calls

strace is the swiss army knife for seeing what a process is doing. It details each system call made by a program (you can filter them down, of course). For example:

$ strace -e read curl https://google.com/
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \236\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P!\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200l\2\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20Q\0\0\0\0\0\0"..., 832) = 832
# ...
read(3, "\0\0\0\0\0\0\0\4\25\345\366\302\273sE6\365wI\225\321|\3435Z\362\216\372\215\251aO"..., 253) = 253
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
read(3, "\27\3\3\0!", 5)                = 5
# ...

Brendan Gregg has a nice guide on strace and alternatives.

… with docker

When the application is running in docker you can usually just strace it from the host with the correct PID (from e.g. docker inspect --format '{{.State.Pid}}' my_container). Sometimes you may need to trace the startup of an application though, which is a bit trickier. Instead you can run a new container using the same PID namespace as your target, and the permissions needed to strace:

$ docker run --rm -it --pid=container:my_container \
  --net=container:my_container \
  --cap-add sys_admin \
  --cap-add sys_ptrace \
  alpine

From within the new container you can install strace, and trace any running program within the target container using strace -p as normal. To start a new program you need access to the target container’s filesystem, which you can get to via /proc/1/root (PID 1 being the main process that docker started in the target container).

Files

Sometimes the problem might relate to file access. There are a couple of straight forward — but nonetheless useful — tools which might help here. inotifywait uses the Linux inotify subsystem to watch files or directories for operations. For example:

$ inotifywait -mr site/content
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ OPEN 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ CLOSE_WRITE,CLOSE 2019-05-08-debugging-beyond-the-debugger.md
# ...

Here the -m switch makes inotifywait monitor the files forever (instead of exiting on the first modification, which is the normal behaviour) and r makes it recurse into the directory and monitor each file and subdirectory in there.

If you want to see what processes currently have a file open, fuser is the go-to tool. For example:

$ fuser -v /
                     USER PID ACCESS COMMAND
/:                   root     kernel mount /
                     chris      2961 .rc.. systemd
                     chris      2986 .r... gdm-x-session
                     chris      2994 .r... dbus-daemon
                     chris      3001 .r... gnome-session-b
# ...

Honourable mentions

These aren’t really debugging tools, but I feel it’s worth mentioning as they often feature somewhere along the debugging-of-weird-problems journey.

I’ve seen some weird and wonderful problems happen because a disk is full, so a quick df early on in the debugging process never hurts. Some apps may hang, some may corrupt their config, some may fall over and die; sometimes the manner in which they fail doesn’t obviously point to a disk space issue.

Another issue that comes up now and then — especially inside VMs or other environment that don’t have a decent amount of “noise” happening — is entropy exhaustion. A quick look at /proc/sys/kernel/random/entropy_avail should be enough to confirm that everything is ticking along nicely. If it’s exceedingly low then you may find that anything involving random number generation stalls (TLS connections for example).

Creating an offline GnuPG master key with Yubikey-stored subkeys

2016-08-11T00:00:00Z

A pair of Yubikeys

I recently noticed that I’d accidentally lost my previous GPG private key — whoops. It was on a drive that I’d since formatted and used for a fair amount of time, so there’s no hope of getting it back (but, on the plus side, there’s also no risk of anyone else getting their hands on it). I could have created a new one in a few seconds and been done with it, but I decided to treat it as an exercise in doing things properly.

Background: GPG? Yubikey?

GPG or GnuPG is short for Gnu Privacy Guard, which is a suite of applications that provide cryptographic privacy and authentication functionality. At a basic level, it works in a similar way to HTTPS certificates: each user has a public key which is shared widely, and a private key that is unique to them. You can use someone else’s public key to encrypt messages so only they can see them, and use your own private key to sign content so that others can verify it came from you.

A Yubikey is a small hardware device that offers two-factor authentication. Most Yubikey models also act as smartcards and allow you to store OpenPGP credentials on them.

Introducing subkeys

GnuPG supports subkeys, which provide fairly significant security advantages. Instead of just having a single public and private key, you have a master pair and then any number of subkey pairs. The subkeys are automatically associated with the master key, but they can be revoked independently.

Having a master key fall into the wrong hands is a problem — you have to revoke the whole thing (assuming you have access to a revocation certificate) and start again, convincing everyone else that your new key is the “real” you. With subkeys, you can issue a revocation signed with your master key and then sign some new subkeys. There’s no loss of trust, and as long as people refresh your key from a keyserver, everything carries on as normal.

The other advantage to using subkeys is that you can keep the master key elsewhere. It doesn’t need to be routinely accessible, and using it doesn’t require access to the Internet. The master key is kept offline, significantly reducing the risk of anything bad happening to it.

Setting up a secure environment

My main desktop runs Windows, and most of my other devices are work ones which come with automatic backups and network mounts that I don’t fully grok. Neither of those is a particularly good option if I want to do something security sensitive.

Tails is the defacto choice for a secure, live linux system, so I went through their installation process and eventually ended up with a USB drive that can boot into Tails. The installation process from Windows is slightly convoluted, as it involves creating a bootable Tails image, then booting to that and using the Tails installer to create the real image on a different drive. If you’re starting on a Linux box, you can just use the Tails installer directly instead of doing the two-drive shuffle.

Just to be completely paranoid, I disconnected my PC from the network before booting Tails. This is known as air-gapping, and is done to eliminate the possibility of a remote attacker doing something to your system. Ideally the machine never would have been connected to the network, but I didn’t happen to have an unused machine laying around.

The final thing I needed was a secure place to store my master key. I opted for an IronKey — a hardware-encrypted USB drive that self-destructs if there are too many unsuccessful attempts to access it. It works out-of-the-box on both Windows and Linux, presenting a small unencrypted drive with software to run to interact with the secure partition.

Creating the keys

Now I had a nice over-the-top setup it was time to actually the keys. There is an excellent guide by Simon Josefsson that walks through the entire process of creating the master key, creating three subkeys, and then transferring them to a Yubikey.

The only point where I had to deviate from Simon’s guide was setting the machine up to work with the Yubikeys. I was setting up two keys (a nano and a neo), and one just worked out of the box with the version of libykpers-1-1 that was in Tails’ apt repository. The other needed a slightly newer version but that was also available in apt and can be selected by specifying the version manually as pointed out in this GitHub issue. The version numbers have since changed, but apt-policy makes it easy to figure out what’s needed. As I was using an air-gapped system this process was a bit more complicated than it sounds, involving several USB drive transfers.

After finishing the guide, I had my master key and a pre-generated revocation certificate stored securely on my IronKey, and the three subkeys stored on each Yubikey. Time to go back to Windows.

GPG, Windows and SSH

Now with the IronKey disconnected and the master key out of harms way, it’s time to go back to Windows. I downloaded the GnuPG Modern distribution and followed the instructions at the end of Simon’s guide to import my public key and make GPG aware of the subkeys on the Yubikey. After that Enigmail was able to sign and encrypt e-mail in Thunderbird.

XKCD #979: Wisdom of the ancients

Next up, I enabled PuTTy support and started the GPG agent, as documented over on Yubico’s site. This allows you to use the authentication GPG key to authenticate SSH sessions from PuTTy. To find the SSH key you need to add to .authorized_keys, simply run gpg --export-ssh-key. At first I could SSH into a host but not use agent forwarding. After lots of unsuccessful Googling, I realised that GPG couldn’t access the key anymore locally. Another quick search and I found a forum thread where someone had the same issue and found it was a problem with exclusive access to the card. They even passed on their wisdom and updated the thread with a solution, which got everything working for me.

VMWare

I have a Ubuntu image running inside VMWare on my desktop that I use for most development activities. I’m unlikely to want to sign e-mail, but I probably want to sign commits (especially now that GitHub exposes verified signatures).

To do that I need to pass the Yubikey through to the virtual machine. In its default configuration, VMWare recognises the Yubikey device but doesn’t pass it through correctly. You need to configure it to allow HIDs, after which gpg --card-status starts working from the VM.

Unfortunately passing through the device makes it exclusively available to the VM so the host OS can no longer use it. As I have two Yubikeys, I just configured VMWare to ignore the nano key that’s always plugged in, and pass through the neo when I plug that in. While swapping the configuration around, I found out that GPG remembers the ID of the smartcard that store credentials - if you swap the two keys with identical subkeys it demands the other one is reinserted. You can get rid of the references to the previous card using gpg --delete-secret-keys.

In conclusion…

… I have a new PGP key you can use to verify things I sign or encrypt messages to me.