Chameth.com - posts like debugging-beyond-the-debugger, the-meaning-of-life but not migrating-from-github-to-forgejo, why-you-should-be-using-https

The meaning of life

2026-01-18T00:00:00Z

Now that’s a grandiose title for a blog post! Blame Jeremiah Lee, who selected it as the theme for January’s IndieWeb Carnival. I’ve not taken part in this before, but it’s basically a shared writing prompt. One person selects a theme, other people write about it on their own sites, and then the host does a round-up linking to all the various responses. I’ve come across it a few times before, but the topics have never really engaged me; they don’t feel like the sort of thing I’d write about. But… why not?

A Guide to the Good Life

The thing that actually tipped the scales for this particular topic was a book I was reading: A Guide to the Good Life by William B. Irvine. It’s basically a modern introduction to Stoicism. Not only is there a strong overlap between that and the meaning of life, one particular idea resonated with me:

In ancient Greece, when schools of philosophy were still prominent features of the cultural landscape, there were any number of schools to which parents could send their children. Suppose we could travel back in time to 300 bc and take a thinking person’s walking tour of Athens. We could begin our tour in the Agora, where Socrates a century earlier had philosophized with the citizens of Athens. On the northern side of the Agora we would see the Stoa Poikile, or Painted Porch, and holding forth there might be Zeno of Citium, the founder of the Stoic school of philosophy. This “porch” was actually a colonnade decorated with murals.

The idea that you could just wander around a city and come across people giving their considered opinions on philosophy is… something. Do we have any actual equivalent of that in the modern ages? Obviously you do get people espousing their views on the street, but they tend to either be proselytising or grinding a particular axe. And even if there was a modern day Socrates talking in Speaker’s Corner, would anyone actually stop to listen? We do have books, and speaking engagements, and so on, but they feel a lot more impersonal, and in a lot of cases motivated by money or politics or some other agenda. The philosophers of ancient Greek and Rome seemed to actually practice what they preached; as well as talking in public, and instructing in their schools, they’d write letters helping and advising people, or debating points of philosophy.

Some of that behaviour exists in academia, I’m sure, but it’s not really accessible to the person on the street in the way that it once was. But we do have personal websites and blogs. People just exchanging ideas, speaking out in public, with (generally) no expectation of reward or recognition for it. Is the “IndieWeb” the modern Agora? Probably not, but maybe it gives us some aspects of it that we otherwise wouldn’t have?

The Stoic approach to life

The Stoics believed in living a virtuous life; living as we were designed to live, in accordance with nature. A lot of those terms might be a bit loaded to the modern ear, and “designed to live” is awfully vague. I think this quote expands on it pretty well, though:

And for what function were people designed? To answer this question, the Stoics thought, we need only examine ourselves. On doing this, we will discover that we have certain instincts, as do all animals. We experience hunger; this is nature’s way of getting us to nourish ourselves. We also experience lust; this is nature’s way of getting us to reproduce. But we differ from other animals in one important respect: We have the ability to reason. From this we can conclude, Zeno would assert, that we were designed to be reasonable.

And if we use our reason, we will further conclude that we were designed to do certain things, that we have certain duties. Most significantly, since nature intended us to be social creatures, we have duties to our fellow men. We should, for example, honor our parents, be agreeable to our friends, and be concerned with the interests of our countrymen.

It all seems eminently reasonable. But in a world of fake news, social media, culture wars, and everything else, our ability to reason often seems forgotten. Hell, even being agreeable seems like it’s out of vogue at the moment. Would the Stoics despair if you transported one of them to the present day? Probably not. One of the other tenets they held was, well, being stoic. There was a concept of tranquility:

Stoic tranquility was a psychological state marked by the absence of negative emotions, such as grief, anger, and anxiety, and the presence of positive emotions, such as joy.

To get to this magical state, you need a form of detachment. Not in the sense of being completely passive and impartial as you float through life; that’s not how we’re “designed to life”. But in the sense of not letting outside events influence your inner state. When someone says something horrible, they don’t have the power to actually make you feel bad; that’s something you do. And I fully accept that it’s very easy to say that, and very hard to actually practice it.

This is actually a concept shared with many other religions and philosophies. Buddhism has the idea of Nekkhamma, Jainism and Hinduism have Vairāgya, and so on. Even if you don’t put any stock at all into religious teachings, it’s interesting that this concept comes up again and again.

If you’re like me, you probably have thoughts on all this. Sure, anxiety-free, anger-free life is probably nice, but how does that relate to the meaning of life and, anyway, shouldn’t you be angry about some things? The Stoics (particularly the Roman ones, who tended to not be so into physics and ethics) believed that this tranquility made it easier to focus on being virtuous. That makes sense to me; if we manage to get off the hedonistic treadmill of wanting shiny new things, of hoping people like our posts, and so on, then there’s surely more time in life for other pursuits. As for anger, I think the point is you don’t let the emotions drive your behaviour. There’s a difference between getting angry at what someone is saying and punching them in the face, and hearing someone, calmly and rationally assessing matters, and then punching them in the face. Maybe it’s best for society to punch certain people in the face. The Stoics were certainly not pacifists. Marcus Aurelius is one of the most well known, and he was Emperor of Rome. He’s hailed as one of the few good emperors, but you can bet he had an awful lot of blood on his hands even so.

Putting it into practice

It’s all well and good going on about not letting things bother you, but it’s very much easier said than done. The book introduces the concept of “negative visualisation”: imagining the negative things that could happen in your life, and thus starting to accept them. It also helps you to appreciate things that you do have, that you otherwise taken for granted. I’m sure everyone’s imagined what would happen if you lost your home, or your job, or so on… but what if you didn’t have your favourite mug? A particular tool? A certain piece of art?

I’ve heard of gratitude journals before. I’ve even tried keeping one for a while. There’s science showing it’s beneficial for mental health. But I always faded off after a short while. Once you tick off the big things: friends, family, food, etc, it feels like a stretch. Negative visualisation feels like a much better approach to me. I’m about to eat an apple; at the end of the day I’m not going to think back and write about an apple in a gratitude journal. But I can think briefly, in the moment, about not having the apple. Do I need to prepare myself for an apple-less life? Not really. But do I now appreciate the apple a little bit more, and take it for granted a little bit less? Yes. Does that make me a bit more tranquil? A bit less likely to wish I had something tastier than an apple? Also yes.

This process dovetails into the concept of impermanence, another concept found in Buddhism and elsewhere. Each apple could be the last one you ever have. If you knew that in advance, how would it change your approach? You’d probably savour it a bit more, appreciate it a bit more, and most of all not take it for granted. There’s a lot of pithy sayings about treating each day as if it’s your last, but at that level they’re a lot more nebulous than helpful; appreciating an apple, though? That’s something I can do.

It seems at first glance that thinking about negatives should be depressing or anxiety inducing. There are only so many reasons that this could be my last apple, and most of them involve something pretty unideal happening to me. But I don’t think it’s actually that glum. You don’t dwell on the negative, instead you use it to inspire a positive. And the alternative is to just sleepwalk through the process, not really paying attention, not gaining any enjoyment from it.

Bringing it back

Thanks for sticking with me on that tangent. You’ll be glad to know I’ve finished eating my apple now. The topic for this post was meant to be “the meaning of life”, not “thoughts on eating apples” or “living a Stoic life”, so allow me to try to tie it all together and pretend that I had a plan.

Life has no inherent meaning, it’s just a biological process. But that doesn’t mean we can’t seek to have a meaningful existence. The Stoics believed that involved being reasonable, and one of the approaches to give the reasoning room to happen is to strive towards an inner state of tranquility.

Obviously the Stoic approach isn’t the only one. And I’m not sure I did their philosophies any justice in my writing here. If it sparked your interest, though, I highly recommend picking up the book. If not, other approaches to life are available…

Be excellent to each other… and party on, dudes.

Debugging beyond the debugger

2019-05-08T00:00:00Z

Real-life debugging tools

Most programming — and sysadmin — problems can be debugged in a fairly straight forward manner using logs, print statements, educated guesses, or an actual debugger. Sometimes, though, the problem is more elusive. There’s a wider box of tricks that can be employed in these cases but I’ve not managed to find a nice overview of them, so here’s mine. I’m mainly focusing on Linux and similar systems, but there tend to be alternatives available for other Operating Systems or VMs if you seek them out.

Networking

tcpdump

tcpdump prints out descriptions of packets on a network interface. You can apply filters to limit which packets are displayed, chose to dump the entire content of the packet, and so forth.

Typical usage might look something like:

# tcpdump -nSi eth0 port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:03:35.577781 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [S], seq 2815779044, win 64800, options [mss 1440,sackOK,TS val 2378811665 ecr 0,nop,wscale 7], length 0
16:03:35.586853 IP6 2001:db8::2.80 > 2001:db8::1.54742: Flags [S.], seq 1522609102, ack 2815779045, win 28560, options [mss 1440,sackOK,TS val 3063610173 ecr 2378811665,nop,wscale 7], length 0
16:03:35.586877 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [.], ack 1522609103, win 507, options [nop,nop,TS val 2378811674 ecr 3063610173], length 0
16:03:35.620678 IP6 2001:db8::1.54742 > 2001:db8::2.80: Flags [P.], seq 2815779045:2815779399, ack 1522609103, win 507, options [nop,nop,TS val 2378811708 ecr 3063610173], length 354: HTTP: GET / HTTP/1.1

Here you can see the start of a plaintext HTTP request: the three-way handshake as the TCP connection is established followed by a GET request. Even if the data is encrypted as it will be in most cases, it’s often useful to see the “shape” of the transmissions: did the client start sending data when it connected, did the server ever respond, etc.

Daniel Miessler has a good tutorial on tcpdump if you’re not familiar with it and don’t want to jump straight into the man page.

… with Docker

Docker sets up separate network namespaces for each container. To see the traffic across the interfaces of a single container you can nsenter the container’s network namespace:

# nsenter -t $(docker inspect --format '{{.State.Pid}}' my_container) -n tcpdump -nS port 80

This retrieves the PID for the container, and tells nsenter to enter the network (-n) namespace from the given target (-t) PID, and then run the given command (in this case tcpdump ...).

openssl s_client / s_server

When a connection is using TLS it’s often useful to try connecting to the server and see what certificate it presents, algorithms it negotiates, and so forth. OpenSSL offers two useful subcommands which can help with this: s_client for connecting as a client, and s_server for listening to connections.

For example, using s_client to connect to google.com on the standard HTTPS port shows us details about the server cert and its verification status:

$ openssl s_client -connect google.com:443
CONNECTED(00000003)
depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
verify return:1
depth=1 C = US, O = Google Trust Services, CN = Google Internet Authority G3
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com
   i:C = US, O = Google Trust Services, CN = Google Internet Authority G3
 1 s:C = US, O = Google Trust Services, CN = Google Internet Authority G3
   i:OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
---
# ...

Whereas connecting to my webserver and providing an unknown host in the SNI field results in an SSL alert 112 (“The server name sent was not recognized”) and no server certificate is sent:

$ openssl s_client -connect chameth.com:443 -servername example.com
CONNECTED(00000003)
140384831313024:error:14094458:SSL routines:ssl3_read_bytes:tlsv1 unrecognized name:../ssl/record/rec_layer_s3.c:1536:SSL alert number 112
---
no peer certificate available
---
# ...

Often if you hit this kind of alert in an application the exact error will be lost somewhere in the many layers between the SSL library and the logs, so being able to directly connect and test can help diagnose a lot of issues.

Once a connection is established you can read and write plain text and it will be encrypted and decrypted automatically.

Java apps

If a Java app is involved in the connection, you can enable a lot of built-in debugging with a simple JVM property: javax.net.debug. You can tweak what exactly gets logged, but the easiest thing to do is just set the property to all and you’ll see information about certificate chains, verification, and packet dumps:

$ java -Djavax.net.debug=all -jar ....
# ...
found key for : duke
chain [0] = [
[
  Version: V1
  Subject: CN=Duke, OU=Java Software, O="Sun Microsystems, Inc.",
  L=Cupertino, ST=CA, C=US
# ...

More information about Java’s debugging options is available on docs.oracle.com.

Thread and core dumps

Higher-level languages frequently provide an interactive way to dump the current execution state of all of their threads (a “thread dump”). This is useful to spot deadlocks, some types of race conditions, and as a quick and dirty method of investigating hangs or excessive CPU usage.

With both Java and Go applications you can send a QUIT signal to have a thread dump printed out; Go applications will quit after doing so, Java ones will carry on running. At most terminals you can hit Ctrl and \ to send a QUIT signal.

For Java you can also use the jstack tool from the JDK to dump threads by PID; this can be useful if the application is running in the background or has redirected sysout:

$ jstack 8321
Attaching to process ID 8321, please wait...
Debugger attached successfully.
Client compiler detected.

Thread t@5: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=-1107318896 (Interpreted frame)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Interpreted frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Interpreted frame)

# ...

A core dump provides more complete information about the state of a process, but is often more complex to interpret. The gcore utility from GDB will create a core dump of a process with a given PID. You can then generally load the core file using your normal debugger, depending on the language in question.

System calls

strace is the swiss army knife for seeing what a process is doing. It details each system call made by a program (you can filter them down, of course). For example:

$ strace -e read curl https://google.com/
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \236\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P!\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200l\2\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20Q\0\0\0\0\0\0"..., 832) = 832
# ...
read(3, "\0\0\0\0\0\0\0\4\25\345\366\302\273sE6\365wI\225\321|\3435Z\362\216\372\215\251aO"..., 253) = 253
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
read(3, "\27\3\3\0!", 5)                = 5
# ...

Brendan Gregg has a nice guide on strace and alternatives.

… with docker

When the application is running in docker you can usually just strace it from the host with the correct PID (from e.g. docker inspect --format '{{.State.Pid}}' my_container). Sometimes you may need to trace the startup of an application though, which is a bit trickier. Instead you can run a new container using the same PID namespace as your target, and the permissions needed to strace:

$ docker run --rm -it --pid=container:my_container \
  --net=container:my_container \
  --cap-add sys_admin \
  --cap-add sys_ptrace \
  alpine

From within the new container you can install strace, and trace any running program within the target container using strace -p as normal. To start a new program you need access to the target container’s filesystem, which you can get to via /proc/1/root (PID 1 being the main process that docker started in the target container).

Files

Sometimes the problem might relate to file access. There are a couple of straight forward — but nonetheless useful — tools which might help here. inotifywait uses the Linux inotify subsystem to watch files or directories for operations. For example:

$ inotifywait -mr site/content
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ OPEN 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ MODIFY 2019-05-08-debugging-beyond-the-debugger.md
site/content/post/ CLOSE_WRITE,CLOSE 2019-05-08-debugging-beyond-the-debugger.md
# ...

Here the -m switch makes inotifywait monitor the files forever (instead of exiting on the first modification, which is the normal behaviour) and r makes it recurse into the directory and monitor each file and subdirectory in there.

If you want to see what processes currently have a file open, fuser is the go-to tool. For example:

$ fuser -v /
                     USER PID ACCESS COMMAND
/:                   root     kernel mount /
                     chris      2961 .rc.. systemd
                     chris      2986 .r... gdm-x-session
                     chris      2994 .r... dbus-daemon
                     chris      3001 .r... gnome-session-b
# ...

Honourable mentions

These aren’t really debugging tools, but I feel it’s worth mentioning as they often feature somewhere along the debugging-of-weird-problems journey.

I’ve seen some weird and wonderful problems happen because a disk is full, so a quick df early on in the debugging process never hurts. Some apps may hang, some may corrupt their config, some may fall over and die; sometimes the manner in which they fail doesn’t obviously point to a disk space issue.

Another issue that comes up now and then — especially inside VMs or other environment that don’t have a decent amount of “noise” happening — is entropy exhaustion. A quick look at /proc/sys/kernel/random/entropy_avail should be enough to confirm that everything is ticking along nicely. If it’s exceedingly low then you may find that anything involving random number generation stalls (TLS connections for example).