Bits, Bytes, & Radio Waves

A quiet journey through discovery and understanding.

Common Commands Used in Troubleshooting

Overview

I find that I use these commands often when troubleshooting. I thought I would record them here so they are in one place and I can reference them easily, when needed. They are in no particular order.

Reset Photon OS root Account

Restart the virtual machine and on the Photon splash screen, press the ‘e‘ key to interrupt the normal boot sequence.

On the linux line, press Ctrl-e to go to the end of the line, add rw init=/bin/bash and then press Ctrl-e or F10 to boot.

mount -o remount, rw /

To reset the root account, use passwd.

If the appliance also has an admin account, use passwd admin.

Depending on the version of Photon OS;

/sbin/pam_tally2 -r -u <username>

or

/sbin/faillock --user <username> --reset

umount /

reboot -f


Query Network Time Protocol (NTP)

Sometimes, just adding the ntp servers is not enough. It’s also good to check that they are able to query and how far off (skew) the time is.

ntpq -p [Host...]


Setting DNS Servers

In a system managed by systemd-resolved, you can’t just edit the /etc/resolv.conf file anymore. It’s more complicated, unfortunately.

On some systems, use nmtui. Not much needs to be said with this utility.

On some systems, resolvectl is the command to use. This is only going to be temporary! This will not survive a reboot.

resolvectl status

resolvectl dns <interface name> <ip address> <ip address>


Check Open Ports on Linux

ss is used to dump socket statistics. It allows showing information similar to netstat.  It can display more TCP and state information than other tools.

Usage: ss [ OPTIONS ]
       ss [ OPTIONS ] [ FILTER ]
   -h, --help          this message
   -l, --listening     display listening sockets
   -n, --numeric       don't resolve service names          
   -p, --processes     show process using socket
   -t, --tcp           display only TCP sockets
   -u, --udp           display only UDP sockets
   -V, --version       output version information

ss -tuln is the command to run to see open ports on the Linux system.


vmkping

vmkping is a command-line utility in VMware ESXi, executed via SSH, designed to test network connectivity specifically from VMkernel interfaces (used for vMotion, iSCSI, vSAN, and management). It allows administrators to specify source interfaces (-I), troubleshoot Jumbo Frames (-s), and verify routes between subnets, which regular ping cannot. 

vmkping [args] [host]
   args:
      -4               use IPv4 (default)
      -6               use IPv6
      -c <count>       set packet count
      -d               set DF bit (IPv4) or disable fragmentation (IPv6)
      -I <interface>   outgoing interface - for IPv6 scope or IPv4
                       bypasses routing lookup
      -s <size>        set the number of ICMP data bytes to be sent.
                       The default is 56, which translates to a 64 byte
                       ICMP frame when added to the 8 byte ICMP header.
                       (Note: these sizes does not include the IP header).
      -v               verbose
      -S               The network stack instance name. If unspecified
                       the default netstack instance is used.
   NOTE: In vmkernel TCP debug mode, vmkping traverses
         VSI and pings various configured addresses.

When testing for Jumbo Frames (MTU 9000), account for the 28-byte IP/ICMP header overhead. So the maximum size will be 8972. Here is an example of testing an iSCSI VMkernel.

vmkping -4 -c 5 -d -I vmk1 -s 8972 172.16.101.100

If testing vMotion connectivity, that traffic should be on the vmotion stack, so you would use the following example.

vmkping -4 -c 5 -d -I vmk4 -s 8972 -S vmotion 172.16.104.102

One response to “Common Commands Used in Troubleshooting”

  1. Ray Avatar
    Ray

    Just wanted to share a quick ss (socket statistics) tip with the -p flag:

    ss -tuln: https://explainshell.com/explain?cmd=ss+-tuln
    ss -tulpn: https://explainshell.com/explain?cmd=ss+-tulpn

    The -p / –processes option shows the process using each socket, which makes it really useful for troubleshooting.

Leave a Reply to Ray Cancel reply

Your email address will not be published. Required fields are marked *