Category Archives: Tech

Google Ping Update

Update: here’s the traceroute when ping performance is 50+ msec:

1 pfsense.nw 1.578 ms
2 * * *
3 * * *
4 tge7-2.ausbtx5202h.texas.rr.com 22.567 ms
5 tge8-5.ausbtx5201h.texas.rr.com 18.703 ms
6 tge0-12-0-6.ausutxla01r.texas.rr.com 15.788 ms
7 agg22.dllatxl301r.texas.rr.com 21.964 ms
8 107.14.17.136 21.865 ms
9 ae1.pr1.dfw10.tbone.rr.com (107.14.17.234)
10 207.86.210.125 (207.86.210.125) 26.794 ms
11 207.88.14.182.ptr.us.xo.net 21.649 ms
12 207.88.14.189.ptr.us.xo.net 17.312 ms
13 ip65-47-204-58.z204-47-65.customer.algx.net 20.543 ms
14 72.14.233.85 24.646 ms
15 72.14.237.219 24.955 ms
16 209.85.243.178 27.354 ms
17 72.14.239.136 43.774 ms
18 216.239.50.237 48.918 ms
19 209.85.243.55 44.747 ms
20 ord08s11-in-f20.1e100.net 43.592 ms

Four extra hops. Hitting a google server presumably somewhere in Chicago instead of Dallas. ORD is the Chicago O’Hare airport code; I don’t really know whether the google data centers are in fact at/near airports or whether they are just using airport codes as a convenient naming scheme for a general area.

So, sometimes I get directed to Dallas, sometimes to Chicago. Will report if I ever see any other server locations.

As an aside, “1e100.net” is google’s clever name for their network.

Hilltop Google Ping Performance

For the past month I have been measuring internet ping (ICMP ECHO) performance from my hilltop network. I do this with a script that runs every 30 minutes and measures the response time of www.google.com as reported by ping.

The script first pings www.google.com exactly once in an attempt to cache the DNS lookup (to avoid DNS time affecting the results). It then invokes the unix ping program with “-c 10” to do 10 pings. I throw out the highest and lowest result times and average the remaining 8. I record the results in a NumerousApp metric (of course).

The results are shown here:

Hilltop Google Ping

Raw data available in the numerous metric: Hilltop Google Ping

Ignore the occasional spikes when obviously some network disruption was causing consistently high ping times for that measurement (and there is one zero data point where a bug in the script caused a zero reading when the network was completely down).

What’s left is two different consistent readings – something in the low-mid 20msec response range and something averaging in the 50msec response range. It seems pretty apparent that there are two different routes between my hilltop network and www.google.com and for whatever reason sometimes I’m hooked up to the faster/shorter route and sometimes the longer one.

Here, for your amusement, is the heart of my “pinggoo” script:

ping -c 10 $TARGET |
grep from | grep 'time=' |
sed -e 's,.*time=,,' | 
awk ' { print $1 } ' | sort -n | sed -e '1d' -e '$d' | 
awk 'BEGIN {SUM=0; N=0} {SUM=SUM+$1; N=N+1} END {print SUM/( (N>0) ? N : 1)}'

Here’s what a typical output line from ping looks like on my mac:

64 bytes from 74.125.227.114: icmp_seq=0 ttl=53 time=22.388 ms

This is some truly fine shell hackery. It turns out the two grep statements are redundant (either one alone suffices) but I put them both in as a way to ensure I was really looking only at the successful ping lines (the ping program itself puts out a lot of other verbose output). Then the sed deletes everything prior to the ping time. The awk program print $1 separates out the time from the trailing “ms”. What we then have is (hopefully) a list of 10 numbers, one per line. I use sort to put them in numeric order, then sed to delete the first and last line (highest/lowest ping time) and then the final awk program to calculate the average.

I’m sure I could have done all this with a single pass of awk or a python program or something along those lines; however, one nice thing about this hackery is that it is fairly robust across ping variants; so far this has worked just fine on my mac and on Debian wheezy, even though the two ping programs have different output formats (but the essential “time=” part is similar enough on both to work unchanged with this script).

Here’s a traceroute I just did while I appear to be getting the faster performance:

 
 1  pfsense.nw 1.379 ms
 2  * * *
 3  * * *
 4  tge7-5.trswtx1202h.texas.rr.com 20.527 ms
 5  tge0-12-0-14.ausxtxir02r.texas.rr.com 18.915 ms
 6  agg22.hstqtxl301r.texas.rr.com 28.277 ms
 7  107.14.19.94 24.099 ms
 8  ae-0-0.cr0.dfw10.tbone.rr.com 27.132 ms
 9  ae0.pr1.dfw10.tbone.rr.com 24.956 ms
10  207.86.210.125 21.302 ms
11  207.88.14.182.ptr.us.xo.net 25.221 ms
12  207.88.14.189.ptr.us.xo.net 24.043 ms
13  ip65-47-204-58.z204-47-65.customer.algx.net 27.043 ms
14  72.14.233.77 26.225 ms
15  64.233.174.137 25.225 ms
16  dfw06s32-in-f18.1e100.net 23.943 ms

I edited out a bunch of the output detail to make it fit on this page better. Sixteen hops to whichever google server is serving me. If we interpret the hostname at face value my google server (at this moment) is in DFW somewhere.

I’m on Time Warner cable and was recently upgraded to 100Mb performance. This (unsurprisingly) doesn’t seem to have had any material impact on the ping times (throughput and latency being somewhat independent).

I’ll report back if I get any other interesting traceroute data especially when I’m in the 50msec performance arena.

Murphy’s Law

This will reaffirm your faith in Murphy’s Law.

Time Warner recently replaced my cable modem – upgraded for higher performance.

On Friday I went to check the physical install. The modem is down the hill – a quarter mile away – and connects to the house network via a long fiber run.

A long time ago I installed a “remote power rebooter” device for the cable modem so that on those all-too-often occasions when the modem needs to be physically reset it could be power cycled remotely/automatically. Of course the cable guy didn’t plug the new modem into this device, instead he plugged it directly into the wall.

As an aside: the remote control power gizmo I’m using is from Synaccess and it is awesome:

http://www.synaccess-net.com/remote-power.php/1/8

You set this box up to ping a remote address (e.g.,www.google.com) and if it loses connectivity it will power-cycle the outlet. So any time my cable modem wedges it gets rebooted automatically when this device detects loss of internet connectivity.

On Friday I should have unplugged the cable modem and moved it back to the rebooter power outlet. But I was literally on my way out of the house to go out of town for the weekend. The Number One Rule of IT was looming large in my mind: “If it’s working, don’t mess with it”.

The cable modem was working. Power cycling it right before leaving seemed foolish. I could fix it at my leisure on Monday when I got back.

Ah, Murphy, I am so sorry I tempted you that way.

Of course within hours of me actually *being* out of town, the cable modem wedged for some inexplicable reason. I had no VPN access to my home network the entire time I was gone. It wasn’t a big deal, but it was annoying, especially since I knew exactly WHY the modem hadn’t rebooted itself automatically after getting wedged and yet I was a few thousand miles away from it to fix it.

Maybe Thoreau was right.

NumerousApp

Here’s what I’ve been working on in the “office” … integrating different things into www.numerousapp.com

You can monitor:

Check out the iPhone numerous application (Android coming soon; these guys have just gotten started). Disclaimer: they are friends of mine and I am an investor/advisor.

SNMP at home

Several devices on my home network support SNMP. Here’s one way I’m making use of this.

Want to know the last time your router rebooted? Try this:

snmpget -c public server-name system.sysUpTime.0

That UNIX command will work as-is on a Mac. It will also work on Debian Linux but you may have to install the snmp package first and you’ll also have to download the so-called “MIB” to define the “system.sysUptime.0” name.

Without the MIB installed you can still just use the raw low-level object ID:

snmpget -c public server-name 1.3.6.1.2.1.1.3.0

Some (older) Apple airports, most Cisco devices, many routers, and other such networking equipment support SNMP. So, for example, on my (large) home network I have four Cisco WiFi transmitters. I’ve been interested in charting how often the power fails at my house. So I wrote a script to send each WiFi box this command once a day:

nw% snmpget -c public wap-1 system.sysUpTime.0

DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (1122240526) 129 days, 21:20:05.26

(Command to server “wap-1” shown) and I record the results. Obviously when there’s been a power failure all the WAPs reboot and now I can know if that has happened (the server these monitoring scripts run on has a UPS so it stays up; otherwise I could of course just look at the server’s own uptime).

If you google snmp system object you will find some pages with more names/OIDs for other variables that might be useful. Also your devices likely have many interesting device-specific parameters you can fetch.  Google is your friend for finding more MIB/OIDs to try.

My pfSense router supports SNMP but you have to enable the SNMP service first (not enabled by default). This may be true on other device types as well so if they don’t respond to SNMP browse through their admin interfaces and see if you have to (i.e., can) enable SNMP.

Installing FreeBSD 10.0 NanoBSD on a Soekris net6501

I recently built a NanoBSD configuration of FreeBSD 10.0 for a Soekris net6501 box, in a VirtualBox build environment on my mac.

I had to piece this procedure together using information from a variety of sources, so I’m documenting it here in the (longshot) hope that it might be useful to someone else somewhere/someday.

What You Need

  • Familiarity with FreeBSD. This is not a UNIX tutorial.
  • VirtualBox on your mac. If you are new to VirtualBox read up on the basics on your own. I’ll outline my approach and give details on one tricky part: accessing a USB stick from a virtual machine.
  • Some familiarity with NanoBSD. Again, this is not a tutorial but I will talk about the configuration options I used for the Soekris.
  • A Soekris 6501. I’m using the -50 model (1GB memory, 1GHz).
  • A Mac with lots of free disk space. Probably 32GB minimum.
  • A null-modem cable
  • An ancient vt100-ish terminal or a USB/Serial adapter on your mac. I’m using this Keyspan product and “screen” (built into OS X).
  • A low-profile USB stick (to fit into the Soekris case) such as the Sandisk Cruzer Fit. I’m using a 16GB stick.

What You’ll End Up With

Your Soekris box will be running NanoBSD booting off the USB stick. You get a wonderful environment that runs read-only from the USB and is a solid environment for hosting embedded applications. You will (presumably) use the serial console terminal just enough to configure network access and other items specific to your project and then work via ssh or whatever after that.

First: Get the terminal working

Connect your terminal (or emulator) to the 6501 and power on. The factory delivers the Soekris set to 19200 on its serial port. Here’s my command for accessing the console on my mac:

screen /dev/ttySOMETHING 19200

where “SOMETHING” depends on your particular adapter. Mine is /dev/tty.USA19H142P1.1 (nice, huh?). An old real terminal would also work (set to 19200).

You should see the power on self test output and eventually see a “> ” prompt. Type show and press return. If you see something like this then everything is working:

> show

ConSpeed = 19200
ConLock = Enabled
ConMute = Disabled
BIOSentry = Enabled
PCIROMS = Enabled
PXEBoot = Enabled
FLASH = Primary
BootDelay = 5
FastBoot = Disabled
BootPartition = Disabled
BootDrive = 80 81 F0 FF 
ShowPCI = Enabled
Reset = Hard
CpuSpeed = Default

FreeBSD runs the console at 9600, so change the Soekris BIOS to also use 9600 on the serial port. This is the path of least resistance (vs changing FreeBSD). Type this:

set conspeed=9600

Reboot the box and restart your terminal emulator at 9600 and verify you still have connectivity.

VirtualBox FreeBSD

We’ll make a virtual machine environment using VirtualBox and use that to install FreeBSD and build the Soekris installation.

I imagine that the outline of this procedure would work just as well for other OS variants you might want to build/install on the Soekris, though of course the kernel configuration details would differ.

Go to the FreeBSD site and download the amd64 disc1 CD .iso file.

Create a machine in VirtualBox but do not start it yet. I used these parameters:

  • Type: BSD
  • Version FreeBSD (64bit)
  • Memory Size 1024MB (not sure how much this matters)
  • Create Virtual drive. I used 64GB. You need space to hold the build tree and the eventual USB stick image. In my case I appear to be using about 10GB of the virtual drive I created, so you could certainly get by with less than 64GB if you are short on disk space.

Before starting the virtual machine, attach the .iso file CD image to it as a CD in the drive. To do this select the virtual machine, click the settings gear, go to the storage pane and click around until you figure out how to put the .iso file into the virtual CD drive. Also go to the System pane and verify that it is booting off the CD drive before the hard drive.

Now you can start the machine. It should boot from the FreeBSD “CD” (.iso file) and launch you into the FreeBSD install dialog. Follow the instructions. IMPORTANT: don’t forget to select the option to install the source code when you are asked about optional components (it’s not installed by default).

When it is done it will reboot, which of course will launch you right back into the CD install menu again. So before you let it reboot take the FreeBSD .iso image out of the virtual CD drive (click on the CD icon at the bottom of the VirtualBox window for your machine). Now let the machine reboot and you should be presented with a login prompt (password as however you set it during install).

You now have a virtualized FreeBSD environment running on your mac. Now we’ll use that environment to build the NanoBSD disk image that will eventually boot on the Soekris.

Configuring a net6501 kernel

We need to build a custom kernel because the GENERIC FreeBSD kernel won’t run on the Soekris box; you need to add “device mptable” to it to make it work. So do that. On your FreeBSD virtual machine go to /usr/src/sys/amd64/conf and copy the GENERIC config file to SK6501 (or whatever you want to call it). Edit this file and add “device mptable” and (for good housekeeping) change the ident from GENERIC to something else. Here’s the totality of a diff from the GENERIC config to my config:

22c22
< ident		GENERIC 
--- 
> ident		SK6501
26a27
> device	mptable	  # for Soekris

I put the device line right before the plethora of standard options “SCHED_ULE”, “PREEMPTION” etc.

You also need to insert a delay during booting because the USB stick doesn’t always become “ready” in time to mount the root file system. There are two ways to do this. Either put

options CAM_BOOT_DELAY=10000

into the kernel config, or else you need to get

set kern.cam.boot_delay=10000

put into your /boot/loader.conf boot parameters file. You need to do that via a NanoBSD customization command. See the NanoBSD docs for an example and you can follow the “nobeastie” example (see below) as a template.

Without either of these your system may not always boot reliably; it will sometimes drop you into the mountroot prompt on the console after failing to mount the / filesystem. The issue, as far as I understand it (which is not very well) has to do with the USB stick being “ready” in time for the mounting of / at boot time. The delay (that’s 10 seconds) makes it work reliably; I did not spend any time trying to determine the minimal delay necessary.

Of the two methods I prefer building the delay into my kernel configuration because that way it is always “there” regardless of the boot parameters file.

Creating a NanoBSD configuration

Next we need to customize a few NanoBSD options, including telling NanoBSD to use our modified kernel configuration. The NanoBSD tools are in /usr/src/tools/tools/nanobsd (yes, “tools” twice).

Create a file “sk.conf” with your NanoBSD particulars. Some of this is up to your specific application, but some of it is important for getting the Soekris to work. Here’s what I think is the minimum set of requirements:

NANO_BOOTLOADER="boot/boot0"
NANO_KERNEL=SK6501
NANO_MEDIASIZE=`expr -e 6442450944 / 512`
NANO_NAME=sk6501
NANO_DRIVE=da0
cust_nobeastie() (
	touch ${NANO_WORLDDIR}/boot/loader.conf
	echo "beastie_disable=\"YES\"" >> ${NANO_WORLDDIR}/boot/loader.conf
)

customize_cmd cust_nobeastie
customize_cmd cust_comconsole

The “SK6501” for NANO_KERNEL must match whatever name you picked for your kernel config file. The “NANO_NAME” can be whatever you want, that will control where the build goes (it will show up later in a “nanobsd.sk6501” directory if you set the NANO_NAME to “sk6501”). For MEDIASIZE pick something that makes sense for your application. My choice results in two 3GB partitions (active and backup) and of course the standard tiny /cfg partition (read up on NanoBSD for details on all this). The extra space on my 16GB USB sticks is going to be used for another partition in my application. You’ll likely want to adjust your MEDIASIZE for your own purposes.

The nobeastie thing simplifies the boot menu; painting the BSD daemon in ASCII art at 9600 isn’t worth waiting for. In case your web browser has mangled that code, note that the “echo” command line needs to be all on one line.

The comconsole function (built in to NanoBSD environment) alters the /etc/ttys file so that the serial port console will have a getty running on it when the system boots. The default is to look for a VGA console which isn’t there on the Soekris.

You will likely want/need to read up on more NanoBSD options because you’ll probably eventually want to change more than just these parameters. I’m just showing how to get to the point where you can build a working system and boot it and you can iterate from there.

Building NanoBSD

When you’ve got your NanoBSD config file “sk.conf” (or whatever you called it) all ready, it’s time to build the world:

# cd /usr/src/tools/tools/nanobsd
# sh nanobsd.sh -c sk.conf

I suggest redirecting that output to a logfile and running in the background in whatever your favorite UNIX idiom for that is.

It took a long time on my mac, come back in about two hours YMMV.

If you are lucky, you’ll end up with an exit 0 and a complete build.

There was a bug in earlier releases of FreeBSD10 that caused me to write the next few paragraphs that are now all indented and italicized. The bug is fixed and so you shouldn’t have to hack out the “conv=sparse” as described below. For now I’m leaving this information here for reference but you really should ignore it unless you for some reason are trying an older FreeBSD10:

I ran into a problem that seems to be a bug in FreeBSD10 and is triggered by the use of “conv=sparse” in a dd command in the nanobsd.sh script. You can read a thread about this and patch your kernel, OR you can do the much simpler thing and edit the nanobsd.sh script to remove the “conv=sparse” from the dd command that is creating the secondary (backup) partition on the NanoBSD image.

If your build fails with an error during the dd of the secondary partition, then look for this line in nanobsd.sh:

dd conv=sparse if=/dev/${MD}s1 of=/dev/${MD}s2 bs=64k

and simply delete the “conv=sparse” part. This will work-around the bug. Restart your nanobsd build using the “-b” option (to skip the building of the world and the kernel; you don’t need to rebuild from scratch at this point) and hopefully this time it will make your disk image correctly. Whether you hit the bug or not depends on a bunch of things so your mileage may vary.

The conv=sparse thing just saves disk space in your build environment and therefore on your mac; it has no semantic effect on the disk image. Indeed the nanobsd.sh script in FreeBSD9.2 didn’t specify this in the dd command; it’s new in the version 10 release.

When you are done you will be rewarded with a bunch of files in /usr/obj/nanobsd.sk6501 (where “sk6501” is whatever NANO_NAME you set in the .conf file you made).

There should be a file _.disk.full there. This is what you are now going to transfer to your USB stick in the next step.

Shut down (“# shutdown -h now”) your virtual FreeBSD environment and use VirtualBox to power off the virtual machine. We’re going to be editing its parameters next (which can only be done when it is shut down).

USB Stick Magic

Now we need to get that _.disk.full image file onto a USB stick. There might be a better way to do this, but what I did was make the physical USB stick show up as a disk drive in the FreeBSD virtual machine, so I could then just dd the disk image directly onto the stick.

THIS PROCESS IS FRAUGHT WITH PERIL. If you get the device name wrong you might wipe your mac. That’s your problem, not mine. Pay attention to what you are doing!

Put your target USB stick into a USB port on your mac. Pull up a shell window and type “diskutil list”. From that you should be able to tell which disk device your USB stick is showing up as. It is /dev/disk3 on my mac. YOU NEED TO GET THIS RIGHT so be sure.

For the rest of this writeup I will use /dev/disk999 as a placeholder for whatever /dev/diskN device is appropriate for your configuration.

If there are filesystems on this device that your mac recognized you need to unmount them:

MyMac:$ diskutil unmountdisk /dev/disk999

You need read/write access to this device as an ordinary user.  On my mac the device is only read/write by root, so I did:

MyMac:$ sudo chmod 666 /dev/disk999

You will likely have to repeat one or both of these actions every time your mac reasserts dominion over the USB stick (e.g., if you reinsert it)

At this point we’re ready to create a “VMDK” file which VirtualBox will use as a descriptor/pointer to the physical device. There is a VirtualBox command line tool for this, VBoxManage. Go to some suitable directory (you’ll need to remember this later) and use this command (all on one line of course):

MyMac:$ VBoxManage internalcommands 
 createrawvmdk -filename stick.vmdk 
 -rawdisk /dev/disk999

The file (“stick.vmdk”) this creates is a text file – you can examine it and see that the device name (/dev/disk999 or whatever is right on your system) is in there. You could edit this later if that changes for some reason.

Now you can attach this to your FreeBSD virtual machine. Go back into the VirtualBox GUI. Your virtual machine should be powered off at this point (“# shutdown -h now” and then power it virtually off). Go into its settings and use the add disk icon on the storage pane. You are “adding an existing disk” and choose the .vmdk file you just added.

Now your physical USB stick will show up as a virtual raw device in your FreeBSD virtual machine. Start the FreeBSD virtual machine and:

# cd /usr/obj/nanobsd.sk6501
# ls -l _.disk.full

You should see a large file – this is the drive image that NanoBSD created. Your virtualized USB stick drive should show up in your FreeBSD environment as /dev/ada1. Verify that that device exists and then you are ready for:

# dd if=_.disk.full of=/dev/ada1 bs=1m

On my mac this took almost an hour for a 6GB image. I don’t know if there is a better faster way. Of course this is the step where if you specified the wrong /dev/diskN in your VMDK file you are now obliterating *some* device on your mac somewhere instead of writing to the USB stick. Don’t do that; be very sure you got all the device names correct in the VMDK setup.

Booting the Soekris

Now the easy and fun part! Shut down your FreeBSD virtual machine again (“# shutdown -h now” and then power it off in VirtualBox) and pull the USB stick. Put the USB stick into your Soekris. Start your terminal emulator up again and power up the Soekris.

You should be rewarded with a login prompt on the console. Log in as root (no password) and start configuring your system. Be sure you understand how NanoBSD works and how to save configuration changes so they’ll persist to the next boot; the rest of the process from here is beyond the scope of this write-up and obviously depends a lot on what you are trying to do.

Going back and forth

Once you’ve reached this point, you can put the USB stick back into your mac too and access it (mount it) as a regular filesystem from your VirtualBox FreeBSD environment. That can sometimes be a useful way to get stuff onto the Soekris. Just be careful to be sure you’ve got the correct /dev/diskN configuration every time you re-insert the USB stick into your mac.

Revisit your NanoBSD conf file

I do recommend that after you’ve configured your system the usual “trial and error and iteration” way on the Soekris itself that you go back to the NanoBSD build environment and automate all your configuration changes into the build process itself. Between chroot hacks and just general cleverness (and taking advantage of all the infrastructure nanobsd.sh already provides) you should be able to automate whatever it is you’ve done interactively to set the box up. The advantage of building those changes into the build process is that they will happen automatically the next time you need to update from source for a new BSD release or any other reason.

Don’t forget to save away your .conf files and everything else you put into the VirtualBox environment!

 

Repairing a $30 network switch

One of my 8 port network switches stopped working. Before I threw the $30 switch away I decided to try to fix it. The symptom was “no lights at all” so I figured the power supply was the most likely culprit.

I broke out my multimeter and tested the wall wart; that was working fine.

So I decided to take the device apart and void my warranty on a $30 device I probably bought 10 years ago.

2013-10-17 15.38.09Step 1: Ignore this sticker.

 

 

 

 

 

The sticker provided an important clue as to how the device comes apart. Just pull/pry.

2013-10-17 15.45.51Step 2: Pry it open.

 

 

 

 

 

Removing the circuit board revealed one important fact: I had probably only bought this thing 6 years ago. Now we definitely need to fix it rather than toss it.

2013-10-17 15.46.55

Visual inspection of the circuit board didn’t provide any clues; no tell-tale signs of any “blue smoke” having been let out of any of the devices. Sometimes the solder joints on the circuit boards go bad, so in the hope that this was the problem I got out the heat gun.

2013-10-17 15.52.27

I blasted the power connector area with the gun on high, being careful not to burn anything important (like me). Here’s the after picture.

2013-10-17 15.53.09

It might have been my imagination, but the pads looked a little shinier to me after the heat gun. Time to plug it in and see if it explodes!

2013-10-17 15.51.32

Success! So I buttoned everything up and put the switch back into service.

2013-10-17 15.54.20

Saved $30 and kept another piece of equipment out of the landfill. Nerds with tools rock!