pfsense router – almost 1 year uptime

I’m running a pfsense router on a (somewhat obsolete now but still serviceable) dedicated soekris box as the router for my (40 acre) hilltop “empire”. I have a somewhat complicated network topology: my Time Warner cable modem is in a building at the bottom of my hill but my house is uphill about a quarter-mile (as the wires run) away. I have a multi-mode fiber connection between the cable modem downhill and the router uphill, from which four internal networks emanate:

  • HILLTOP: my internal network.
  • GARAGE: runs back down the hill (on another fiber pair) to a separate maintenance equipment garage building.
  • DMZ: a true, isolated, DMZ topology for a few servers I want to allow access to from the network at large.
  • PUBLICWIFI: a no-password open-WiFi network for my guests; it is also appreciated by maintenance personnel working on stuff up here on the hill. Access control for this otherwise-open WiFi access point is provided primarily by the fact that the signals don’t reach the edge of my property line (at least in all the places I’ve tested for this). If you can get the signal the presumption is you should probably be allowed to surf the internet on my dime 🙂

The pfsense software makes it easy to configure these networks with appropriate firewall rules; for example no traffic is routed between the PUBLICWIFI network and any of the other networks (other than the WAN network to the outside world of course).

About a year ago I put the router on a UPS just because the mean time between power failures up here is about 2-3 months. Anecdotally, thunderstorms are the primary source of periodic, short, power outages/glitches.

Here’s the network statistics report:

Statistics for the hilltop network

As always, you may wish to click to view the image full size.

During the 322 days of uptime:

  • 2.5 terabytes have arrived at my router from the internet. That’s about 7-8GB per day, most of which is presumably netflix or porn (for network tests of course).
  • Of the 2.5TB, 2TB went to the house network and 0.5TB went to the maintenance building where my property manager works.
  • The DMZ served out 5.6GB (a whopping 17MB per day). There are some status servers on this network that my property manager and I periodically surf to check in on things if I’m not here.
  • The public WiFi pulled 20GB down from the internet (about 62MB/day). The maintenance workers don’t seem to surf very much porn 🙂

I had a 12-pair multi-mode fiber installed to run up/down the hill and I am using two of the pairs. One pair runs from the cable modem up to my router. Another one runs back down the hill to the GARAGE to carry the (internal) network back down into the maintenance building. These runs are each about 1300 feet, or about 400 meters. The fiber transceivers I’m using (StarTech MCMGBSC055) are supposed to be good for 550m at 1Gb and appear to be functioning well. There are zero errors across the board except for one output error recorded in the entire year. My guess (and it is purely a guess) is that the output error occurred during a power failure. My router is on a UPS but the transceivers are not, so a power failure that happens during a packet transmission might show up as an output error. Otherwise I’m at a loss to explain how there could be an *output* error detected by the router; I’m guessing what really happened is that the ethernet connection between the router and the fiber transceiver went down when the power failed in the middle of a packet transmission. The router is on the UPS but other network gear is not; the only point of the UPS here is to shield the router from the strain of unnecessary power-glitch reboots; it is not intended to keep my network up during outages.

Knowing a little bit about the waveforms and the technology that makes all this stuff work, I am always impressed that stuff like this actually *does* work.

So far the 1Gb link is faster than my cable modem connection. I’m not going to be able to go to 10Gb without laying new (single mode) fiber. The fiber run is in a conduit with periodic access/pull points, so supposedly this will be possible, but it won’t be especially cheap or easy. Since I have some spare pairs I may be able to get more bandwidth in the future with link aggregation; I’ll worry about that technical problem when/if my internet connection exceeds 1Gbps from the ISP (when is google fiber coming here?!!!)