Proxmox is good software for a home datacenter. It’s an OS you install on server hardware that lets you easily run multiple virtual machines and LXC containers. It also manages disk storage and has some more complex support for high availability in a cluster, distributed storage via Ceph, etc. But even with a single small server running a single VM Proxmox offers advantages.

I’ve had a Linux server in my home for 20+ years now. Every few years I have to rebuild it, often from the ashes of failed hardware, and it’s always a tedious manual process. Now my server is truly virtualized, a nice tidy KVM/QEMU virtual machine with a disk I can snapshot and back up. And migrate an exact copy to new hardware in minutes.

Right now I’m mostly running my stuff in one big VM under Proxmox that I migrated from the old server. But I’m slowly moving services to separate VMs and LXC containers. So now my SMB server for Sonos lives in one container, and my Plex server in another, and my Unifi router manager in a third. All running isolated from each other. This feels tidier, more manageable.

Proxmox does a lot of nice things for home-scale servers. It handles ZFS for filesystems, including snapshots and backups. It has a nice web GUI for managing things, even graphical consoles where needed. And I like how it supports both VMs and containers as a first class things. There’s other ways to manage guest systems, like Docker (containers only) or VMware ESXi (proprietary, VMs only). Proxmox feels the right scale for me. I’ve spent about a month tinkering with it and like the software quite a bit. It’s usable, well documented, and seems well designed.

Starlink is good technology. I’m posting this blog entry from space. By which I mean Starlink, SpaceX’s new LEO satellite Internet service. I’ve been beta testing it for six months and using it exclusively for two. It is terrific. Some notes from a United States perspective.

Does it work well? Hell yeah! It’s more like having cable Internet than satellite. Over the last month my average bandwidth has been 100Mbits/s down, 12Mb/s up. Average ping to 8.8.8.8 of 41ms with 0.5% packet loss. Bandwidth is highly variable (50-200Mbps) but latency is pretty solidly 30-50ms. The main failure is occasional outages of ~10 seconds. That got a lot better mid-July with a Starlink change and will only get better as they launch more satellites. The service is still technically beta and there are some rough patches but it’s totally usable.

Is Starlink for you? Maybe. If you can get wired service (cable, fiber, faster DSL) that is probably a better choice. If you’re in a poorly served rural area in Ajit Pai’s America and you’re struggling with ViaSat or Hughes or using cellular, definitely. For me it’s an upgrade to my 12/1 Mbps fixed wireless service.

Will Starlink work at your house? Probably! It requires a clear view to the north. The free Starlink mobile app has an augmented reality tool to show you whether you have a good view. A few small obstructions are OK but if you live in the middle of a bunch of trees you need to go higher on your roof or get a mast.

Can you get it? Probably not soon. They are enormously back-ordered; Starlink has a limited amount of bandwidth per satellite and they are slow to add new users. They just passed 100,000 installs globally and are rumored to have 500,000+ customers on the waiting list. The best thing you can do is pre-order and put down a $99 deposit. It may be a year. (I got super lucky.)

Is it nerd friendly? Totally. You can use your own router; Starlink provides one but does not require it. Dishy has an open gRPC interface for getting detailed stats. The Internet service is quite solid and not messed with in any way I can tell. They sorta support IPv6 already and promise more. The main drawback is that (at least in IPv4) the service is cgNAT, you really can’t run a server behind Starlink in any reasonable way. The cgNAT is for good reason: they’re doing some very sophisticated routing, your packets may be relayed through several base stations hundreds of miles apart and it’s remarkable you have a stable external IP address at all.

Will Starlink succeed as a business? That’s hard to say. The program is still beta and currently has no bandwidth caps or significant throttles. And it’s $99/month: a lot by US urban ISP standards but competitive for rural areas. The problem is the satellites can only handle so many users and it seems too early to tell whether it works out to be profitable. Launching thousands of satellites is expensive but SpaceX are experts at that. Also crappy companies like ViaSat keep suing to stop Starlink because they can’t compete. Amazon is trying to make up for being years behind by trying to get the FCC to harass Starlink. The legal attacks seem to be failing so far in the US but you never know.

What’s next for Starlink? They launched their first full shell of satellites earlier this year, then took a pause. They seem to be fine-tuning algorithms and transceiver power settings right now. The next major change is a plan to use laser links so satellites can route packets directly (currently everything is relayed to the ground). It seems like a very hard problem but they are serious about doing it; ultimately Starlink might be better than wired service.

I’m not an Elon Musk fan but I have to say Starlink is amazing. And audacious; I never would have believed it would work (remember Iridium?). But it does work and it’s been a significant upgrade for me, so thank you SpaceX. The hilarious thing is the whole idea of Starlink apparently is about Mars; the project started as a way to design networking infrastructure for a colony on a new planet. Oh and then completely upended Earth’s ISP market as a sort of proof of concept.

The CyberPower CP350SLG is a good small uninterruptible power supply. Its only rated for 250W and it only has a few minutes of battery life. Not suitable for a big computer. But it’s perfect for backup power for network gear, like a router or a modem or the like. And it’s pretty small, just 7x4x3 inches. I made a mistake and bought APC’s small UPS first and the damn thing is ungrounded, which is ridiculous and dumb. I’ve had better luck with CyberPower UPSes anyway and this small one is exactly what I needed.

I’m a big fan of small UPSes. I don’t need something to carry me through a 30 minute power outage, I just want some backup that will keep my equipment running if the power drops for a couple of seconds. Because PG&E, you know? It’s a shame there’s no DC power standard, I bet you could make a DC-only UPS 1/4th the size with a lithium battery. But instead it’s all lead-acid batteries and producing 110V AC just to be transformed back to DC by all the equipment its powering. (That APC UPS does have powered USB ports, a small step towards DC UPS.)

Some day I should look into whole-house UPS units. A quick look suggests it’s about $2500 for 2.7kW, plus installation. This discussion suggests $10k is more realistic if you really mean a whole house.

The router is the most important computer in your home but most consumer routers are junk hardware with terrible software. For years my Linksys WRT54GL + Tomato firmware has been doing me well but Tomato hasn’t had an update in two years and the WRT54GL doesn’t do 802.11n.

The modern equivalent is an ASUS RT-N16 running Toastman’s Tomato build. Good stuff. The RT-N16 does 802.11n well and is overpowered hardware. The Toastman builds have all the goodness of stock Tomato along with nice features like USB drives and file serving. And my favorite feature, per-device network monitoring; perfect for figuring out what the heck is using all your bandwidth.

The stock ASUS firmware is garbage. Replacing it is a bit tricky; your choices are using their weird Windows software, installing a signed DD-WRT build first, or doing it by hand with tftp. I did the tftp trick and it worked fine. Unfortunately Toastman distributes his builds on a server that requires a login, but it does actually work. I used build tomato-K26USB 1.28.7500 4MIPSR2Toastman-RT-Ext.

Some alternatives.. The ASUS RT-N66U is fancier hardware that does 5GHz 802.11n for more wireless speed. But it’s about twice the price. The Shibby Tomato builds are also under active development and popular. And some people like OpenWRT or DD-WRT firmware; I prefer Tomato’s simplicity.

For our recent trip Ken and I took our iPhones. It's nice to stay in touch back home and I've become pretty reliant on my iPhone for getting around. Here's some tips on using an iPhone internationally for Americans stuck with AT&T. (There's a whole alternative of unlocking the phone and using a European SIM; not discussed here). See also AT&T's iPhone travel tips, they're pretty useful.

AT&T's roaming is pretty unreliable. Half the incoming phone calls I know about never arrived, not even to voicemail. Caller ID doesn't work. SMS messages disappear. Rates are outrageous, something like $2/minute. It's a bit cheaper if you set up "AT&T World Connect" on your account for $4/month before you leave. But who cares about the phone, you really want your iPhone as an Internet terminal. And it works pretty well for that in Europe, provided you either find free WiFi or else you don't mind paying through the nose.

If you don't buy the "Data Global Add-On" you will be paying $10/mb for data from the cell network (EDGE or 3G). That's absolutely hideous pricing. You can pre-order a chunk of 20, 50, 100, or 200 mb/month at roughly $1 / megabyte. That's still outrageous but bearable. Note: you can order the data for just a few weeks and cancel. Overage is $5/mb.

How much bandwidth do you need? If I was being careful, it was 2 mb/day. That was enough bandwidth to catch up on email and Twitter twice a day, maybe get a couple of web pages or upload a photo. Then I had one bad day where I looked up a few maps and restaurants and blew through 10 megabytes. You really have to be careful.

How do you be careful? #1 way is to find free WiFi. There's a lot of free WiFi in Europe, more-so than the US. Public town squares, train stations, and busy cafes are good bets. Sometimes the cafe WiFi is password protected, just ask and they'll give you a password. Hotels are hit and miss. A lot of my rooms only had wired Internet, I regretted not bringing a little WiFi router.

If you're stuck with using cell for data, the #1 option for being careful is to turn off "Data Roaming" in Settings. If you do that, you'll use no bandwidth. You'll also not be able to use the Internet. Geolocation won't work well either. I found it really irritating to have Data Roaming off all the time. So instead I configured the phone to be lean by turning off Notifications and "Mail / Fetch New Data" in Settings. I also reset my Usage counters to track what I was using in Europe.

iPhone apps will still use a lot of bandwidth when you launch them. The real killer is Google Maps, a total hog. OffMaps is a nice alternative with cached maps in offline mode. Expect to pre-load 500-1000mb of maps for a long trip, you really want the finest grain detail when walking around. Apple's Mail app is reasonably network efficient, particularly compared to Gmail in Safari. The New York Times app is great for caching a bunch of news articles on WiFi then reading them later. And Twitter is a great low bandwidth way to keep in touch with people, although apps like Tweetie are not particularly network efficient.

AT&T's data roaming charges are ridiculous and it's a real pain watching your bandwidth. But it's totally worth it, the iPhone makes a great little companion while travelling. I particularly liked being able to use Twitter as a travel diary complete with photos, it was a lot of fun seeing responses from people to what I was doing that day.

Every network port on your computer has a MAC address, a unique 48 bit identifier. It's a bit like an IP address but lower level; your wireless or ethernet delivers Internet packets to you by knowing your MAC address.

It's very important that every computer on a network has a unique MAC address. So important, that all network hardware has a unique ID burned into its firmware (one of only two sources of standard unique bits on a typical PC). However, it's common for routers to support "MAC address cloning", where your router impersonates your computer when talking to your ISP. That feature was placed there to work around inflexible ISP networks and network policies, and it's mostly useful.

But MAC address cloning can be quite harmful, as I learned today. See, I cloned my laptop's MAC address to my router. Then two years ltaer I cloned my laptop's MAC address to my new router, too, in my new house three miles away. Miraculously this worked fine for a year, until this morning. When my network connection would go down at random intervals. I'm guessing the layer 2 stuff wasn't broken by the duplicate MAC address but rather it confused some DHCP housekeeping in my ISP's network management back office.

Three cheers to my wonderful ISP, sonic.net, for helping me figure out this bizarre problem. I love that when I call them I get a tech who will happily discuss DHCP leases, MAC addresses, and non-standard router firmware wtih me. They were as mystified as I was at first, but talking it over at the phone we figured out something was going bad with address assignment. No way either of us could have figured this out without working together.

A few weeks ago I asked for advice on Vista or XP for a new computer. Responses were mixed, but most of the people I talked to who actually use Vista are fine with it. So I got a new Vista 64 box and after 24 hours it seems to be working great. Sure is nice having fast new hardware.

I'm long past the point where new computers are fun; it's just a chore to set one up. But I spend all day in front of a computer, so fast hardware is nice. And the software experience is like a home to me, my environment. So I care a fair amount about the details. Here they are.

More inside ...

The Tomato firmware for routers seems to be good software. It's a simple replacement firmware for the Linksys WRT54GL and a few other Broadcom based routers. It's tightly focussed on being a good router, nothing more. And it has decent documentation.

The main feature it has over most routers is Quality of Service routing. The default configuration once you turn it on gives priority to DNS requests, small Web requests, and outbound ACKs. That last feature is really important; it should solve problem that a single upload totally kills all downloads on an asynchronous connection like residential DSL. I also tweaked it to give ssh and ntp priority.

The other main Tomato feature is fancy bandwidth graphing. Overkill for me, but cool. And the firmware has lots of knobs to turn; I like being able to tweak the NAT connection tracking.

Tomato isn't going to solve my problem of flaky router hardware, but at least the software may work sensibly. The QoS feature should be a significant improvement. Hopefully some day a consumer router manufacturer will get smart and build QoS in to their products.

My Netgear FWG114P router has failed three times in the last two days, requiring a reboot. Last time was locked up dead, couldn't even ping the router over the LAN. It's working now, but in my experience once these things start failing they have a week before they die entirely.

My friends now make fun of me for how often I complain about routers on my blog. But WTF? Why do all my routers die in a few months? I'm not doing anything that strange with them. My only thought is that the 80° server closet they're in is causing heat problems. But that's nowhere near out of the 104° spec.

I'd thought paying extra for Netgear blue box hardware would help, but I guess not. My next option is to either go with a Draytek 2800G at $200, hoping for reliability, or else go back to a junky Linksys router with the hopes that Tomato firmware will make it work reliably.

The iPhone has a pretty nifty WiFi implementation. It worked great in my house with my cheap Belkin router for about 15 minutes, then never worked again. The iPhone thought it had a connection but nothing would load. Then I switched routers at home to the Netgear FWG114P and now the iPhone Wifi is fine.

Was the problem Apple's or Belkin's? Was it the ARP implementation, or the DHCP lease maintenance, or the WEP-128 implementation, or the 802.11g switching magic? How could I possibly know? Wifi is more voodoo than science and the complexity of Internet protocols is so high I'd never be able to figure out the specific problem. And if I did, I couldn't fix it anyway; all the firmware is closed. So I just keep going through routers until I find one that works.