Overused services on abused servers
Things I’ve learned while running a public BitTorrent tracker that handles about 66k torrents and, on average, about 70k peers. On a 7 year old desktop machine turned server, nonetheless.
Load averages can get up to the high three hundreds, heh:
19:22:59 up 11 days, 1:56, 1 user, load average: 380.23, 384.55, 292.47
Once you’ve reached a critical efficiency level in software, the hardware will screw you over: went from handling 15-20 requests/sec via a PHP and flat-file based tracker (lighttpd + fastcgi + php-cgi) to handling around 200 reqs/sec and 600 live connections at any moment with xbtt reverse proxied from lighttpd. But the line was so saturated that I was seeing latencies of 16-20s on connections. And when you’re running on 256 MBs of RAM, … well, welcome to swap hell.
Although never explicitly mentioned anywhere, lighttpd can use the source IP as a parameter in a rewrite. It’s just a question of doing the rewrite inside a regex IP address conditional:
$HTTP["remoteip"] =~ "(.*)" {
url.rewrite-once = ( "\/target\?(.*)$" => "/target?ip=%1&$1" )
}
The %1 parameter is the match of the $HTTP[“remoteip”] check, while the $1 parameter is the match of the rewrite check itself. So you can basically reproduce nginx’s $remote_addr in rewrites, and then use it for reverse proxying, thus passing the original IP to the target webserver as a GET parameter, not just as a header.
Unless IPv6 is enabled. Because apparently if it is, you can’t just do matching against $HTTP[“remoteip”], probably because IPs show up as v4-mapped IPv6s (::ffff:127.0.0.1). And as I’m an IPv6 enthusiast, that’s a no-no.
Consumer-grade routers suck. I’ve been blaming that NetGear router for a while for dropping connections like mad. How is it that a shitty 7 year old system with a load in the hundreds which actually processes requests can still handle more connections than a one year old router?! And it isn’t exactly one of the cheap ones either.
Praise be to lighttpd! Quirky, buggy software, might exchange it for nginx one day. But for now, it’s everything I need and keeps everything running smooth.
And in the end, when all was said and done, I decided that running a BitTorrent tracker is too much hassle for no reward whatsoever, I’ve already gleaned all the knowledge I could from this experience and I had simply become bored with it. So I just CNAMEd tracker.token.ro to tracker.publicbt.com - let them handle all those thousands of peers! That’s what they’re there for.
