Perl script to grab the fastest (by latency) OpenBSD HTTP mirrors and return the top {n}

I've recently had some frustration with a particular server when doing package installations on OpenBSD, so I wrote a quick snippet of Perl to grab the FTP list from the OpenBSD main page, then parse through the list and "ping" via TCP/80 every server to get the response time.  The fastest {n} responses are returned.

#!/usr/bin/perl
use strict;
use warnings;
#
# ONLY OpenBSD modules included with the default installation are permitted!
#
use HTTP::Tiny;
use Net::Ping;
use Time::HiRes;

# Configuration hash (editable)
my %config = (
  'timeout'  => 1.0,    # How long to wait before giving up on server response, in seconds
  'debug'    => 1,      # Debug output?
  'top'      => 5,      # How many entries to return?
  'protocol' => 'tcp',  # tcp, udp, icmp, et al
  'port'     => 'http', # (used in getservbyname sub)
);

# hash to store servers and response times to protocol/port requests
my %serverstats;

# create HTTP::Tiny instance
my $http = HTTP::Tiny->new;

# Get a list of all current OpenBSD FTP servers
my $response = $http->get('http://ftp.openbsd.org/pub/OpenBSD/ftplist');
die "Failed!\n" unless $response->{success};

# Iterate through server list and get TCP/80 response time in ms
foreach my $line (split("\n", $response->{content})) {
  if ($line =~ /(http:\/\/)(.+?)(\/\S+)/) {
    my $response = &httping($2);
    $serverstats{$1.$2.$3} = $response if ($response);
  }
}

# Sort & print servers by response time in ms
my $i = 0;
print ("\n");
foreach my $key (sort {$serverstats{$a} <=> $serverstats{$b} } keys %serverstats) {
  $i++;
  print "$key\n";
  last if $i eq $config{'top'};
}

#
# Ping TCP/80 and return response time or 0 if unresponsive + some diagnostic output
#

sub httping ($) {
  my $host = shift;
  my $ping = Net::Ping->new($config{'protocol'});
  $ping->hires();
  $ping->{port_num} = getservbyname($config{'port'}, $config{'protocol'});
  print ("Trying $host... ") if $config{'debug'};
  my ($retval, $duration, $ip) = $ping->ping($host, $config{'timeout'});
  $ping->close();
  $duration = int($duration * 1000);

  if ($retval) {
    print ("$duration ms\n") if $config{'debug'};
    return $duration;
  } else {
    print ("unresponsive\n") if $config{'debug'};;
    return 0;
  }
}

AND, the output, when debug is enabled:

# ./rtservermon.pl                                                                                                         
Trying mirror.internode.on.net... 238 ms
Trying mirror.aarnet.edu.au... 236 ms
Trying ftp.iinet.net.au... 283 ms
Trying ftp5.eu.openbsd.org... 165 ms
Trying ftp2.eu.openbsd.org... 152 ms
Trying mirror.telepoint.bg... 169 ms
Trying ftp.OpenBSD.org... 92 ms
Trying athena.caslab.queensu.ca... 71 ms
Trying openbsd.cs.toronto.edu... 64 ms
Trying mirrors.ucr.ac.cr... 127 ms
Trying mirror.steadynet.cz... 172 ms
Trying ftp.openbsd.dk... 169 ms
Trying mirror.one.com... 153 ms
Trying ftp.aso.ee... 176 ms
Trying ftp.estpak.ee... 167 ms
Trying ftp.fr.openbsd.org... 133 ms
Trying ftp2.fr.openbsd.org... 144 ms
Trying mirrors.ircam.fr... 135 ms
Trying openbsd.cs.fau.de... 146 ms
Trying ftp.spline.de... 159 ms
Trying ftp.bytemine.net... 155 ms
Trying ftp.halifax.rwth-aachen.de... 160 ms
Trying artfiles.org... 150 ms
Trying ftp.hostserver.de... 151 ms
Trying ftp.cc.uoc.gr... 188 ms
Trying ftp.fsn.hu... 151 ms
Trying kartolo.sby.datautama.net.id... 263 ms
Trying mirror.poliwangi.ac.id... unresponsive
Trying ftp.heanet.ie... 149 ms
Trying openbsd.mirror.garr.it... 161 ms
Trying ftp.jaist.ac.jp... 191 ms
Trying www.ftp.ne.jp... 168 ms
Trying ftp.nluug.nl... 146 ms
Trying ftp.bit.nl... 150 ms
Trying stingray.cyber.net.pk... 265 ms
Trying piotrkosoft.net... 190 ms
Trying ftp.icm.edu.pl... 163 ms
Trying mirror.yandex.ru... 187 ms
Trying mirrors.isu.net.sa... 218 ms
Trying www.obsd.si... unresponsive
Trying mirror.is.co.za... 303 ms
Trying mirror.codigo23.net... 221 ms
Trying ftp.eu.openbsd.org... 166 ms
Trying mirror.switch.ch... 150 ms
Trying ftp.ch.openbsd.org... 154 ms
Trying www.mirrorservice.org... 152 ms
Trying mirror.bytemark.co.uk... 131 ms
Trying mirror.ox.ac.uk... 154 ms
Trying mirror.exonetric.net... 144 ms
Trying ftp5.usa.openbsd.org... 74 ms
Trying mirrors.sonic.net... 106 ms
Trying ftp3.usa.openbsd.org... 53 ms
Trying openbsd.mirrorcatalogs.com... 782 ms
Trying mirrors.syringanetworks.net... 86 ms
Trying mirrors.gigenet.com... 46 ms
Trying mirror.team-cymru.org... 47 ms
Trying filedump.se.rit.edu... 62 ms
Trying mirrors.nycbug.org... 61 ms
Trying openbsd.mirror.frontiernet.net... 53 ms
Trying mirror.esc7.net... 127 ms
Trying mirror.jmu.edu... 65 ms

http://mirrors.gigenet.com/pub/OpenBSD
http://mirror.team-cymru.org/pub/OpenBSD
http://openbsd.mirror.frontiernet.net/pub/OpenBSD
http://ftp3.usa.openbsd.org/pub/OpenBSD
http://mirrors.nycbug.org/pub/OpenBSD

PHP Installation on OpenBSD 5.7 relayd-based httpd

In an effort to return to my OpenBSD roots, I've been planning a migration from the licensing conflagration that is pfsense (more on that later).  One of the features I've been keenly watching across the past 3-4 releases of OpenBSD was that their fork of Apache 1.3 was to be dropped in favor of nginx.  It was subsequently decided that nginx was too bloated (that was quick!), so a new http daemon was borne from relayd.

Given that I have some requirement for PHP, I thought I'd write a quick blurb on how to perform a fast, basic installation.

The new httpd supports a similar architecture as nginx using a unix socket with which to communicate to php-fpm via fastCGI.

Let's assume you have a fresh 5.7 installed.  Let's also assume you're root, though we all know the dangers of acting with unilateral authority as root.

First, let's get httpd running by creating a configuration file.  Create your /etc/httpd.conf as follows (this is very minimalistic):

# cat < /etc/httpd.conf
> server "default" {
>         listen on egress port 80
>         location "*.php" {
>                 fastcgi socket "/run/php-fpm.sock"
>         }
> }
> EOF

Secondly, we'll set a package path.  I chose rit.edu's mirror - you can select your own from OpenBSD's list.

# export PKG_PATH=http://filedump.se.rit.edu/pub/OpenBSD/snapshots/packages/`machine -a`/ 

Next, add the php-fpm package.  I'm forcing the 5.4.38, but you can select any version you like.  The -I switch disables the interactive prompts (assuming defaults) and the -v switch enables more verbosity.

# pkg_add -I -v php-fpm-5.4.38

You'll see a jumble of output, but it should eventually install.

I then created /var/www/htdocs/phpinfo.php to test for PHP's functionality.

# cat < /var/www/htdocs/phpinfo.php
>
> EOF

Lastly, I started php_fpm as well as httpd.  Note the -f on httpd may be necessary - I'm forcing it for the demonstration, but ultimately it's up to you.

# /etc/rc.d/php_fpm start
php_fpm(ok)

# /etc/rc.d/httpd -f start
httpd(ok)

At this point, you should be ready to test your installation of OpenBSD's new httpd + php by visiting your site at http://127.0.0.1/phpinfo.php (if you're local to the machine).

Enjoy!

---

P.S. To run Perl in CGI on the new httpd, you'll need to do a little extra!

Copy the necessary binary and its accompanying libraries to their respective directories in the /var/www root: (don't forget to create any necessary directories first!)

# mkdir -p /var/www/usr/{bin,lib,libexec}
# cp /usr/bin/perl /var/www/usr/bin
# cp /usr/lib/libpthread.so.18.1 /var/www/usr/lib
# cp /usr/lib/libperl.so.17.0 /var/www/usr/lib
# cp /usr/lib/libm.so.9.0 /var/www/usr/lib
# cp /usr/lib/libutil.so.12.1 /var/www/usr/lib
# cp /usr/lib/libc.so.78.1 /var/www/usr/lib
# cp /usr/libexec/ld.so /var/www/usr/libexec


Don't forget to set ownership/permissions!

# chown www:www /var/www/usr/bin/perl \
# /var/www/usr/lib/{libpthread.so.18.1,libperl.so.17.0,libm.so.9.0,libutil.so.12.1,libc.so.78.1} \
# /var/www/usr/libexec/ld.so

# chmod 550 /var/www/usr/bin/perl

# chmod 440 /var/www/usr/lib/{libpthread.so.18.1,libperl.so.17.0,libm.so.9.0,libutil.so.12.1,libc.so.78.1} \
# /var/www/usr/libexec/ld.so

Then be sure to add a section in your /etc/httpd.conf to handle your CGI, here's a simple entry:

location "*.shtml" {

  fastcgi socket "/run/slowcgi.sock"

}

Start slowcgi:

# /usr/sbin/slowcgi

Restart httpd:

# /etc/rc.d/httpd -f restart

httpd(ok)

httpd(ok)

Your server should be serving CGI now (using the Perl included with OpenBSD). Enjoy!