Category Archives: Systems Administration

Using persistent OpenSSH connections

I found out that using persistent connections greatly improves the productivity when working with SSH. However, finding the appropriate configuration turned out to be a complicated task. I wanted it to be as unobtrusive as possible, to restart the connection when the socket is closed, and to work without blocking timeouts.

After reading the ssh_config man page and some articles, here’s the best thing I came up with:

Host *
	ControlPath ~/.ssh/master-%r@%h:%p
	ControlMaster auto
	ControlPersist 4h
	TCPKeepAlive no
	GSSAPIAuthentication no
	ServerAliveInterval 60
	ServerAliveCountMax 2

The only issue with this configuration is with long hosts (eg: a really long name) as it hits the UNIX_PATH_MAX limit. Unfortunately, the proper solution to this issue isn’t merged into upstream.

The OS X users who also use brew may easily include the patch for the path issue by editing the openssh formula for OpenSSH 6.6p1 with “brew edit openssh”:

  patch do
    url "http://www.mirrorservice.org/sites/downloads.sourceforge.net/m/ma/mancha/misc/openssh-6.6p1-mux-hash.diff"
    sha1 "31f6df29ff7ce3bc22ba9bad94abba9389896c26"
  end

With this patch, a value like ~/.ssh/master-%m works for ControlPath. %m is replaced by SHA1(lhost(%l) + rhost(%h) + rport(%p) + ruser(%r)) and it keeps things short and sweet.

Getting a HTTPS certificate information into the shell

Due to the HeartBleed SNAFU, I needed a quick solution for getting the information from a certificate deployed on a remote machine. As I rarely leave the comfort of my terminal, as always, I simply dumped a new function into the shell’s ~/.*rc file.

Here it is:

Defaults to port 443 if the second argument is unspecified. Example:

get_cert google.com
 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            59:fa:65:0e:26:a1:67:3d
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, O=Google Inc, CN=Google Internet Authority G2
        Validity
            Not Before: May  7 12:15:37 2014 GMT
            Not After : Aug  5 00:00:00 2014 GMT
        Subject: C=US, ST=California, L=Mountain View, O=Google Inc, CN=*.google.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                    00:c2:9a:87:d1:79:0a:10:28:64:f3:d7:12:48:93:
                    13:24:c9:05:9e:1b:94:0d:b1:d6:02:54:27:e2:a4:
                    87:45:ab:f8:17:19:db:0d:b0:a9:80:34:a1:2a:5e:
                    98:a7:85:a6:66:2b:69:5c:85:16:fd:43:9f:6f:40:
                    f2:36:d8:47:4f:16:cd:ef:f4:67:75:c2:07:89:fa:
                    37:c8:c3:08:37:0b:ec:e3:61:48:86:86:bd:7b:5d:
                    cc:10:96:9d:be:07:e1:c3:e2:c1:23:04:fa:a6:93:
                    99:b5:42:7c:55:ab:91:6b:8a:d4:bf:8d:23:df:9d:
                    4d:96:a7:31:e0:f7:04:39:db:66:d3:d5:64:36:1f:
                    ef:71:af:df:0b:86:d9:6a:fc:12:c4:8c:94:fe:91:
                    6f:d4:6c:c0:f8:ec:68:b0:7e:fd:71:42:43:42:34:
                    ad:a2:fb:3e:12:98:68:d1:b4:23:e6:7d:8a:75:9c:
                    c1:82:bb:95:55:28:15:50:9e:d0:49:21:b7:6f:ce:
                    44:c1:3d:9a:e3:05:28:96:0c:53:44:68:6b:63:b4:
                    f1:4a:79:3d:09:7d:f0:96:e6:78:95:36:aa:89:40:
                    d8:9c:60:f9:b1:1c:23:d5:6c:b7:2d:f4:e4:00:ff:
                    7b:9b:f2:02:43:c1:d2:e1:95:2c:a1:41:d4:88:72:
                    0b:69
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Alternative Name:
                DNS:*.google.com, DNS:*.android.com, DNS:*.appengine.google.com, DNS:*.cloud.google.com, DNS:*.google-analytics.com, DNS:*.google.ca, DNS:*.google.cl, DNS:*.google.co.in, DNS:*.google.co.jp, DNS:*.google.co.uk, DNS:*.google.com.ar, DNS:*.google.com.au, DNS:*.google.com.br, DNS:*.google.com.co, DNS:*.google.com.mx, DNS:*.google.com.tr, DNS:*.google.com.vn, DNS:*.google.de, DNS:*.google.es, DNS:*.google.fr, DNS:*.google.hu, DNS:*.google.it, DNS:*.google.nl, DNS:*.google.pl, DNS:*.google.pt, DNS:*.googleapis.cn, DNS:*.googlecommerce.com, DNS:*.googlevideo.com, DNS:*.gstatic.com, DNS:*.gvt1.com, DNS:*.urchin.com, DNS:*.url.google.com, DNS:*.youtube-nocookie.com, DNS:*.youtube.com, DNS:*.youtubeeducation.com, DNS:*.ytimg.com, DNS:android.com, DNS:g.co, DNS:goo.gl, DNS:google-analytics.com, DNS:google.com, DNS:googlecommerce.com, DNS:urchin.com, DNS:youtu.be, DNS:youtube.com, DNS:youtubeeducation.com
            Authority Information Access:
                CA Issuers - URI:http://pki.google.com/GIAG2.crt
                OCSP - URI:http://clients1.google.com/ocsp
 
            X509v3 Subject Key Identifier:
                2B:56:D4:98:8E:81:28:99:CD:17:89:09:21:EB:3B:8B:EF:7E:19:A0
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier:
                keyid:4A:DD:06:16:1B:BC:F6:68:B5:76:F5:81:B6:BB:62:1A:BA:5A:81:2F
 
            X509v3 Certificate Policies:
                Policy: 1.3.6.1.4.1.11129.2.5.1
 
            X509v3 CRL Distribution Points:
                URI:http://pki.google.com/GIAG2.crl
 
    Signature Algorithm: sha1WithRSAEncryption
        27:35:81:4e:df:79:e9:c7:9c:c1:5b:9c:35:4e:67:00:de:38:
        cb:a0:2f:58:91:61:11:a3:cf:ae:49:63:84:76:74:20:43:35:
        7c:e7:82:3e:7f:43:c8:94:71:9d:33:72:cc:3c:3e:0f:97:00:
        ef:08:65:7c:cc:e3:32:ca:16:b4:fb:73:7c:43:b2:eb:47:2d:
        3b:b9:b2:c8:4b:1a:ca:77:d0:65:55:fc:1c:76:7e:6d:86:d9:
        2f:e5:7b:63:5c:5f:77:9e:75:39:4f:36:0a:c2:8a:35:dc:f7:
        02:cb:03:c6:17:bb:2f:03:a1:de:12:c9:ab:03:ce:6d:17:0e:
        cb:78:24:e4:36:13:0e:e7:12:e5:e2:84:42:cc:d3:aa:32:b7:
        78:07:ae:24:23:69:be:90:9c:d3:38:51:05:5a:69:05:be:e3:
        8d:7b:ae:2b:37:c1:35:3a:b5:51:1f:46:fe:10:a7:ce:af:d2:
        b6:a5:8f:13:a5:57:03:63:25:0e:bc:6e:c7:e5:7b:22:4e:ff:
        67:9d:15:30:93:21:c1:08:03:6f:ab:5a:33:d8:41:c2:2a:8e:
        5a:a9:67:26:6e:6a:c1:46:8e:50:e7:4e:c7:51:66:eb:0b:ef:
        9d:c9:6a:d2:7f:a9:25:89:c2:28:aa:e5:fd:e4:74:9b:95:32:
        5d:15:ed:d0

portspoof trolling

Marius once told me about portspoof. A service to troll those who use various scanners by feeding the scanners with false results. Well, while the idea is good, I’m wary about a service like this as this is the kind of service where you wouldn’t want a buffer overflow.

Giving it a run inside a VM, I noticed something odd when using nmap’s service and version detection probes. This happened on the lower ports (1-50). Then I started to look at something that started to look like a pattern, therefore I increased the port range to include 1-50. portspoof is indeed a tool that trolls baddies and pen testers.

Ran it with:

nmap -sV --version-all -p 1-50
1/tcp  open  smtp    Unrecognized SMTP service (12345 0000000000000000000000000000000000000000000000000000000)
2/tcp  open  smtp    Unrecognized SMTP service (12345 0000000000000000000000000000000000000000000000000000000)
3/tcp  open  smtp    Unrecognized SMTP service (12345 0000000000000000000000000000000000000000000000000000000)
4/tcp  open  smtp    Unrecognized SMTP service (12345 0000000000000000000000000000000000000000000000000000000)
5/tcp  open  smtp    Unrecognized SMTP service (12345 0000000000000000000000000000000000000000000000000000000)
6/tcp  open  smtp    Unrecognized SMTP service (12345 0ffffffffffffffffffffffffffffffffffffffffffffffffffff00)
7/tcp  open  smtp    Unrecognized SMTP service (12345 0fffffffffffff777778887777777777cffffffffffffffffffff00)
8/tcp  open  smtp    Unrecognized SMTP service (12345 0fffffffffff8000000000000000008888887cfcfffffffffffff00)
9/tcp  open  smtp    Unrecognized SMTP service (12345 0ffffffffff80000088808000000888800000008887ffffffffff00)
10/tcp open  smtp    Unrecognized SMTP service (12345 0fffffffff70000088800888800088888800008800007ffffffff00)
11/tcp open  smtp    Unrecognized SMTP service (12345 0fffffffff000088808880000000000000088800000008fffffff00)
12/tcp open  smtp    Unrecognized SMTP service (12345 0ffffffff80008808880000000880000008880088800008ffffff00)
13/tcp open  smtp    Unrecognized SMTP service (12345 0ffffffff000000888000000000800000080000008800007fffff00)
14/tcp open  smtp    Unrecognized SMTP service (12345 0fffffff8000000000008888000000000080000000000007fffff00)
15/tcp open  smtp    Unrecognized SMTP service (12345 0ffffff70000000008cffffffc0000000080000000000008fffff00)
16/tcp open  smtp    Unrecognized SMTP service (12345 0ffffff8000000008ffffff007f8000000007cf7c80000007ffff00)
17/tcp open  smtp    Unrecognized SMTP service (12345 0fffff7880000780f7cffff7800f8000008fffffff80808807fff00)
18/tcp open  smtp    Unrecognized SMTP service (12345 0fff78000878000077800887fc8f80007fffc7778800000880cff00)
19/tcp open  smtp    Unrecognized SMTP service (12345 0ff70008fc77f7000000f80008f8000007f0000000000000888ff00)
20/tcp open  smtp    Unrecognized SMTP service (12345 0ff0008f00008ffc787f70000000000008f000000087fff8088cf00)
21/tcp open  smtp    Unrecognized SMTP service (12345 0f7000f800770008777000000000000000f80008f7f70088000cf00)
22/tcp open  smtp    Unrecognized SMTP service (12345 0f8008c008fff8000000000000780000007f800087708000800ff00)
23/tcp open  smtp    Unrecognized SMTP service (12345 0f8008707ff07ff8000008088ff800000000f7000000f800808ff00)
24/tcp open  smtp    Unrecognized SMTP service (12345 0f7000f888f8007ff7800000770877800000cf780000ff00807ff00)
25/tcp open  smtp    Unrecognized SMTP service (12345 0ff0808800cf0000ffff70000f877f70000c70008008ff8088fff00)
26/tcp open  smtp    Unrecognized SMTP service (12345 0ff70800008ff800f007fff70880000087f70000007fcf7007fff00)
27/tcp open  smtp    Unrecognized SMTP service (12345 0fff70000007fffcf700008ffc778000078000087ff87f700ffff00)
28/tcp open  smtp    Unrecognized SMTP service (12345 0ffffc000000f80fff700007787cfffc7787fffff0788f708ffff00)
29/tcp open  smtp    Unrecognized SMTP service (12345 0fffff7000008f00fffff78f800008f887ff880770778f708ffff00)
30/tcp open  smtp    Unrecognized SMTP service (12345 0ffffff8000007f0780cffff700000c000870008f07fff707ffff00)
31/tcp open  smtp    Unrecognized SMTP service (12345 0ffffcf7000000cfc00008fffff777f7777f777fffffff707ffff00)
32/tcp open  smtp    Unrecognized SMTP service (12345 0cccccff0000000ff000008c8cffffffffffffffffffff807ffff00)
33/tcp open  smtp    Unrecognized SMTP service (12345 0fffffff70000000ff8000c700087fffffffffffffffcf808ffff00)
34/tcp open  smtp    Unrecognized SMTP service (12345 0ffffffff800000007f708f000000c0888ff78f78f777c008ffff00)
35/tcp open  smtp    Unrecognized SMTP service (12345 0fffffffff800000008fff7000008f0000f808f0870cf7008ffff00)
36/tcp open  smtp    Unrecognized SMTP service (12345 0ffffffffff7088808008fff80008f0008c00770f78ff0008ffff00)
37/tcp open  smtp    Unrecognized SMTP service (12345 0fffffffffffc8088888008cffffff7887f87ffffff800000ffff00)
38/tcp open  smtp    Unrecognized SMTP service (12345 0fffffffffffff7088888800008777ccf77fc777800000000ffff00)
39/tcp open  smtp    Unrecognized SMTP service (12345 0fffffffffffffff800888880000000000000000000800800cfff00)
40/tcp open  smtp    Unrecognized SMTP service (12345 0fffffffffffffffff70008878800000000000008878008007fff00)
41/tcp open  smtp    Unrecognized SMTP service (12345 0fffffffffffffffffff700008888800000000088000080007fff00)
42/tcp open  smtp    Unrecognized SMTP service (12345 0fffffffffffffffffffffc800000000000000000088800007fff00)
43/tcp open  smtp    Unrecognized SMTP service (12345 0fffffffffffffffffffffff7800000000000008888000008ffff00)
44/tcp open  smtp    Unrecognized SMTP service (12345 0fffffffffffffffffffffffff7878000000000000000000cffff00)
45/tcp open  smtp    Unrecognized SMTP service (12345 0ffffffffffffffffffffffffffffffc880000000000008ffffff00)
46/tcp open  smtp    Unrecognized SMTP service (12345 0ffffffffffffffffffffffffffffffffff7788888887ffffffff00)
47/tcp open  smtp    Unrecognized SMTP service (12345 0ffffffffffffffffffffffffffffffffffffffffffffffffffff00)
48/tcp open  smtp    Unrecognized SMTP service (12345 0000000000000000000000000000000000000000000000000000000)
49/tcp open  smtp    Unrecognized SMTP service (12345 0000000000000000000000000000000000000000000000000000000)
50/tcp open  smtp    Unrecognized SMTP service (12345 0000000000000000000000000000000000000000000000000000000)

Really smooth guys, really smooth. Sometimes you have to see the big picture:
big-picture

Splitting a string every nth char in shell

I needed some reusable stuff that splits a string every nth char. Then I remembered that bash and zsh, the shells that I usually use, support string slicing. Kinda like Python does. Or the other way around. Made a shell function. Dropped in into .bashrc / .zshrc. Enjoy.

function string_split()
{
	str="$1"
	count=$2
	while [ ! -z "$str" ]
	do
		echo "${str:0:$count}"
		str="${str:$count}"
	done
}

Example:

string_split abcd 2
ab
cd

Use the cache, Luke, Part 2: don’t put all your eggs into the memcached buck … basket

This is the second part of a series called: Use the cache, Luke. If you missed the first part, here it is: From memcached to Membase memcached buckets. Meanwhile, the AWS ElastiCache service proved to have better network latency than our own rolled out Membase setup, therefore the migration was easily done by simply switching the memcached config. No vendor lock in.

However, it took me a while to write this second part.

If you can see this, then you might need a Flash Player upgrade or you need to install Flash Player if it’s missing. Get Flash Player from Adobe. This error may appear if the URL path to the embedded object is broken or you have connectivity issue to the embedded object. Powered BY XVE Various Embed.

Please have a look at the above video. Besides the general common sense guidelines about how to scale your stuff, and the Postgres typical stuff, there’s a general rule: cache, cache, and then cache some more.

However, too much caching in memcache (whatever implementation) may kill the application at some point. The application may not be database dependent, but it is cache dependent. Anything that affects the cache may have the effect of a sledgehammer on your database. Of couse, you can always scale vertically that DB instance, scale horizontally by adding read-only replicas, but the not-so-fun part is that it costs a lot just to have the provisioned resources in order to survive a cache failure.

The second option is to have a short lived failover cache on the application server. Something like five minutes, while the distributed cache from memcache may last for hours. Enough to keep the database from being hit from live traffic, while you don’t have to provision a really large database instance. Of course, it won’t work with stuff that needs some “real time” junk, but it works with data that doesn’t change with each request.

There are a lot of options for a failover cache since there’s no distributed setup to think about. It may be a memcached daemon, something like PHP’s APC API, or, the fastest option: the file based caching. Now you may think that I’m insane, but memcached still has the IPC penalty, especially for TCP communication, while if you’re a PHP user, APC doesn’t perform as expected.

I say file based caching, not disk based caching, as the kernel does a pretty good job at “eating your RAM” with the disk caching stuff. It takes more to implement it since the cache management logic must be implemented into the application itself, you don’t have stuff like LRU, expiration, etc. by default, but for failover reasons, it is good enough to worth the effort. In fact, it ran for a few days on the failover cache without any measurable impact.

The next part for not using the same basket for all of your eggs is: cache everywhere you can. For example, by using the nginx FastCGI cache, we could shave off 40% of our CPU load. Nothing experimental about this last part. It is production for the last 18 months. If you get it right, then it could be a really valuable addition to a web stack. However, a lot of testing is required before pushing the changes to production. We hit a lot of weird bugs for edge cases. The rule of thumb is: if you get the cache key right, then most of the issues are gone before going live.

In fact, by adding the cache control stuff from the application itself, we could push relatively shortly lived pages to the CDN edges, shaving off a lot of latency for repeated requests as there’s no round trip from the hosting data center to the CDN edge. Yes, it’s the latency, stupid. The dynamic acceleration that CDNs provide is nice. Leveraging the HTTP caching capabilities is nicer. Having the application in a data center closer to the client is desirable, but unless your target market is more distributed than having a bunch of machines into the same geo location, it doesn’t make any sense to deploy into a new data center which adds its fair share of complexity when scaling the data layer.