Saturday, 17 November 2018

Slimming down 1-node Elastic cluster

If you ever ran Elastic Search especially quick and dirty - single node and default config, you will notice the health is always showing yellow and that it's a proper hog for the system. Well, yes, it will be, especially in default config, as my good friend Justin Borland pointed out.

I'm a complete newbie when it comes to Elastic, deployed few in Docker containers to quickly ingest data and dig in with Kibana, but that was it. Luckily for me Justin is absolute beast when it comes to all things Elastic - he just looked at my node and right on the spot explained what's wrong with it and how to fix/improve.

Basically my default setup was running 5 shards for each of the indices stored in the system, and I had quite a few daily indices already there - we're talking months of DNS research data and web spider runs across thousands of websites... all repeated daily. This means the optimisation to be really effective needs to also deal with what's in there, not just new data I will be adding.

Plan:
  1. Change default template to run only 1 shard and 0 replicas - it's a single node deployment, so anything more complex doesn't make much sense.
  2. Use reindex API to rewrite all of existing indices as single shard versions, the deleting the old ones using 5 shards - there's no other way to do it than through reindexing.
  3. My indices are treated append-only on the day, then become read-only, so we can merge the segments - leaving technical details behind, this will mean no random access later, just linear file reads, but that's perfectly acceptable in my particular use scenario.

Let's do it!

Friday, 2 November 2018

Solution - Rancher 2 (k8s), private registry, self-signed certificates

Since Rancher switched to Kubernetes in version 2.x, I'm exposed to a lot of stupidity and limitations k8s introduced, but I can live with that, at least for a moment... What I couldn't accept was that I could no longer use my private registry (with self-signed certificate) that works perfectly fine with older Rancher (1.6 - before move to k8s).

That is now resolved!

My cluster setup


  • Rancher 2 cluster (based on Kubernetes), all running on latest RancherOS
  • Private registry available only within the LAB network - hence self-signed certificate
  • Registry has an internal host name, resolvable via internal DNS server
  • Registry does not require user accounts, so no need for credentials, but self-signed certificate prevents it from working, resulting with following error when image is pulled
x509: certificate signed by unknown authority

Dead ends


First of all, please ignore RancherOS documentation - last one I found was for version 1.2, current RancherOS is 1.4.2... anyway, it no longer works (it did for older RancherOS and Rancher 1.6 though, but new Rancher is more Kubernetes than anything else). In my research I also read a bunch of bug reports, feature requests, stack exchange articles, etc... mostly waste of time, but they gave me a good idea on rabbit holes to avoid. Some of the more useful reads are here and here, I also have a feeling this will be useful for me quite soon.
Another trick I noticed was that if I followed RancherOS docs above, the registry CA key was overwritten with something else on node reboot.

Solution (a.k.a "works for me")


Go old school Linux admin style:

  1. SSH to the RancherOS node (user is rancher@<node>), having your private CA certificate at hand
  2. As user rancheros try docker pull <registry:port>/<my image> - you should get a CA error
  3. Check your /etc/resolv.conf - mine was regularly overwritten by dhcp but it was not writing name servers correctly - this should be easily fixed by writing what you want to /etc/resolv.conf.tail (in hopes dhcp will append it when it regenerates resolv.conf).
  4. Now the key element - edit the OS wide trusted CA list (hint hint - may disappear after sudo ros os upgrade, but this can be fixed with sudo chattr +i /etc/resolv.conf)  and add your CA certificate there. Running vi /etc/ssl/certs/ca-certificates.crt and copy'n'paste does the trick!
  5. Try docker pull again, now it worked for me.

Friday, 3 June 2016

Recipe - Docker, web apps and Lets Encrypt

Intro


If you're after easy hosting of dockerized web services with automatic certificate enrolment using Lets Encrypt, then the solution is to use 2 docker containers - nginx as a web proxy and Lets Encrypt Companion to handle certificates. LE Companion can provide either LIVE or STAGING certificates, depending on configuration, but you can run only one at a time.

Container definitions below are in a docker-compose format and the recipe below contains absolutely no security hardening of the Docker installation - this is something you need to consider separately

Web proxy

TLSproxy:
  image: 'jwilder/nginx-proxy:latest'
  ports:
    - '80:80'
    - '443:443'
  volumes:
    - '/etc/letsencrypt:/etc/nginx/certs:ro'
    - /etc/nginx/vhost.d
    - /usr/share/nginx/html
    - '/var/run/docker.sock:/tmp/docker.sock:ro'
  environment:
    - 'DEFAULT_HOST=default.vhost.tld'

TLSproxy is nginx based reverse proxy that automatically discovers and configures virtual hosts running on the same machine. See image description on docker hub for details. TL;DR simple approach is:

docker run -d -e VIRTUAL_HOST=blog.domain.tld ghost

Please note, the DEFAULT_HOST variable - it's quite useful to have it set right :-)

Thursday, 18 February 2016

Adding private insecure registry to Rancher nodes

Quick post before I forget - there's quite a few people asking how to get insecure docker registry running on RancherOS node. Here's what worked well for me.

First thing that helps a lot is to have some DNS entry for your registry - remember you will use this hostname quite often, so better set it up now than use IP addresses going forward.
As I run my own internal DNS server with local zones, I have created registry.rancher.lan entry and pointed to the node running registry container.

All of my nodes were already up and running, so I didn't use cloud-config.yml file for that and had to stick to ssh to get it working, but there's nothing to stop you from adding it right there for node installation time. The ssh process is super simple - please note entire command is a single line:

$ sudo ros config set rancher.docker.args "[daemon, --log-opt, max-size=25m, --log-opt, max-file=2, -s, overlay, -G, docker, -H, unix:///var/run/docker.sock, --userland-proxy=false, --insecure-registry=registry.rancher.lan:5000]"
I've marked in bold the key element. Be aware, the syntax is quite sensitive if you use quotes. I had multiple crashes on boot because single quote was converted to python(ish) three single quotes, which of course didn't parse well going forward. Clearly the config tool tries to be smart, so please, let it be and remove quotes in parameters passed in the array.

Finally, reboot and off you go - the node will now find and correctly use the images hosted in your own registry.

Tuesday, 10 February 2015

Raspberry Pi 2 - first impressions


  1. First impression is that (in my opinion) it is visibly faster than the previous one (1st gen. model B with 512MB RAM), even on tasks that can't use more than one CPU core - this is a good sign. The difference is even more visible when comparing to 1st gen. model B with 256MB RAM...
  2. I measured power consumption at the wall using kill-a-watt type plug and here are the results:
    • No SD card inserted (not booting) - 0.6W
    • Booted up and idle, with Ethernet connected - 1.4W
    • CPU cores under load (via sysbench prime number test, with Ethernet):
      • 1 thread - 1.8W, 296 sec to complete
      • 2 threads - 2.0W, 148 sec to complete
      • 4 threads - 2.5W, 74 sec to complete
  3. I tried the Xeon flash bug (or feature) and yes, it works. Of course it doesn't react to LED generated light and the usual laser pointers - I'd love to test it against a real lightning flash but I guess I'll have to wait for weather to change. Here's the Xeon flash test:

  4. Finally, the main sticking point for use experience is slow SD card access, so pick the fastest card you can get - it's worth it!

Sunday, 9 November 2014

haste-server Base URL Hack/Patch

Recently I came across haste-server, a server behind hastebin, which is a pastebin clone written in node.js. The application is minimalistic, fairly simple and works really well, except for one rather major glitch - it takes over the root directory of the whole website.

I've noticed that several people raised an issue on GitHub asking the author for help, but so far nobody shared fully working solution. Some people tried to work reverse proxy magi, others tried to patch the code - with moderate success. Instead of adding to the problem area I thought I'll try to offer a solution - keep in mind I don't know JavaScript ;-)


Tuesday, 28 October 2014

Wake-on-Lan issues with Intel PRO Series NIC

Over the last few months I was experimenting with setting up my ham-radio station for completely remote operation, so once the rare DX comes on air I can work it regardless of where I am at the time.

The idea seems simple but this means that for a start I need to be able to remotely turn on and off all of the devices. Leaving the design itself for another post, the core element of my remote control concept is rather old ThinkPad x60s laptop. This one comes with Intel PRO/1000 Ethernet NIC and I want to use WoL to boot it up remotely.

What is WoL?
It's a simple way to turn on a machine connected to the network by sending it a single ethernet packet. Very useful if you want to boot up a machine for out-of-hours maintenance run or something similar - like in my case.

Problem
WoL works great but only once, so after you shut down the OS there's no way to do remote start again. This is something that many have encountered judging by the amount of forum posts and questions asked about the same issue.

Once I wasted more time than I should on trying to figure out what's going on, the fix turned out to be "trivial". Lesson learned for sure.