tag:blogger.com,1999:blog-16305341714439981182024-03-16T07:32:25.158+00:00ctrl-alt-del.ccSoft reset site for technology geeksTomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.comBlogger46125tag:blogger.com,1999:blog-1630534171443998118.post-58918685745996112722022-03-15T21:54:00.002+00:002022-03-15T21:55:51.217+00:00AdGuard Home and time based rules<p>I have to say, I really like AdGuard Home (AGH)... and I am a bit torn between PiHole and AGH. PiHole seems to have better dashboard and allows to drill more into logs, but AGH has more features, so in a way it is horses for courses.</p><p>Recently a new use case came up for me - to block certain websites/services based on time of day. Think of it as technical layer of parental controls. I call it technical layer, because I know a conversation with a child is way more effective than any technical solution. At the same time kids being kids (even the most obedient and respectful ones) will sooner or later try to see if something is really blocked or is dad bluffing. Let them... it's good they try.</p><h3 style="text-align: left;">AGH API</h3><p>AGH has a working API that is documented <a href="https://github.com/AdguardTeam/AdGuardHome/tree/master/openapi" target="_blank">here</a>. If you paste the contents of <span style="font-family: courier;">openapi.yaml</span> into web based <a href="https://editor.swagger.io/" rel="nofollow" target="_blank">Swagger Editor</a>, you will be able to easily navigate through the API docs.</p><p>AGH requires user to authenticate when using the API, so let's assume our username is <span style="font-family: courier;">admin</span> and password is <span style="font-family: courier;">password</span>. Invoking API is as simple as adding a HTTP header and encoding admin:password as Base64 string to include in the header - you can use <a href="https://gchq.github.io/CyberChef/" target="_blank">CyberChef</a> for this.</p><p></p><blockquote><span style="font-family: courier;">curl -H 'Authorization: Basic YWRtaW46cGFzc3dvcmQK' </span><span style="font-family: courier;">...</span></blockquote><p><span></span> Here we are interested in 2 API endpoints:</p><p></p><ul style="text-align: left;"><li><span style="font-family: courier;">/control/filtering/add_url</span></li><li><span style="font-family: courier;">/control/filtering/remove_url</span></li></ul><div><span style="font-family: inherit;">Swagger Editor shows us exact invocation method with examples:</span></div><div><span style="font-family: inherit;"><br /></span></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjiSJpS1YizbJ0CEPa499Ap6oR8von92PWKKJClKX2Ud8kImFtygVjFbY7Wu74ffkFlUDuhLeCg20f_n0BZwcsATPin11mhKOxiqNi3ytoLjRl6mxhi92KMa5_uPPNDVzozMN5E897iTPdZpA8B00298oFel4o1X2neTOSvUpYjAqbhAuBMewTksFRh=s1930" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1930" data-original-width="1886" height="640" src="https://blogger.googleusercontent.com/img/a/AVvXsEjiSJpS1YizbJ0CEPa499Ap6oR8von92PWKKJClKX2Ud8kImFtygVjFbY7Wu74ffkFlUDuhLeCg20f_n0BZwcsATPin11mhKOxiqNi3ytoLjRl6mxhi92KMa5_uPPNDVzozMN5E897iTPdZpA8B00298oFel4o1X2neTOSvUpYjAqbhAuBMewTksFRh=w626-h640" width="626" /></a></div><h3 style="text-align: left;"><span style="font-family: inherit;"><span><a name='more'></a></span><br /></span></h3><h3 style="text-align: left;"><span style="font-family: inherit;">Adding a block list</span></h3></div><div><span style="font-family: inherit;"><br /></span></div><div style="text-align: left;">You need to publish somewhere on the web a text file with your blacklist/whitelist rules - any place will do as long as AGH can reach it to load the file. URL for this file will be the value of <span style="font-family: courier;">url</span> field in the request. We give the list entry a <span style="font-family: courier;">name</span> of our choice and define if this is a <span style="font-family: courier;">whitelist</span> entry (when set to <span style="font-family: courier;">true</span>) or not (if set to <span style="font-family: courier;">false</span>). </div><h3 style="text-align: left;"><br /></h3><h3 style="text-align: left;">Example</h3><div style="text-align: left;"><br /></div><div style="text-align: left;">Rules file can be even hosted on pastebin if they are not changing too often - <a href="https://pastebin.com/raw/SHpeyr1C" rel="nofollow" target="_blank">here's Roblox blocking</a> one.</div><div style="text-align: left;"><br /></div><div style="text-align: left;">Adding block:</div><div style="text-align: left;"><span style="font-family: courier;"></span></div><blockquote><div style="text-align: left;"><span style="font-family: courier;">curl -H 'Authorization: Basic YWRtaW46cGFzc3dvcmQK' -d '{"name": "Roblox block", "url": "https://pastebin.com/raw/SHpeyr1C", "whitelist": false}' http://agh.local/control/filtering/add_url</span></div></blockquote><div style="text-align: left;"><span style="font-family: courier;"></span></div><p></p><p></p><ul></ul><p></p><p></p><div style="text-align: left;">Removing block:</div><p></p><p></p><p></p><blockquote><div><span style="font-family: courier;">curl -H 'Authorization: Basic YWRtaW46cGFzc3dvcmQK' -d '{"url":"https://pastebin.com/raw/SHpeyr1C"}' http://agh.local/control/filtering/</span><span style="font-family: courier;">remove_url</span></div></blockquote><div><div><span style="font-family: courier;"></span></div><p></p><p></p><ul></ul><p></p><p></p><div>Now add a bit of <span style="font-family: courier;">cron</span> and you job is done.</div></div>Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-290629503052374082020-01-05T22:09:00.001+00:002020-01-10T18:05:50.188+00:00Mikrotik + Pi Zero + Pi-hole = advertising sinkhole with fail-safe<h2>
Components</h2>
<ul>
<li><b>Mikrotik</b> router with USB port - I tested on <i>RB2011UiAS-2HnD-IN </i>and<i> hAP ac </i>models</li>
<li><b>RouterOS</b> in modern version - I tested with <i>long term</i> (6.44.6)</li>
<li><b>Raspberry Pi Zero</b> - I use old one without "W", with 4GB microSD card running latest Raspbian 10 Buster (minimal, without GUI!)</li>
<li>Short <b>micro-USB data cable</b> - because many cheap cables don't do data</li>
</ul>
<div>
<br /></div>
<div>
Pi Zero actually has more than enough power to run Pi-hole serving even quite large home/family network and running it completely self-contained off Mikrotik seems to work great!</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdKsYcHepUn-MApUqLc-AizUp5t23o0zPL9yKj8vTRsLJreDAdeu8Ei67azYdWfLzLUzy_RXAT-Au1zGS4LDhTlTY_tdE38JWlkrNSFGQ5hxinexGUJ1fyU3FWyc2wA7BQdO5d0DSBZ60/s1600/Screenshot+2020-01-07+at+20.44.28.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="172" data-original-width="452" height="121" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdKsYcHepUn-MApUqLc-AizUp5t23o0zPL9yKj8vTRsLJreDAdeu8Ei67azYdWfLzLUzy_RXAT-Au1zGS4LDhTlTY_tdE38JWlkrNSFGQ5hxinexGUJ1fyU3FWyc2wA7BQdO5d0DSBZ60/s320/Screenshot+2020-01-07+at+20.44.28.png" width="320" /></a></div>
<div>
<br /></div>
<h2>
Initial setup</h2>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8GX0kXycE9WW5YWzq_6VIIeEVgAZR08Nv4IM9CLZ_tnJM2wGd1tnmidW68fD6V0u-Ey-VGyIZXa64u0rUSuO0ViNRFhz8J_yLvitbZoaxFkKZJ31SZITTRe7b0bE6rULpPZe5YN41cE4/s1600/IMG_20200105_203357.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;"><img border="0" data-original-height="1200" data-original-width="1145" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8GX0kXycE9WW5YWzq_6VIIeEVgAZR08Nv4IM9CLZ_tnJM2wGd1tnmidW68fD6V0u-Ey-VGyIZXa64u0rUSuO0ViNRFhz8J_yLvitbZoaxFkKZJ31SZITTRe7b0bE6rULpPZe5YN41cE4/s320/IMG_20200105_203357.jpg" width="305" /></a></div>
<ol>
<li>Download and burn the latest Raspbian onto the SD card - I used for this Etcher and <i>2019-09-26-raspbian-buster-lite.img</i></li>
<li>Connect SD card to a PC and in partition called <i>boot</i> edit two files to enable Ethernet gadget:</li>
<ol>
<li><span style="font-family: "courier new" , "courier" , monospace;">config.txt</span> - at the very end of the file add a line saying <b>dtoverlay=dwc2</b></li>
<li><span style="font-family: "courier new" , "courier" , monospace;">cmdline.txt</span> - add <b>modules-load=dwc2,g_ether</b> directly <u>after</u> 'rootwait' and before any other parameters that may (or not) be there</li>
</ol>
<li>Boot up RPi powering from PC using the port marked as <span style="font-family: "courier new" , "courier" , monospace;">USB</span> on the board - not the PWR IN; it's the one in the centre - only that one does power + gadget</li>
<li>After all boots up, you should be able to run <span style="font-family: "courier new" , "courier" , monospace;">ssh pi@raspberrypi.local</span> (thanks mDNS!) with password <span style="font-family: "courier new" , "courier" , monospace;">raspberry</span></li>
<li>On the RPi create file called <span style="font-family: "courier new" , "courier" , monospace;">/etc/modprobe.d/g_ether.conf</span> with the following content (single line of text)<br /><b>options g_ether idVendor=0x05ac idProduct=0x1402 iProduct=Pi0 iManufacturer=Raspberry</b><br />NOTE - This is required for RPi to show up as LTE interface on Mikrotik!<br /><a name='more'></a></li>
<li>Configure network access on the PC to allow RPi to reach the Internet - NAT or something</li>
<li>Install <b>Pi-hole</b> - <a href="https://github.com/pi-hole/pi-hole/wiki/Getting-Started" target="_blank">instructions are here</a>, follow the steps and you will end up with Pi with static IP address configured on it</li>
<li>If you want to change the static assigned IP address AFTER installing Pi-hole, you can edit <span style="font-family: "courier new" , "courier" , monospace;"><b>usb0</b></span> interface settings in <span style="font-family: "courier new" , "courier" , monospace;">/etc/dhcpcd.conf</span></li>
<li>Once Pi-hole is ready, shut down both RPi and Mikrotik, connect RPi to USB port on Mikrotik, let it boot up... and then you should see <span style="font-family: "courier new" , "courier" , monospace;"><b>lte1</b></span> under both <span style="font-family: "courier new" , "courier" , monospace;">/interfaces</span> and <span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">/interfaces lte<br /><br /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheY3wkAFXDqWmMNBqAfeT72s8bCI-yTgFCe_Lm24KTq-ic2BF-GtkC60Lo2n0abn0RipqdePXIYfjYCOF6ntG_sh-1rH80uN7RwdG6skDPfkWU3-kAgR01Wlavs0JJR0euZjO5uLyZv-A/s1600/Screenshot+2020-01-06+at+21.28.25.png" imageanchor="1" style="font-family: Times; margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="280" data-original-width="1112" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheY3wkAFXDqWmMNBqAfeT72s8bCI-yTgFCe_Lm24KTq-ic2BF-GtkC60Lo2n0abn0RipqdePXIYfjYCOF6ntG_sh-1rH80uN7RwdG6skDPfkWU3-kAgR01Wlavs0JJR0euZjO5uLyZv-A/s400/Screenshot+2020-01-06+at+21.28.25.png" width="400" /></a><br /></span></span></li>
<li>Add IP address to <span style="font-family: "courier new" , "courier" , monospace;">lte1</span> from the same subnet as set on the RPi and enjoy - you should be able to reach RPi via SSH and/or web - if not, check firewall<br />NOTE - you can't add <span style="font-family: "courier new" , "courier" , monospace;">lte1</span> to the bridge, so just treat it as routed destination instead of bridged (sorry, no mDNS broadcasts for you!)</li>
<li>Now you can edit DNS settings in your DHCP server - this sits under <span style="font-family: "courier new" , "courier" , monospace;">Networks</span> - enjoy!</li>
</ol>
<div>
Note - <span style="font-family: inherit;">Mikrotik is not PnP - you have to reboot it after connecting RPi. If after 1st reboot you don't see </span><span style="font-family: "courier new" , "courier" , monospace;"><i>lte1</i></span><span style="font-family: inherit;"> interface, reboot the Mikrotik again. You may also try updating Mikrotik firmware.</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<h2>
Automated fail-safe - when RPi goes down...</h2>
</div>
<div>
Now, with this being your primary DNS server, if Pi-hole or RPi goes down, you lost your DNS so ideally there's some sort of uptime testing for RPi and automatic fall-back to a default DNS server when RPi is non-responsive.</div>
<div>
<br /></div>
<div>
Luckily Mikrotik allows us to use <span style="font-family: "courier new" , "courier" , monospace;">Tools -> Netwatch</span> to do it. It uses ICMP ping to check if host is up, so nothing too fancy but a good start!</div>
<div>
<ol>
<li>Enable DNS server on Mikrotik (of course blocking access from WAN on the firewall)</li>
<li>Change DHCP Server configuration to use Mikrotik as DNS server and configure Mikrotik to use RPi as upstream DNS - you may want to disable 'Use peer DNS' in DHCP Client on Mikrotik</li>
<li>Implement failover to known working DNS when RPi goes down - for example to 1.1.1.1 (CloudFlare) or 8.8.8.8 (Google)... or whatever you want to use :-)<br /><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0FVQhXPj0yTXCll8s362cJ2Tw3fLTY6Bl1Emb745V8OhiVtcjLMi0bPsZ1Q4jpuZkF6XnZ6e_btB5LV4pemDm0v21LiOjVkoZHDMkHS0mAfevBqPalsE7SGxyn2XzPQ08tgEhHzum_WI/s1600/Screenshot+2020-01-05+at+21.40.02.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="490" data-original-width="616" height="315" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0FVQhXPj0yTXCll8s362cJ2Tw3fLTY6Bl1Emb745V8OhiVtcjLMi0bPsZ1Q4jpuZkF6XnZ6e_btB5LV4pemDm0v21LiOjVkoZHDMkHS0mAfevBqPalsE7SGxyn2XzPQ08tgEhHzum_WI/s400/Screenshot+2020-01-05+at+21.40.02.png" width="400" /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj434SgJkR_vx9Orp4mBF-49MedHYg5_q4g7589gTvBE857uJRx8Y_y6oPJoeWNX77LpAUXyVam8vocMH_WqkeOZ-EESaRXbwYPijuPhG_FZGxuxCtL0LwkDfrwW2OBdhJ30Q2AWB1EnyM/s1600/Screenshot+2020-01-05+at+21.40.17.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="490" data-original-width="616" height="315" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj434SgJkR_vx9Orp4mBF-49MedHYg5_q4g7589gTvBE857uJRx8Y_y6oPJoeWNX77LpAUXyVam8vocMH_WqkeOZ-EESaRXbwYPijuPhG_FZGxuxCtL0LwkDfrwW2OBdhJ30Q2AWB1EnyM/s400/Screenshot+2020-01-05+at+21.40.17.png" width="400" /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2tcd8e023ED75BbIQ3GqhJK_zUnuQGLruls54cU9AGSGBWLQoJcQnFcfZL9-AmxXulpMz5ecld9yyi1rHno1VZc_JiglPFhKfxwF8RwMwXQhGJlt0-z28tI4tMrQgwRlvIfGvdr8v0OE/s1600/Screenshot+2020-01-05+at+21.40.42.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="490" data-original-width="618" height="315" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2tcd8e023ED75BbIQ3GqhJK_zUnuQGLruls54cU9AGSGBWLQoJcQnFcfZL9-AmxXulpMz5ecld9yyi1rHno1VZc_JiglPFhKfxwF8RwMwXQhGJlt0-z28tI4tMrQgwRlvIfGvdr8v0OE/s400/Screenshot+2020-01-05+at+21.40.42.png" width="400" /></a></li>
</ol>
<h2>
UPDATE:</h2>
<div>
In fact it would be also possible to write UP script that would trigger in the background once host comes up (ping ok) and script would check if DNS resolver also works, before pointing Mikrotik to RPi resolver. I may look into this at some later time.</div>
</div>
Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com26tag:blogger.com,1999:blog-1630534171443998118.post-40058922019173028552019-11-30T08:34:00.002+00:002020-07-29T08:22:08.799+01:00Hacking smart plugs for fun and profit<h2>
Smarter, the smart way...</h2>
You'd like to have your home a bit smarter but not spend a ton of money and you don't like the fact that you have to trust an unnamed 3rd-party company with your data and more importantly access to things that can trigger kinetic actions in your household.<br />
<br />
Chinese company <a href="https://www.itead.cc/" target="_blank">Itead</a> is the maker of well known Sonoff devices (do not confuse with audio gear from Sonos). They have a vast range of wifi controlled relays, in various formats and sizes - overall very cool stuff. There is a really nice <a href="https://www.youtube.com/watch?v=uZjZZKiKlNY" rel="nofollow" target="_blank">video on youtube</a> that will give you more idea about what I mean by this.<br />
One of the best parts of Sonoff devices is that they use ESP8266 as the main driver chip and attach relays to it, so being ESP8266 based they are very hackable! That's just what we need!<br />
<br />
These days, Itead and others are making smart devices using the same tested pattern and rebrand them under hundreds of names. You will find on Amazon various devices using the same management app (usualy eWeLink or Smart Life), which means OEM devices - good for us!<br />
<br />
<b>NOTE:</b><br />
Since 2019 more and more devices switch from ESP82xx to Realtek chips - the OEM ecosystem is also a curse. For example in 2018 I bought Teckin SP27 smart plug - worked perfect... bought some <a href="https://www.amazon.co.uk/gp/product/B07YY9NKSF" rel="nofollow" target="_blank">SP27</a> and <a href="https://www.amazon.co.uk/gp/product/B07STZ1V3J/" rel="nofollow" target="_blank">SP23</a> this week and they run Realtek, so DO NOT BUY those if you want to reflash :-)<br />
<br />
What works? I got <a href="https://www.amazon.co.uk/gp/product/B07RYPJTJX/" rel="nofollow" target="_blank">this set</a> and as of late November 2019 it was still ESP82xx based. Others reported at the same time that <a href="https://www.amazon.co.uk/ANOOPSYCHE-Monitoring-Wireless-Sockets-Control/dp/B07PH28N1D" rel="nofollow" target="_blank">this</a> is also working. Basically depends on which batch you get and a bit of luck :-)<br />
<br />
<a name='more'></a><br />
<h2>
If it works, why change it?</h2>
I like hacking and automating things, but I don't like sharing my data with random companies. When you run an off-the-shelf smart plug or switch, what happens is that to integrate it with Alexa, you need to create trust relationship between Amazon ecosystem and whoever is running the cloud backend for your plug.<br />
<br />
We have two issues here - someone else runs the system that can tell your plug to turn on (kettle, lights, etc.) and it's invisible to you - that layer is magic, as far as you know it may well not exist at all... but the same layer links to your Alexa or Google Home and there's a trust relationship between them. I won't speculate on data sharing, let's stick to the facts.<br />
<br />
One of my main goals was to integrate my smart plugs and switches with some sort of home automation (more on that in next post, soon). I don't have massive ambitions here, but want to use this with or without internet access, so if this requires cloud to work, it's a non-starter for me.<br />
<br />
<h2>
Meet Tasmota!</h2>
<a href="https://github.com/arendst/Tasmota" rel="nofollow" target="_blank">Tasmota</a> is an open source firmware for ESP8266 based smart plugs/switches. It is very mature and extremely powerful! Here's just some of it's features:<br />
<br />
<ul>
<li>MQTT support</li>
<li>HTTP support - offers effectively simple RESTful API</li>
<li>OTA firmware updates - directly from Internet or local network</li>
<li>Remote logging (syslog)</li>
<li>Ability to plug additional sensors - some devices have them, some will accept simple mods</li>
<li>Device templates - pre-set configurations for various branded devices; there's a good chance that at leat one will work with your clone!</li>
</ul>
<div>
<br /></div>
<h2>
Let's get flashing</h2>
<div>
You will need a Linux machine (Debian/Ubuntu is the easiest and fully supported), WiFi adapter that you can put into AP mode (those USB dongles for RaspberryPi usually work well - as long as they are not Realtek based) and <a href="https://github.com/ct-Open-Source/tuya-convert" target="_blank">tuya-convert</a> software.</div>
<div>
<br /></div>
<div>
Clone software from GitHub, edit config file to set the name of your WiFi interface (that will be AP). Depending on which fork of tuya-convert you use, wifi may be either password protected or wide open, but that's a separate issue... </div>
<div>
<br /></div>
<div>
Here's how it goes (run all operations below as <b>root</b>):</div>
<div>
<ol>
<li>Run start_flash.sh</li>
<li>If you get meesage systemd is getting in the way, respond Y to question(s) and you will get to a numbered list of tasks - almost ready!</li>
<li>Connect another device to this wifi - you will get IP address from DHCP and for mobile devices hotspot login screen - leave it there, that's all.</li>
<li>At this point it helps to open another terminal session and run <b style="font-style: italic;">screen -r smarthack-web </b>to see if the process goes as planned; if at any point you see message<br /><br /><i>WARNING: it appears this device does not use an ESP82xx and therefore cannot install ESP based firmware</i><br /><br />then it means you are out of luck - most likely your plug is using Realtek chip - sorry.</li>
<li>Power up the plug, after approximately 3sec press plug button once, you hear click and it lits up - good</li>
<li>Press and hold the button again for approximately 5sec, until the LED goes off - at this point release... few seconds later the LED will be flashing fast (2x/sec) - you are in a smart programming mode</li>
<li>Hit ENTER in the <b><i>tuya-convert</i></b> main window... sit back and relax :-P</li>
</ol>
<div>
At this stage plug will listen for broadcast info with wifi name and password, it will join the wifi, try to register with Smart Life app using fake cloud server provided by <i>tuya-convert.</i> Plug will be told to load new software from the web (<i>tuya-convert</i> again) and once this is done, an original firmware will be downloaded from the plug and save to file. Once all that is done, main screen asks if you want to flash another one... HOLD IT HERE!</div>
</div>
<div>
<br /></div>
<div>
Open another terminal (regular user will do) and run <i style="font-weight: bold;">curl http://10.42.42.42/flash2</i> to change bootloader (or <b style="font-style: italic;">http://10.42.42.42/undo</b> to abort flashing and revert to stock firmware), then <i style="font-weight: bold;">curl http://10.42.42.42/flash3</i> to write Tasmota permanently into the plug. Few seconds after hitting <i>/flash3</i> you will see web-server sending a file called <i style="font-weight: bold;">thirdparty.bin</i> which is a Tasmota binary.</div>
<div>
<br />
Please also note that <i>10.42.42.42</i> is the usual IP assigned to the plug, but it may be different. You can resume <i style="font-weight: bold;">smarthack-wifi</i> screen session to see what IP is assigned in your case.<br />
<br /></div>
<div>
Once plug reboots it's up and good to go - it will start in AP mode, so find open network called <i style="font-weight: bold;">sonoff-xxxx</i> and connect to it, configure your WiFi name and password, plug will reboot and join that one.</div>
<div>
<br /></div>
<div>
Once you're done with all the plugs, there's a separate script to stop AP from running.</div>
<div>
<br /></div>
<h2>
What's next?</h2>
<div>
Check your router logs to see what IP the plug got, open the IP in web browser and hit TOGGLE button to hear if pug clicks. If it doesn't then it doesn't work at all - you will want to check which ready-made template will work.</div>
<div>
<br /></div>
<div>
First thing I do is usually update firmware by going to <i style="font-weight: bold;">Firmware Upgrade</i> and using the default URL that is already there.</div>
<div>
<br /></div>
<div>
If you bought <a href="https://www.amazon.co.uk/ANOOPSYCHE-Monitoring-Wireless-Sockets-Control/dp/B07PH28N1D" rel="nofollow" target="_blank">the same set as I did</a>, then you go to <b><i>Configuration --> Configure Module</i></b> and use <b><i>Module Type</i></b> dropdown to set <b><i>BlitzWolf SHP (45)</i></b>. Plug will reboot, reconnect to wifi and TOGGLE should work... and to my surprise, this plug has also power monitoring so when I plug something into it and turn on, I get this:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxnZUylSyvIVvQDcjE_ReL2zRfak4kf9D368uw728GcLFBhsfYG8rKBUxRUhzPM7BJ1NGrKCPt5_y_WNMftp3MjMJ34gcLIYwts5E20odp0u1iQQYw7kJEk8mgyGS9-r_NUN6xqar1N2k/s1600/Screenshot+2019-11-28+at+07.34.50.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1516" data-original-width="700" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxnZUylSyvIVvQDcjE_ReL2zRfak4kf9D368uw728GcLFBhsfYG8rKBUxRUhzPM7BJ1NGrKCPt5_y_WNMftp3MjMJ34gcLIYwts5E20odp0u1iQQYw7kJEk8mgyGS9-r_NUN6xqar1N2k/s640/Screenshot+2019-11-28+at+07.34.50.png" width="292" /></a></div>
<div>
<br /></div>
<div>
Now... explore menus, thinking cap on, read the Tasmota docs... because what it can do is quite spectacular - a separate post for sure.</div>
<div>
<br /></div>
<h2>
Final note</h2>
<div>
Standard disclaimers of "works for me" and "do it at your own risk" apply.</div>
Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com1tag:blogger.com,1999:blog-1630534171443998118.post-40030553272627098392019-01-29T22:53:00.001+00:002019-01-29T22:57:23.197+00:00More range, more fun - ADS-B setup updatesQuick post today...<br />
<br />
I've updated my ADS-B receiving station about a week ago and today, most likely due to radio propagation conditions change, I set a new detection range record of <b>241nm</b> (or 466km) - here's the view from my <a href="https://www.flightradar24.com/" target="_blank">FlightRadar24</a> dashboard, and the day's not over yet!
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbXyTWKhuObwXOIMLmjNGFvdsBq6xpBbzMGWj19nBohStYpVa2UTPgY-5uq_qSdNoNyl4ulTf0nMWgpwBWy7J_D6E7dAVHmP2osZ0yG5kK8y6zArOmAGb93kuiqwrpJJlEGdiM0mLXgA0/s1600/dashboard.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1251" data-original-width="1600" height="499" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbXyTWKhuObwXOIMLmjNGFvdsBq6xpBbzMGWj19nBohStYpVa2UTPgY-5uq_qSdNoNyl4ulTf0nMWgpwBWy7J_D6E7dAVHmP2osZ0yG5kK8y6zArOmAGb93kuiqwrpJJlEGdiM0mLXgA0/s640/dashboard.png" width="640" /></a></div>
<br />
Strangely most detections are due South today, which wasn't the case over the rest of the week (most are N-NW due to window where the antenna is being exactly North facing). This is the magic of radio propagation - band conditions change constantly, sometimes in most surprising way.<br />
<br />
Right, so what what were the updates I made last week? I made only one...<br />
<a name='more'></a><br />
Updated RTL-SDR dongle from old generic one to <a href="https://radarbox24.myshopify.com/collections/adsb-receivers/products/radarbox-ads-b-flightstick-dongle-ads-b-receiver" target="_blank">one built by RadaBox24</a> (a.k.a. AirNav.Systems) - aircrafts seen count jumped approx 50% right away. I noticed this dongle thanks to a <a href="https://www.rtl-sdr.com/radarbox24-specialty-ads-b-rtl-sdr-reduced-to-9-95-shipping/" target="_blank">blog post on rtl-sdr.com</a> as a very cheap, purpose built (LNA and filters) upgrade and at the price they go at, it was obvious I'll have at least one!<br />
Cool thing for us folks in UK - I ordered directly from the RadarBox online store (no Amazon or anything), it was 3 working days to get the package - it was <b>posted from UK</b> as RM48.<br />
<br />
As for the antenna - well, as the dongle has SMA connector and my antenna had TV-type one, I tried for a day my 2m/70cm mag-mount mobile antenna, it was really "ok" but once proper SMA adapter arrived and I hooked up the old antenna (now the real comparison begins), I was blown away... My <a href="https://twitter.com/tomaszmiklas/status/987432239544897537" target="_blank">home made ADS-B antenna</a> is performing sooooo much better and making it took maybe 10 minutes - it's just a piece of TV coax cable (that would be 75 Ohm instead of 50 Ohm I should be using, but that's what I had in my drawer at the time). Design is simple and similar thing is <a href="https://www.eham.net/ehamforum/smf/index.php?topic=112703.0" target="_blank">presented here as "Handy Spider"</a>, my dimensions are also very similar to what's shown there. I don't even cut the coax cable - just make antenna at one end, coil surplus close to the dongle and use a plastic zip tie to make sure the 8 counterpoises don't fall out on their own.<br />
<br />
Finally, as I was already running dump1090-mutability (Rasbian package) and bought receiver from Radabox, I also started feeding data to them. On a normal propagation day, my polar plot looks like the one below and there's a ton of spare CPU to feed a few more systems.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9B0f0wY9bgLMmPQzhkoln1gTWp3zPZu6E8HnHRfqu9LYuXmgyhHUjUTek69qqgQxeJ4T-ImAfvor_BpdXqd3-xIWclj5fSp5DxssCBJje-_W7dzhhT2EUZWHeknbnBiw2O0TqWuxewwQ/s1600/radarbox.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="930" data-original-width="1230" height="481" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9B0f0wY9bgLMmPQzhkoln1gTWp3zPZu6E8HnHRfqu9LYuXmgyhHUjUTek69qqgQxeJ4T-ImAfvor_BpdXqd3-xIWclj5fSp5DxssCBJje-_W7dzhhT2EUZWHeknbnBiw2O0TqWuxewwQ/s640/radarbox.png" width="640" /></a></div>
<br />
So, onwards and upwards. Go experiment, it's fun... and you get business account on both FR24 and RB24, just in case you have an actual use for it :-)Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-52551381829280785652018-11-17T22:25:00.002+00:002018-11-18T09:03:36.394+00:00Slimming down 1-node Elastic clusterIf you ever ran Elastic Search especially quick and dirty - single node and default config, you will notice the health is always showing yellow and that it's a proper hog for the system. Well, yes, it will be, especially in default config, as my good friend Justin Borland pointed out.<br />
<br />
I'm a complete newbie when it comes to Elastic, deployed few in Docker containers to quickly ingest data and dig in with Kibana, but that was it. Luckily for me Justin is absolute beast when it comes to all things Elastic - he just looked at my node and right on the spot explained what's wrong with it and how to fix/improve.<br />
<br />
Basically my default setup was running 5 shards for each of the indices stored in the system, and I had quite a few daily indices already there - we're talking months of DNS research data and web spider runs across thousands of websites... all repeated daily. This means the optimisation to be really effective needs to also deal with what's in there, not just new data I will be adding.<br />
<br />
Plan:<br />
<ol>
<li>Change default template to run only 1 shard and 0 replicas - it's a single node deployment, so anything more complex doesn't make much sense.</li>
<li>Use reindex API to rewrite all of existing indices as single shard versions, the deleting the old ones using 5 shards - there's no other way to do it than through reindexing.</li>
<li>My indices are treated append-only on the day, then become read-only, so we can merge the segments - leaving technical details behind, this will mean no random access later, just linear file reads, but that's perfectly acceptable in my particular use scenario.</li>
</ol>
<div>
<br /></div>
<div>
Let's do it!</div>
<div>
<a name='more'></a><br /></div>
<div>
1. Setting general template to use 1 shard and no replicas is easy:</div>
<div>
<br />
<script src="https://gist.github.com/tmiklas/7016fea5c3bc4c4ff5ddfe8ffa6aef04.js"></script></div>
<div>
2. Reindexing of all indices can be done using this script, provided as-is (i.e. worked for me, use at your own risk). <u>Warning:</u> this step deleted all my visualisations and dashboards!</div>
<div>
<br />
<script src="https://gist.github.com/tmiklas/e69b6dbc52223d602f873e0c4817d476.js"></script></div>
<div>
3. Merging segments - this should be ran periodically; it accounts for about a half of disk reclaim achieved in this exercise.<br />
<br />
<script src="https://gist.github.com/tmiklas/acb8779942cc27b95cc1ae18303044cb.js"></script>
<br />
<h4>
Was it worth it? </h4>
What is the actual benefit, if any? <br />
<br />
Actually it's quite a massive difference, even Justin didn't expect it to work out so well for me:<br />
<br />
<ul>
<li>JVM Heap use <b>decreased by 62%</b> (32GB is what I allocated, host has more memory)</li>
<li>Disk space used by data <b>decreased by 56%</b></li>
<li>Primary shards count (obviously) <b>decreased by 79.5%</b> which translates to document count <b>decreased by almost 89%</b></li>
<li>Max response time <b>decreased by 22%</b></li>
<li>Cluster health shows <b><span style="color: #6aa84f;">green</span></b> instead of <b><span style="color: #f6b26b;">yellow</span></b></li>
</ul>
<br />
Pics or it didn't happen - here's before and after adjustment:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJarcX7f4KPp_LlOieh4ZA3n3r-cRTF46W85N1pmL8l9Npakj8avPMI-B_FnvBnXPoigZ5GyxvNe1WmT6hqCIfXn4auzb1I7heHd-kbaQR7F0xvVOPbke0NP7pcdJUNirEE6jvNiK2tpo/s1600/Screen+Shot+2018-11-16+at+12.24.56.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="Before" border="0" data-original-height="554" data-original-width="1600" height="221" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJarcX7f4KPp_LlOieh4ZA3n3r-cRTF46W85N1pmL8l9Npakj8avPMI-B_FnvBnXPoigZ5GyxvNe1WmT6hqCIfXn4auzb1I7heHd-kbaQR7F0xvVOPbke0NP7pcdJUNirEE6jvNiK2tpo/s640/Screen+Shot+2018-11-16+at+12.24.56.png" title="Before" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjn37jxSBe0ciUDl67NZQbaSetLwzQoHoCqSYsb7T_X5l-F4kzq11qpAKj-75XPlm5t4G_JimYiKjLH4eN13gDb3s6j40mSFPjbvqlwVhAEWhBAIzbmiOQ2zFMQU-D9hSu6RbpXkE1cmCQ/s1600/Screen+Shot+2018-11-16+at+14.22.14.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="After" border="0" data-original-height="556" data-original-width="1600" height="222" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjn37jxSBe0ciUDl67NZQbaSetLwzQoHoCqSYsb7T_X5l-F4kzq11qpAKj-75XPlm5t4G_JimYiKjLH4eN13gDb3s6j40mSFPjbvqlwVhAEWhBAIzbmiOQ2zFMQU-D9hSu6RbpXkE1cmCQ/s640/Screen+Shot+2018-11-16+at+14.22.14.png" title="After" width="640" /></a></div>
<br />
<br />
Thanks Justin, that's amazing!</div>
Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-34247532792785725162018-11-02T16:47:00.001+00:002018-11-03T00:05:18.112+00:00Solution - Rancher 2 (k8s), private registry, self-signed certificatesSince Rancher switched to Kubernetes in version 2.x, I'm exposed to a lot of stupidity and limitations k8s introduced, but I can live with that, at least for a moment... What I couldn't accept was that I could no longer use my private registry (with self-signed certificate) that works perfectly fine with older Rancher (1.6 - before move to k8s).<br />
<br />
That is now resolved!<br />
<br />
<h3>
</h3>
<h3>
My cluster setup</h3>
<br />
<ul>
<li>Rancher 2 cluster (based on Kubernetes), all running on latest RancherOS</li>
<li>Private registry available only within the LAB network - hence self-signed certificate</li>
<li>Registry has an internal host name, resolvable via internal DNS server</li>
<li>Registry does not require user accounts, so no need for credentials, but self-signed certificate prevents it from working, resulting with following error when image is pulled</li>
</ul>
<blockquote class="tr_bq">
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; color: #6a737d; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 11.9px; line-height: 1.45; margin-bottom: 16px; overflow-wrap: normal; overflow: auto; padding: 16px;"><code style="background: initial; border-radius: 3px; border: 0px; box-sizing: border-box; display: inline; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 11.9px; line-height: inherit; margin: 0px; overflow-wrap: normal; overflow: visible; padding: 0px; word-break: normal;">x509: certificate signed by unknown authority</code></pre>
</blockquote>
<h3>
Dead ends</h3>
<br />
First of all, please ignore <a href="https://rancher.com/docs/os/v1.2/en/configuration/private-registries/" rel="nofollow" target="_blank">RancherOS documentation</a> - last one I found was for version 1.2, current RancherOS is 1.4.2... anyway, it no longer works (it did for older RancherOS and Rancher 1.6 though, but new Rancher is more Kubernetes than anything else). In my research I also read a bunch of bug reports, feature requests, stack exchange articles, etc... mostly waste of time, but they gave me a good idea on rabbit holes to avoid. Some of the more useful reads are <a href="https://github.com/kubernetes/kubernetes/issues/43924" rel="nofollow" target="_blank"><span id="goog_1592499917"></span>here</a> and <a href="https://github.com/rancher/rancher/issues/13676" rel="nofollow" target="_blank">here</a>, I also have a feeling <a href="https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/" rel="nofollow" target="_blank">this</a> will be useful for me quite soon.<br />
Another trick I noticed was that if I followed RancherOS docs above, the registry CA key was overwritten with something else on node reboot.<br />
<br />
<h3>
Solution (a.k.a "works for me")</h3>
<div>
<br /></div>
Go old school Linux admin style:<br />
<br />
<ol>
<li>SSH to the RancherOS node (user is <span style="background-color: white; font-family: "courier new" , "courier" , monospace;">rancher@<node></span>), having your private CA certificate at hand</li>
<li>As user <span style="font-family: "courier new" , "courier" , monospace;">rancheros</span> try <span style="background-color: #eeeeee;"><span style="font-family: "courier new" , "courier" , monospace;">docker pull <registry:port>/<my image></span></span> - you should get a CA error</li>
<li>Check your <span style="background-color: #eeeeee;"><span style="font-family: "courier new" , "courier" , monospace;">/etc/resolv.conf</span></span> - mine was regularly overwritten by dhcp but it was not writing name servers correctly - this should be easily fixed by writing what you want to <span style="background-color: #eeeeee;"><span style="font-family: "courier new" , "courier" , monospace;">/etc/resolv.conf.tail</span></span> (in hopes dhcp will append it when it regenerates <span style="font-family: "courier new" , "courier" , monospace;">resolv.conf</span>).</li>
<li>Now the key element - edit the OS wide trusted CA list (hint hint - may disappear after <span style="background-color: #eeeeee;"><span style="font-family: "courier new" , "courier" , monospace;">sudo ros os upgrade</span></span>, but this can be fixed with <span style="background-color: #eeeeee; font-family: "courier new" , "courier" , monospace;">sudo chattr +i /etc/resolv.conf</span>) and add your CA certificate there. Running <span style="background-color: #eeeeee;"><span style="font-family: "courier new" , "courier" , monospace;">vi /etc/ssl/certs/ca-certificates.crt</span></span> and copy'n'paste does the trick!</li>
<li>Try <span style="background-color: #eeeeee;"><span style="font-family: "courier new" , "courier" , monospace;">docker pull</span></span> again, now it worked for me.</li>
</ol>
Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-80922279079803164902016-06-03T20:23:00.002+01:002016-06-03T20:23:30.147+01:00Recipe - Docker, web apps and Lets Encrypt<h3>
Intro</h3>
<br />
If you're after easy hosting of dockerized web services with automatic certificate enrolment using Lets Encrypt, then the solution is to use 2 docker containers - nginx as a web proxy and <a href="https://letsencrypt.org/" target="_blank">Lets Encrypt</a> Companion to handle certificates. LE Companion can provide either LIVE or STAGING certificates, depending on configuration, but you can run only one at a time.<br />
<br />
Container definitions below are in a docker-compose format and the recipe below contains absolutely no security hardening of the Docker installation - this is something you need to consider separately<br />
<br />
<h3>
Web proxy</h3>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">TLSproxy:<br /> image: 'jwilder/nginx-proxy:latest'<br /> ports:<br /> - '80:80'<br /> - '443:443'<br /> volumes:<br /> - '/etc/letsencrypt:/etc/nginx/certs:ro'<br /> - /etc/nginx/vhost.d<br /> - /usr/share/nginx/html<br /> - '/var/run/docker.sock:/tmp/docker.sock:ro'<br /> environment:<br /> - 'DEFAULT_HOST=default.vhost.tld'</span></blockquote>
<br />
TLSproxy is nginx based reverse proxy that automatically discovers and configures virtual hosts running on the same machine. See image description on docker hub for details. TL;DR simple approach is:<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">docker run -d -e VIRTUAL_HOST=blog.domain.tld ghost</span></blockquote>
<br />
Please note, the <span style="font-family: Courier New, Courier, monospace;">DEFAULT_HOST</span> variable - it's quite useful to have it set right :-)<br />
<br />
<a name='more'></a><br />
<h3>
TLS support</h3>
Staging certs are issued from another <span style="font-family: Courier New, Courier, monospace;">ACME_CA_URI</span> different to the default one, which is defined as environment variable for the container:<br />
<h4>
Staging </h4>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">TLSproxy-LE-agent-STAGING:<br /> image: 'jrcs/letsencrypt-nginx-proxy-companion:latest'<br /> environment:<br /> - 'ACME_CA_URI=https://acme-staging.api.letsencrypt.org/directory'<br /> volumes:<br /> - '/etc/letsencrypt:/etc/nginx/certs'<br /> - '/var/run/docker.sock:/var/run/docker.sock:ro'<br /> volumes_from:<br /> - TLSproxy</span></blockquote>
<h4>
Live</h4>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">TLSproxy-LE-agent:<br /> image: 'jrcs/letsencrypt-nginx-proxy-companion:latest'<br /> volumes:<br /> - '/etc/letsencrypt:/etc/nginx/certs'<br /> - '/var/run/docker.sock:/var/run/docker.sock:ro'<br /> volumes_from:<br /> - TLSproxy</span></blockquote>
<h3>
Starting the web app</h3>
Before you start, make sure the hostname you want to use points to the actual IP address - do the DNS config first and make sure it works (wildcard DNS entries FTW!). As an example, let's run a Ghost based blog over HTTPS with automatic redirect HTTP->HTTPS:<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;"> docker run -d -e VIRTUAL_HOST=blog.domain.tld \<br /> -e LETSENCRYPT_HOST=blog.domain.tld \<br /> -e LETSENCRYPT_EMAIL=my.mail@domain.tld \<br /> ghost</span></blockquote>
That's all... you can watch in separate terminal windows as things get set up, just run <span style="font-family: Courier New, Courier, monospace;">docker logs -f TLSproxy</span> and <span style="font-family: Courier New, Courier, monospace;">docker logs -f TLSproxy-LE-agent</span> before starting the first container. LE agent will renew certs for you as well as long as the backend web service (here Ghost) is running. Keep in mind that the certificates persist on the host - in my example in <span style="font-family: Courier New, Courier, monospace;">/etc/letsencrypt</span> directory.<br />
<br />
<h3>
RTFM</h3>
<div>
<br /></div>
This is very simplistic solution and will do just well for many of us, however if you have more specific needs I suggest you review the documentation for<span style="font-family: inherit;"> <a href="https://hub.docker.com/r/jwilder/nginx-proxy/" target="_blank">jwilder/nginx-proxy</a> a</span>nd <a href="https://hub.docker.com/r/jrcs/letsencrypt-nginx-proxy-companion/" target="_blank">jrcs/letsencrypt-nginx-proxy-companion</a> docker images.<br />
<br />
That's all folks!Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-5126924333768454552016-02-18T19:41:00.002+00:002016-02-18T19:41:51.108+00:00Adding private insecure registry to Rancher nodesQuick post before I forget - there's quite a few people asking how to get insecure docker registry running on RancherOS node. Here's what worked well for me.<br />
<br />
First thing that helps a lot is to have some DNS entry for your registry - remember you will use this hostname quite often, so better set it up now than use IP addresses going forward.<br />
As I run my own internal DNS server with local zones, I have created <i>registry.rancher.lan</i> entry and pointed to the node running registry container.<br />
<br />
All of my nodes were already up and running, so I didn't use<i> cloud-config.yml</i> file for that and had to stick to ssh to get it working, but there's nothing to stop you from adding it right there for node installation time. The ssh process is super simple - please note entire command is a single line:<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;">$ sudo ros config set rancher.docker.args "[daemon, --log-opt, max-size=25m, --log-opt, max-file=2, -s, overlay, -G, docker, -H, unix:///var/run/docker.sock, --userland-proxy=false<b>, --insecure-registry=registry.rancher.lan:5000</b>]"</span></blockquote>
I've marked in bold the key element. Be aware, the syntax is quite sensitive if you use quotes. I had multiple crashes on boot because single quote was converted to python(ish) three single quotes, which of course didn't parse well going forward. Clearly the config tool tries to be smart, so please, let it be and remove quotes in parameters passed in the array.<br />
<br />
Finally, reboot and off you go - the node will now find and correctly use the images hosted in your own registry.Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-57064764675731598592015-02-10T09:36:00.001+00:002015-02-10T09:38:15.723+00:00Raspberry Pi 2 - first impressions<br />
<ol>
<li>First impression is that (in my opinion) it is visibly faster than the previous one (1st gen. model B with 512MB RAM), even on tasks that can't use more than one CPU core - this is a good sign. The difference is even more visible when comparing to 1st gen. model B with 256MB RAM...</li>
<li>I measured power consumption at the wall using kill-a-watt type plug and here are the results:</li>
<ul>
<li>No SD card inserted (not booting) - 0.6W</li>
<li>Booted up and idle, with Ethernet connected - 1.4W</li>
<li>CPU cores under load (via <i>sysbench</i> prime number test, with Ethernet):</li>
<ul>
<li>1 thread - 1.8W, 296 sec to complete</li>
<li>2 threads - 2.0W, 148 sec to complete</li>
<li>4 threads - 2.5W, 74 sec to complete</li>
</ul>
</ul>
<li>I tried the Xeon flash bug (or feature) and yes, it works. Of course it doesn't react to LED generated light and the usual laser pointers - I'd love to test it against a real lightning flash but I guess I'll have to wait for weather to change. Here's the Xeon flash test:<br /><br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/6gcNZg7CwK8/0.jpg" frameborder="0" height="266" src="http://www.youtube.com/embed/6gcNZg7CwK8?feature=player_embedded" width="320"></iframe></div>
</li>
<li>Finally, the main sticking point for use experience is slow SD card access, so pick the fastest card you can get - it's worth it!</li>
</ol>
Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-44417092363330924362014-11-09T20:57:00.000+00:002014-11-09T21:00:14.902+00:00haste-server Base URL Hack/PatchRecently I came across <a href="https://github.com/seejohnrun/haste-server" target="_blank">haste-server</a>, a server behind <a href="http://hastebin.com/" target="_blank">hastebin</a>, which is a pastebin clone written in node.js. The application is minimalistic, fairly simple and works really well, except for one rather major glitch - it takes over the root directory of the whole website.<br />
<br />
I've noticed that several people raised an issue on GitHub asking the author for help, but so far nobody shared fully working solution. Some people tried to work reverse proxy magi, others tried to patch the code - with moderate success. Instead of adding to the problem area I thought I'll try to offer a solution - keep in mind I don't know JavaScript ;-)<br />
<b></b><br />
<div>
<b><b></b></b><br />
<a name='more'></a><b><b><br /></b></b></div>
<b>
haste-server architecture</b><br />
<b><br /></b>
This is quite simple - we have haste-server backend implemented in <i>server.js</i> file that handles all dynamic interactions and HTML/CSS/jQuery frontend running int the browser only to call the backend and presents the results. The backend handles all dynamic aspects of the application - key generation, storing and fetching the documents. URIs for actions are fixed in the backend code as well as in the frontend - this is where we are by default.<br />
<br />
The idea of adding a configuration parameter to haste-server (as suggested on GitHub) will not help because the configuration affects only backend process and will not inform the frontend about changes - we would have to generate frontend files on the flight which is an unnecessary complexity.<br />
As we are using a reverse proxy in front of haste-server, we don't need to modify backend code at all. Our reverse proxy will take care of mapping our directory path to the correct URL within node.js. The only changes we need to do are in frontend, that is <i>application.js</i> and <i>index.html</i> to call the URI path we want.<br />
<br />
The trickiest bit was to make it universal - I want my install to run under <i>/paste</i>, you may want <i>/haste</i> and another person will run it under <i>/tools/coding/snippets</i>. The easies way to make it portable was to write a patch generator, so you can apply/revert patch as needed. Pull request coming up, but in the meantime the <a href="https://github.com/tmiklas/haste-server/tree/master/patches/baseurl_change_via_reverseproxy" target="_blank">code is available in my repo</a>.<br />
<br />
<blockquote class="tr_bq">
haste@localhost:~haste-server/patches/baseurl_change_via_reverseproxy$ <i>./haste-baseurl-patch-generator.pl haste</i><br />
<i>Patch file basepath.patch generated - your hastebin will reside in http://<servername>/haste/</i><br />
<i>Please change the directory into haste-server and apply the patch:</i><br />
<i> Install: patch -p0 < baseurl.patch</i><br />
<i> Uninstall: patch -p0 -R < baseurl.patch</i></blockquote>
<br />
That's all, now off to configure your proxy of choice... and to make it easier, here's very basic setup for the 4 most common ones I've seen/used, setting it up for <i>/haste</i>.<br />
<br />
Please note, in all cases the URL will be ending with / to ensure we have a strict match (otherwise any word starting with your prefix - here <i>haste</i> - would match).<br />
<br />
<b>HAProxy</b><br />
<blockquote class="tr_bq">
frontend shared-http-frontend<br />
mode http<br />
bind 0.0.0.0:80<br />
default_backend main_website<br />
# ACLs for request routing<br />
acl acl_haste path_beg /haste/<br />
use_backend haste_backend if acl_haste</blockquote>
<blockquote class="tr_bq">
backend haste_backend<br />
server haste 127.0.0.1:7777<br />
reqirep ^([^\ :]*)\ /haste/(.*) \1\ /\2 </blockquote>
<blockquote class="tr_bq">
backend main_website<br />
server main_web 127.0.0.1:8000</blockquote>
<br />
<b>Nginx</b><br />
<blockquote class="tr_bq">
location ^~ /haste/ {<br />
proxy_buffering off;<br />
rewrite /haste/(.*) /$1 break;<br />
proxy_pass http://127.0.0.1:7777/;<br />
proxy_redirect default;<br />
}</blockquote>
<br />
<b>Lighttpd</b><br />
<br />
This one comes with a caveat - we have to use 2 listeners because as Eric Bouchut pointed out <a href="https://gist.github.com/ebouchut/1939752" target="_blank">here</a>, Lighttpd can't do reverse proxy and URL rewriting at the same time. Luckily Eric was also kind enough to show us how to do it - here's working code:<br />
<blockquote class="tr_bq">
server.modules += ( "mod_rewrite", "mod_proxy" ) </blockquote>
<blockquote class="tr_bq">
# Matching Proxy<br />
#<br />
$HTTP["url"] =~ "(^/haste/)" {<br />
proxy.server = ( "" => (<br />
"servername:80" => # name<br />
( "host" => "127.0.0.1",<br />
"port" => 82<br />
)<br />
)<br />
)<br />
}<br />
<br />
# URL Rewriting Proxy<br />
#<br />
$SERVER["socket"] == "127.0.0.1:82" {<br />
url.rewrite-once = ( "^/haste/(.*)$" => "/$1" )<br />
proxy.server = ( "" => (<br />
"servername:82" => # name<br />
( "host" => "127.0.0.1", # Set the IP address of servername<br />
"port" => 7777<br />
)<br />
)<br />
)<br />
}</blockquote>
Please note, the socket listener on port 82 binds to localhost only. If we had it as ":82" as Eric did, we would have another port open to the world, so either firewall it off... or simply listen or localhost as done above.<br />
<br />
<b>Apache</b><br />
<br />
First we need to enable <i>mod_proxy</i> and <i>mod_proxy_http</i>:<br />
<blockquote class="tr_bq">
a2enmod proxy<br />
a2enmod proxy_http</blockquote>
Then add to the vhost configuration file:<br />
<blockquote class="tr_bq">
ProxyPass /haste/ http://127.0.0.1:7777/<br />
ProxyPassReverse /haste/ http://127.0.0.1:7777/</blockquote>
That's all in principle, just remember to properly set up mod_proxy or you may end up becoming completely open proxy and your machine will be used in DDoS attacks or as proxy for up-voting some crap on the Internet (if you get lucky - it can quickly get worse).<br />
<br />
If you use Apache for that, you should really read about <a href="https://wiki.apache.org/httpd/ProxyAbuse" target="_blank">Apache Proxy Abuse</a> and possibly <a href="http://stackoverflow.com/questions/12407209/configuring-mod-proxy-for-apache-to-reject-wrong-domain-requests" target="_blank">this StackOverflow thread</a>. <br />
<br />
That said, it's your machine, your problem. You've been warned ;-)<br />
<br />
Finally, hat tip to @herkii for discussing ideas.Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-65257283932314618542014-10-28T21:19:00.000+00:002014-10-28T21:19:18.407+00:00Wake-on-Lan issues with Intel PRO Series NICOver the last few months I was experimenting with setting up my ham-radio station for completely remote operation, so once the rare DX comes on air I can work it regardless of where I am at the time.<br />
<br />
The idea seems simple but this means that for a start I need to be able to remotely turn on and off all of the devices. Leaving the design itself for another post, the core element of my remote control concept is rather old ThinkPad x60s laptop. This one comes with Intel PRO/1000 Ethernet NIC and I want to use WoL to boot it up remotely.<br />
<br />
<b>What is WoL?</b><br />
It's a simple way to turn on a machine connected to the network by sending it a single ethernet packet. Very useful if you want to boot up a machine for out-of-hours maintenance run or something similar - like in my case.<br />
<br />
<b>Problem</b><br />
WoL works great but only once, so after you shut down the OS there's no way to do remote start again. This is something that many have encountered judging by the amount of forum posts and questions asked about the same issue.<br />
<br />
Once I wasted more time than I should on trying to figure out what's going on, the fix turned out to be "trivial". Lesson learned for sure.<br />
<a name='more'></a><br />
When I say WoL works only once, I mean it works for the first time since the laptop has a power supply connected to it (WoL is not available at all if power supply is disconnected). The WoL packet is sent from the firewall, machine boots up into Windows 7, I connect to it and do what needs to be done, then shut it down remotely... and WoL no longer works - very annoying!<br />
<br />
Some people reported it happens also if the machine goes to sleep for longer than certain amount of time, so clearly that would be related to power management and sleep modes, however WoL should still work... in theory.<br />
<br />
<b>Solution</b><br />
Of course WoL has to be enabled in BIOS. Then the OS has to support it as well, because OS can change the configuration of the device once it boots up. This OS (mis)configuration is of course a plausible reason for WoL to stop working after OS booted up and can also explain why only removing power supply restores it back to working state (electrical power-down of the NIC controller).<br />
<br />
Browsing the net for possible solutions I came across <a href="http://www.sevenforums.com/network-sharing/201225-wake-lan-not-working-after-long-sleep.html#post2122771" rel="nofollow" target="_blank">this forum post</a> that brings up an interesting tool called <i>powercfg</i>. Two most useful flags in our case are <i>wake_programmable</i> and <i>wake_armed</i>.<br />
<br />
More digging found me <a href="http://download.intel.com/support/network/sb/advset.pdf" rel="nofollow" target="_blank">this PDF</a> detailing advanced settings of Intel PRO Series NICs. As it turned out, all that's actually needed is to change two options and WoL starts working just fine.<br />
<br />
Go to the <i>Device Manager</i> and display properties of the network adapter, go to the <i>Advanced</i> tab and change <i>Enable PME</i> and <i>Wake on Settings</i> options. Please see the PDF for detailed information on what they mean.<br />
<br />
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div style="text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWcPq-qz5_3fJ8NEAZ2PJg1GqYEI5GfEmUhIgbafw00X-Ek5rDgLaI2Hu0hkPoJDHyyh1rV38Ri4dehY5sThZvmE5YLQco98181aUbWesXe8uFuChea9lQorZTCnRkfagXx-bHnOuDcjQ/s1600/PRO1000-PME.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWcPq-qz5_3fJ8NEAZ2PJg1GqYEI5GfEmUhIgbafw00X-Ek5rDgLaI2Hu0hkPoJDHyyh1rV38Ri4dehY5sThZvmE5YLQco98181aUbWesXe8uFuChea9lQorZTCnRkfagXx-bHnOuDcjQ/s1600/PRO1000-PME.PNG" /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkeU8r3d3prP12elA02EqqiGsiQdjfmhsoUJDwo7KOcfiI5JdpPy0JQDuqcXdl80FNnFhYW0NcnBDrcu3EjIpZRGk2dCPFrYbYNjiKBgAdTLq_gNGPGDWytucu6YSF2WyJ7tGBY-gjUqY/s1600/PRO1000-WoL.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkeU8r3d3prP12elA02EqqiGsiQdjfmhsoUJDwo7KOcfiI5JdpPy0JQDuqcXdl80FNnFhYW0NcnBDrcu3EjIpZRGk2dCPFrYbYNjiKBgAdTLq_gNGPGDWytucu6YSF2WyJ7tGBY-gjUqY/s1600/PRO1000-WoL.PNG" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
That's all - worked for me :-)</div>
<div class="separator" style="clear: both; text-align: left;">
Good luck!</div>
Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-27676467665611085312014-03-01T22:31:00.001+00:002014-03-01T22:36:32.838+00:00Running AirView2 [EoL] on Windows 8.1This is purely "note to self" type post for getting End-of-Life AirView2 device (introduced <a href="http://www.ctrl-alt-del.cc/2009/12/airview2-spectrum-analyzer.html" target="_blank">here</a>) to run under the latest version of Windows.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8DvlmPPakZwrlRH_8n1bktLsRKwfaujCK_oJSXvzAzNRWH6RG6fqntP1WMzKWMhR7ivVgtCXYHTg3faA5PIMpax_6Tep34IIWhRNAChhECZgQd4PsJbkB94ChhjMAP_9K2I3P29sGhCE/s1600/AirViewAppFail.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8DvlmPPakZwrlRH_8n1bktLsRKwfaujCK_oJSXvzAzNRWH6RG6fqntP1WMzKWMhR7ivVgtCXYHTg3faA5PIMpax_6Tep34IIWhRNAChhECZgQd4PsJbkB94ChhjMAP_9K2I3P29sGhCE/s1600/AirViewAppFail.PNG" height="145" width="320" /></a></div>
<br />
The AirView2 requires an app and a driver. AirView tool installer (msi format) that checks the OS version and aborts installation if it's different than XP or Vista. The viewer app is written in Java but the AirView2 needs a driver as well (technically it will show up as simple COM port afterwards).<br />
<br />
Manual Installation:<br />
<br />
<ol>
<li>Download and install Java JRE (ouch!)</li>
<li>Download the latest software (32 or 64bit) from <a href="http://www.ubnt.com/airview/downloads" rel="nofollow" target="_blank">http://www.ubnt.com/airview/downloads</a></li>
<li>Manually unpack the MSI file to some location. In command line window this goes like:<br /><i><b>msiexec /a AirView-Spectrum-Analyzer-v1.0.11_win32-setup.msi /qb TARGETDIR=C:\AirView2</b></i></li>
<li>Plug in the dongle into USB port and go to the Device Manager - you will see AirView2 having driver issues. Update driver and tell Windows to look for a new one under <i>C:\AirView2</i></li>
<li>Double click on <i><b>airview-o.jar</b></i> to run the app - happy scanning!</li>
</ol>
<div>
AirView app doesn't care where it was unpacked so you can move it anywhere you want to ;-)</div>
Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-50326788764182182052013-11-25T13:28:00.002+00:002013-11-25T13:35:08.068+00:00LG SmartTV (47LW640S) confirmed to be "snooping"Following some revelations from <a href="http://doctorbeet.blogspot.co.uk/2013/11/lg-smart-tvs-logging-usb-filenames-and.html" target="_blank">DoctorBeet's Blog</a> about LG Smart TVs snooping on our watching habits and further information posted on <a href="http://rambles.renney.me/2013/11/lg-tv-logging-filenames-from-network-folders/" target="_blank">Mark's blog</a>, I realised my parents recently bought one of those... :-)<br />
<br />
First of all we should be rational and assume that any "smart device" is doing that. Unfortunately (for LG) this is pretty bad timing for this kind of news to come out in the light of the recent NSA/Snowden/whatever leaks. Oh well, nothing to see and almost moving on....<br />
Here is a screenshot of traffic from a TV running in Poland, model 47LW640S (also visible in the request headers).<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEilp9o1tc6-6QAwKkLGpThtlhgJx8qTBZgpB_daRj7k0TaW9T5AG7jOuzD-yhn5rEgI_p4nmg6U0zxzGLl3p3AFYo2zQE1g6XmFKacbzo5-o96FcRGAwVsYu_tWSclUPimOsZqo2r1dwb4/s1600/LG1.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="552" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEilp9o1tc6-6QAwKkLGpThtlhgJx8qTBZgpB_daRj7k0TaW9T5AG7jOuzD-yhn5rEgI_p4nmg6U0zxzGLl3p3AFYo2zQE1g6XmFKacbzo5-o96FcRGAwVsYu_tWSclUPimOsZqo2r1dwb4/s640/LG1.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">TV turned ON</td></tr>
</tbody></table>
<br />
<br />
<a name='more'></a><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvuy3_eavSE6niZY8U2lQPkWre0B7FMo2iczsaIMDthqYjVZaHiLxGI9e2Dn0MBaD6OhlqXT67nXezF6c52q8k7P9sADf265kqwEUdyEgC7NqPqVjdzj3ZCnoWUVkUFnP46i7fQ0X4z5Y/s1600/LG2.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="523" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvuy3_eavSE6niZY8U2lQPkWre0B7FMo2iczsaIMDthqYjVZaHiLxGI9e2Dn0MBaD6OhlqXT67nXezF6c52q8k7P9sADf265kqwEUdyEgC7NqPqVjdzj3ZCnoWUVkUFnP46i7fQ0X4z5Y/s640/LG2.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">TV turned OFF</td></tr>
</tbody></table>
<br />
I completely agree that it's none of LG's business to know what files are on my USB sticks or network shares and that when I turn the reporting OFF it should be completely disabled (no exceptions), no reporting of any kind except for maybe periodic checks for new firmware updates. Otherwise than that, the monitoring they do is done on opt-in basis - if we decide to buy a "smart device" we get what we asked for, so the real question is if we users are smart enough to have "smart devices"?<br />
<br />
That's it for me I think. Although the sniffer is still running, the TV is off and I don't have the time to dig into this one too much - unless something spectacular stands out.<br />
<br />
Here are some questions I find somewhat interesting:<br />
<ol>
<li>Which models of LG "Smart TV" snoop on users (if you have confirmed another model does it, feel free to add it in the comment)?</li>
<li>Are there any changes in observable behaviour between firmware versions (I'm sure there will be) on the same model and across models?</li>
<li>Is there a different behaviour based on the country where the TV operates (possible awareness of legal restrictions maybe or simply adaptation to capabilities available in a particular country)?</li>
</ol>
<div>
<br /></div>
<div>
One final note - the option to turn this logging off is called as my father read it to me "Pomoc w chmurce" which in direct translation means "Help in the cloud" but could also mean "Help in a popup/tooltip", at least for non-technical users, which I guess would be quite a few... Ooops!</div>
<div>
<br /></div>
<div>
I'd like to thank DoctorBeet and Mark for their blog posts. Good wake up call guys, well done!<br />
<br />
<b>Update:</b><br />
I forgot to mention that when I base64 decoded the <i>X-Authentication</i> header value it contained ASCII string "(Upx" - go figure :-)</div>
Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-73066916339104766172013-10-26T00:59:00.000+01:002015-05-17T12:04:21.080+01:00Virtual radar - Raspberry Pi and RTL-SDRAs you probably know it's been over a year since Raspberry Pi hit the market. Mine spent most of that year in the drawer, so I decided to see if I can use it as low power server for a particular task. One of the ideas I had for a long time was to build a virtual radar type system that would allow me to see airplanes that fly over my area. There are purpose built systems that are quite expensive so why not to do it on a budget and have some fun with it?<br />
<br />
<h3>
ADS-B Basics </h3>
<h3>
</h3>
<blockquote class="tr_bq">
<b>Automatic dependent surveillance-broadcast</b> (ADS-B) is a cooperative surveillance technology for tracking aircraft. --<a href="http://en.wikipedia.org/wiki/Automatic_dependent_surveillance-broadcast" target="_blank">Wikipedia</a></blockquote>
<br />
This is one of the ways in which the airplanes report their basic flight parameters (identification, current position, altitude, speed, etc) to the Air Traffic Control. All of this happens over radio broadcasts (completely public) at 1090MHz - at lest this is the frequency we are interested in. The full Wikipedia article is quite interesting and contains more information.<br />
<br />
<h3>
Why bother?</h3>
<h3>
</h3>
If you have ever seen the <a href="http://flightradar24.com/" target="_blank">flightradar24</a> website before, you have possibly thought (I know I did) about how cool would it be to do it yourself - own one of those virtual radars and maybe even feed the data into flightradar24 to improve their coverage (and get <a href="https://premium.flightradar24.com/premium/" target="_blank">premium account</a> for free)? If you like this idea, you will like the results your own radar station will produce - in general it would be something like this<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9097Xoi0jszjMRDyFNrYn_aZArwO-t0tkLwr03Nr4UV10o3cg6byozt4N0bm2hkVuFQC-1QLXB2_jvanBtEg9FCtNK_EAvzYTQszNcRraB5y63-F2OXccM4f5jfN8sBkc7embjs17dPw/s1600/ADSB_sample.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="215" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9097Xoi0jszjMRDyFNrYn_aZArwO-t0tkLwr03Nr4UV10o3cg6byozt4N0bm2hkVuFQC-1QLXB2_jvanBtEg9FCtNK_EAvzYTQszNcRraB5y63-F2OXccM4f5jfN8sBkc7embjs17dPw/s400/ADSB_sample.png" width="400" /></a></div>
<br />
All of that is easy to build and the cost is really minimal, using either a PC or a Raspberry Pi as I describe in this post.<br />
<br />
<a name='more'></a><h3>
Design</h3>
<ol>
<li><b>Raspberry Pi</b> - From my point of view, the key benefit of Raspberry Pi is how little power it consumes. Mine never made it above 2W, so that would be something like ~£0.20/month in electricity.</li>
<li><b>SDR Receiver</b> - When Software Defined Radios started showing up I wanted to buy one but the prices were out of my reach back then. Right now it turned out you can get a cheap (but also fairly basic) SDR for about £8. There is several main lines of those receivers (different chipsets, different capabilities - <a href="http://sdr.osmocom.org/trac/wiki/rtl-sdr" target="_blank">see here</a> for comparison) but I found the two below to be the most common:</li>
<ul>
<li>R820T - covers from 24MHz to 1766MHz (this is the one I use)</li>
<li>E4000 - covers from 52MHz to 2200Mhz with gap 1100-1250MHz</li>
</ul>
<li><b>Raspbian</b> - minimal version, actually custom install instead of ready made image I could download. One of the reasons was that I had a spare 2GB SD card and these days stock Raspbian comes as 4GB image. </li>
</ol>
<h3>
General configuration</h3>
<div>
<br /></div>
<div>
My Raspbian build runs only ntpd, sshd and dhcp client on the ethernet port - all other services are turned off.<br />
<br />
The key tool we need is called <i>dump1090</i> and can be found on GitHub. The installation is really straightforward, thanks to an excellent <a href="http://www.satsignal.eu/raspberry-pi/dump1090.html" target="_blank">blog post</a> by <a href="https://twitter.com/gm8arv" target="_blank">David GM8ARV</a>. There is actually several versions of dump1090 - original application created by <a href="https://github.com/antirez/dump1090" target="_blank">Salvatore 'antirez' Sanfilippo</a> and an extended one released by <a href="https://github.com/MalcolmRobb/dump1090" target="_blank">MalcolmRobb</a>. There are differences under the hood as well as in the user interface. First of all, Malcolm's version includes improvements in decoding and processing functions. The difference became apparent when I switched from 'antirez' to 'MalcolmRobb' code base - the amount of airplanes picked up almost doubled, more planes had their flight numbers attached to them, etc. Another immediately visible change is the web interface - airplane icons replaced triangles and a panel was added to show the table with flights currently in range of our station (as seen in the screenshot above). Both versions can also show interactive table with the live data they receive - here's how it looks on my screen.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizn7zeq_QoXaTv6YCMsLGXDIj6uUGCh5JrkBWc2NFx4exBsUjGxlJolkyagSr1dDCpuyUroS3wL-Yh2XPqNnL88xZR6leFIAVGaF3QSpoPsP_bPaulO1OfH4csuo4_bkZbZcxxaDEtmwo/s1600/media-20131025.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizn7zeq_QoXaTv6YCMsLGXDIj6uUGCh5JrkBWc2NFx4exBsUjGxlJolkyagSr1dDCpuyUroS3wL-Yh2XPqNnL88xZR6leFIAVGaF3QSpoPsP_bPaulO1OfH4csuo4_bkZbZcxxaDEtmwo/s400/media-20131025.jpg" width="400" /></a></div>
<br />
<br />
The second element of the configuration is the feed to Flightradar24.com - this is again described on their <a href="http://www.flightradar24.com/dvbt-stick" target="_blank">website</a>. The description covers Windows version only, which I had problems with because the USB dongle didn't want to register almost anything, however <a href="http://www.flightradar24.com/software" target="_blank">this page</a> mentions there's a Linux version available for both x86 and ARM architectures. <br />
Feeding to Flightradar24 requires registration as a data source to obtain a <i>sharing key</i>. Unfortunately I didn't find (yet) how to register using the Linux client or the website, so for the registration I used Windows version of the software and waited till it picked up two planes and activated my account.<br />
<b>UPDATE:</b> According to tweet by Flightradar24.com, at the time I'm writing this the sharing key can be obtained by either running their Windows client or sending an email to <a href="mailto:support@fr24.com">support@fr24.com</a>.<br />
<br />
<h3>
Hello, ADS-B Pi</h3>
<br />
Once everything is installed it is time for a little automation. The Raspbian image I created has both versions of dump1090 installed in <i>/opt</i> and the currently used (MalcolmRobb) is symlinked as <i>/opt/dump1090</i>, so I can switch versions without any problems. The Flightradar24 feed agent is saved as <i>/opt/fr24feed_arm-le_233</i>.<br />
Both applications are started at system boot time, but that happens via <i>inittab</i> instead of regular runlevel and dependency based boot process. I decided to use inittab because the image is built for a particular purpose - almost like an embedded system. Of course I will not let the applications runs as root, because dump1090 implements it's own http server and fr24feed agent is closed source - both will run as an unprivileged user.<br />
<br />
Inittab runs (only once, no restarts) a start script for dump1090 which executes the app as user <i>adsb</i> and redirects the interactive output (flights table) to <i>tty1</i>. Yes, the getty process is still there but I left it running to sort out device permissions (dirty hack - user adsb is added to the tty group to write to tty, if getty was not running device would be owned by root:root). In case the USB dongle is not connected, the app will of course die and will be restarted every 10 seconds. Additionally I moved the web interface from the default port 8080 to more usual 80.<br />
Identical approach was taken with Flightradar24 feed agent, however this one tries to connect to the server (that is dump1090) on port 30003/tcp and if that fails it will keep trying. The main difference is that the agent requires a <i>sharing key</i> to run. I decided to save it in a file located on the <i>/boot</i> partition because it's formatted as vfat and can be accessed from any Windows/Linux/Mac machine, making it easy to add the key. Of course if the file with the key does not exist, the feed agent will not be started. Once the agent starts, its output is sent to <i>tty2</i> for debugging purpose, so the first usable local terminal is <i>tty3</i> :-)<br />
<br />
<h3>
Summary</h3>
<br />
Putting it all together takes no more than 20-30 minutes if you have all the parts. I encourage you to try and build it yourself (follow the articles I have linked in this post). However if you feel it's too much work, you can try to use my Raspbian image, but keep in mind that you get it as-is, without any support, warranty or promises - you have been warned.<br />
<br />
<ol>
<li>Download <a href="https://copy.com/M08z326VFbhx" rel="nofollow" target="_blank">20131014-ADS-B-Pi.img.bz2</a> (SHA1: 94fcb743f2ce02f01197d105f59f4409674eb27f) and decompress it - it is a bzip2 compressed 2GB <i>dd</i> image</li>
<li>Write the image to your SD card</li>
<li>If you have Flightradar24 sharing key, you can save it in <i>/boot/fr24feed.key</i> - there's actually a <i>/boot/fr24feed.key-disabled</i> file with some instructions as well</li>
<li>Connect rtl-sdr dongle, place antenna in a good location, boot up Raspberry Pi</li>
<li>Change passwords for both users - default ones are (login/password) <i>root/adsb</i> and <i>adsb/adsb</i></li>
<li>If all worked ok you should see flights table on tty1, feed upload status on tty2, have a web server running on 80/tcp to watch your airspace and ssh server to manage the it all remotely.</li>
</ol>
<div>
I have to say I am a bit surprised by how little RAM it uses with everything running, but this can only be a good thing :-)</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJnzQZuVRZYwpLiqUhYPr5IYD16K_5-M4iEAiMVT7p8f449gL_qsJO940Vni0UhCkVwBXl-DIrYiAOOuFrAAlqpeaEqq76HzWtfi623Z5IKWeQ2lGTj_a9ax93lNxPKWhhAjpl1RerGwc/s1600/adsb-pi-ram.png" /></div>
<br />
... and if you don't have the parts to build it, you can always see the <strike><a href="http://adsb.ctrl-alt-del.cc/" rel="nofollow" target="_blank">airplanes in my area</a> (busy airspace during the day;</strike> sorry - no longer exposed to the Internet), that is if my receiver is currently on-line.<br />
<br />
Enjoy!</div>
Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-83507993928017588412012-05-19T16:03:00.000+01:002013-09-22T19:40:13.281+01:00Fixing HAProxy configuration in pfSenseSome time ago I was experimenting with <a href="http://pfsense.org/">pfSense</a> and <a href="http://haproxy.1wt.eu/">HAProxy</a> to deploy both as firewall and load balancer for one of the websites I was working on at the time. The key incentive was that pfSense is great BSD based firewall distribution with amazing features offered out of the box, and if that was not enough, you can install additional packages to add features you need.<br />
<br />
One of those packages is HAProxy (proxy/load balancer) and both work together very nicely but...<br />
<br />
<b>Problem</b><br />
<br />
I have installed pfSense with HAProxy several times, more than several in fact... and every time I did it, the configuration file generated using HAProxy web configurator (integrated with pfSense interface) was broken. First row in the table showing defined backends was empty - the configuration file itself had just variables but no values. The second entry was just fine... Obviously HAProxy refused to start.<br />
<br />
<b>Quick fix</b><br />
<br />
Just so I remember next time what I did. Get shell on pfSense console or install file manager package and edit <i>/usr/local/pkg/haproxy.inc</i> to add the line highlighted below:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiSZsf90swU-Ks2b5KATeB0x0rLmNmGeAPU04rheVGV48Rm7aRFi4YbG9-376x3fBtEZJ-_IPJu4YAeTuu9NtrfkzgMXu4qspEckxBmhvQgdhw6PQIBCiMTdKpmxPOhkHtiV105loeLvc/s1600/haproxy-edit.png" /></div>
<br />
Not a rocket science, just shift() the first (empty) backend definition and let the script do the rest. <br />
<br />
<b>Note</b><br />
<br />
Manually fixing config file is pointless because new config is generated every time HAProxy (re)starts and the code above is used to generate it.<br />
<br />Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-25827252422484992552012-05-08T22:49:00.000+01:002013-09-22T21:09:07.954+01:00Raspberry Pi meets Edimax EW-7811Un wireless adapterThis post contains my notes - what I did to make it work properly, so next time I build the system, I have a step by step guide. In case you lived under the rock for the last months and don't know what Raspberry Pi is, you should visit <a href="http://www.raspberrypi.org/">www.raspberrypi.org</a> now.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img alt="Raspberry Pi" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfmwH_rX-pFwHJzZsTDNfWsI3rpaQyYxv6Ptj67kOQjnW1ihu6oPcJv7fgkvsr629RSGoL92lgDDeC2UNgk-qGgxpaFqdM86boHSOHWueWTCLXN5K3gUACGwfvz5i8MfWO6sVqgsRdwLU/s1600/RaspberryPi.jpg" title="" /></div>
<br />
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"></span>I bought <a href="http://www.edimax.co.uk/en/produce_detail.php?pd_id=328&pl1_id=1&pl2_id=44">Edimax EW-7811Un</a> adapter for my Pi - small factor, b/g/n type, so why not... especially when vendor says it provides Linux drivers (wohoo!). Sadly as it turns out, compiling drivers on x86 or x64 and ARM architectures can be totally different experience - especially with Raspberry Pi being in it's software infancy. In all seriousness - Pi is for those that like to solve problems (at least at the current stage), but the community works hard to catch up very quickly - great stuff!<br />
<b></b><br />
<a name='more'></a><b>The wireless adapter</b><br />
<br />
Under the hood, EW-7811Un runs Realtek's rtl8192cu chipset. Kernel module for this chipset is actually included in Debian 6 image (19-04-2012) distributed via Raspberry Pi website, but it will not work with the card. This is for device with USB ID 7392:7811 - there may be other hardware revisions that will.<br />
Actually after the whole procedure described here, running lsusb shows totally different chipset<br />
<br />
<blockquote>
EW-7811Un 802.11n Wireless Adapter [Realtek RTL8188CUS]</blockquote>
so I'm not sure which one it really is... and to be fair, I don't care - it works :-)<br />
<br />
<b>Debian6-19-04-2012 image</b><br />
<br />
Debian image is really nice to start with but it has some issues. Maybe I'm purist but working for some time with ARM devices I learned to value resources they offer. I was a bit surprised how many unneeded things were turned on by default and that SSH was actually disabled. We will get those things fixed a bit further down...<br />
<br />
<b>Putting the bits together</b><br />
<br />
I had to change the procedure a bit, because the whole system wasn't very stable. I was getting a lot of kernel panics and segfaults that were causing Pi to freeze (read hang up) all the time. First suspicion was power supply (I use iPhone PSU - it's rated 5V/1A) but it turned out to be firmware/kernel issue it seems, so we start with fixing it first. Some of the steps below (especially firmware part) were found on the <a href="http://www.raspberrypi.org/forum/">Raspberry Pi forums</a> in thread about XBMC so kudos to their authors - you guys rock!<br />
<br />
Firmware:<br />
<ul>
<li>Start with debian6-19-04-2012 image</li>
<li>Download the <a href="https://github.com/raspberrypi/firmware">latest firmware</a> from GitHub - I used revision a8f8d24</li>
<li>Copy all files from <i>firmware/boot</i> to <i>/boot</i></li>
<li>Replace <i>/opt/vc</i> with <i>firmware/opt/vc</i></li>
<li>Replace <i>/lib/modules/3.1.9+</i> with <i>firmware/modules/3.1.9+</i></li>
<li>Download the <a href="https://github.com/raspberrypi/tools">latest tools</a> from GitHub - I used revision 3aba47b</li>
<li>Copy <i>arm-bcm2708/linux-x86/arm-bcm2708-linux-gnueabi/sys-root/lib/libstdc++.so.6.0.14</i> from https://github.com/raspberrypi/tools to <i>/usr/lib</i> and run: <i>sudo ldconfig</i></li>
</ul>
Now the wireless part:<br />
<ul>
<li>Download the compiled driver module from <a href="http://www.electrictea.co.uk/rpi/8192cu.tar.gz">here</a>, unpack and move to <i>/lib/modules/3.1.9+/kernel/net/wireless/</i></li>
<li>Run: <i>sudo depmod -a</i></li>
<li>We need to block the kernel module that comes with Debian image - edit <i>/etc/modprobe.d/blacklist.conf</i> and add the following line: <i>blacklist rtl8192cu</i></li>
<li>We want the new module to always load on boot, regardless of hardware being present or not - edit <i>/etc/modules</i> and add the following line: <i>8192cu</i></li>
</ul>
Automatically connect to WPA2 network at boot - no GUI needed:<br />
<ul>
<li>Configure wpa_supplicant - edit <i>/etc/wpa_supplicant.conf</i>:</li>
</ul>
<blockquote>
<blockquote>
ctrl_interface=/var/run/wpa_supplicant<br />
network={<br />
ssid="MyWPA2wifi"<br />
scan_ssid=1<br />
proto=RSN<br />
key_mgmt=WPA-PSK<br />
pairwise=CCMP<br />
group=CCMP<br />
# to get encoded PSK run: wpa_passphrase <ESSID><br />
psk=<psk returned by wpa_passphrase><br />
}</blockquote>
</blockquote>
<ul>
<li>Make interface come up automatically - edit <i>/etc/network/interfaces</i>:</li>
</ul>
<blockquote>
<blockquote>
auto wlan0<br />
iface wlan0 inet dhcp<br />
pre-up wpa_supplicant -Dwext -i wlan0 -c /etc/wpa_supplicant.conf -B</blockquote>
</blockquote>
<br />
<b>Various fixes for Debian6-19-04-2012 image</b><br />
<br />
<ul>
<li>Enable SSH at boot if you need it (I do, very much):</li>
</ul>
<blockquote>
<blockquote>
sudo update-rc.d ssh defaults</blockquote>
or rename <i>boot_enable_ssh.rc</i> to <i>boot.rc</i> and reboot - this file is on FAT partition so you can do it even under Windows</blockquote>
<ul>
<li>Broken <i>/etc/apt/sources.list</i> - apt-get complains about duplicate sources, easy to fix - debian sources should be in one line, not two:</li>
</ul>
<blockquote>
<blockquote>
deb http://ftp.uk.debian.org/debian/ squeeze main contrib non-free</blockquote>
</blockquote>
<ul>
<li>Disable services you possibly don't need (I know I don't) but come enabled by default:</li>
</ul>
<blockquote>
<blockquote>
sudo update-rc.d -f portmap remove<br />
sudo update-rc.d -f nfs-common remove<br />
sudo update-rc.d -f xinetd remove</blockquote>
</blockquote>
<ul>
<li>Fix NTP drift file location permissions - Raspberry Pi doesn't keep time (no battery) so it syncs with NTP after every boot:</li>
</ul>
<blockquote>
<blockquote>
sudo chown root:root /var/lib/ntp</blockquote>
</blockquote>
<br />
<b>Summary</b><br />
<br />
Raspberry Pi boots up, brings up wireless interface and connects to the network. After firmware update I have not seen a single kernel panic or segfault yet, which is huge change to how my Pi behaved before. Basically it was dying on any operations that required some more wifi network use (wget was enough), more CPU and/or more RAM... and having 192MB of usable RAM (because we share RAM with GPU) made it really common situation.<br />
<br />
Enjoy!<br />
<br />
<span style="font-size: 0.8em;">Standard disclaimer - it works for me!</span><br />
<br />Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com16tag:blogger.com,1999:blog-1630534171443998118.post-31913211156412938312012-03-04T01:47:00.000+00:002013-08-20T22:14:19.666+01:00Logfile tail the web wayRecently I needed something like web based equivalent of <i>tail -f</i> and <i>tail -n</i> commands, so I could display running tail or last N lines from specific log file. To avoid reinventing the wheel I started looking at previous works on-line and found some interesting bits here and there - one of the most useful being <a href="http://commavee.com/2007/04/13/ajax-logfile-tailer-viewer/">AJAX Logfile Tailer & Viewer</a>, so I based my work on this one.<br />
<br />
The trick is, that as far as it does exactly what I needed, this solution requires web server with PHP... and installing web server (not to mention PHP) is not really what I want on my logserver.<br />
<b><span style="font-size: 1.25em;"><br />Mojolicious to the rescue!</span></b><br />
<br />
<a href="http://www.mojolicio.us/">Mojolicious</a> is a very powerful Perl web framework that comes without bloat (almost unheard of these days!) - all you need is standard Perl interpreter and core Perl modules as they come preinstalled with your Linux distro and you can install Mojolicious - no other dependencies. On Debian systems installation is as simple as <br />
<br />
<blockquote>
apt-get install libmojolicious-perl</blockquote>
<br />
and we're up and running. Writing Mojolicious::Lite app is really simple and the best part is that it comes with it's own, built in web server (operating in several different modes if needed). Sounds like nice way to go - no dedicated web server on the machine, self-contained application, etc. One more thing - writing, testing and deploying the whole code to actual machine took less than 10 minutes!<br />
<b style="font-size: 1.25em;"></b><br />
<a name='more'></a><b style="font-size: 1.25em;">Implementation details</b><br />
<br />
I decided to take HTML and JavaScript elements from the <a href="http://commavee.com/2007/04/13/ajax-logfile-tailer-viewer/">AJAX Logfile Tailer & Viewer</a> as they seemed to do just what I need and because JavaScript is just not my cup of tea so certainly, I wouldn't write it myself.<br />
<br />
All of the code is written as <i>Mojolicious::Lite</i> app, with HTML and JavaScript stored as embedded templates (see DATA section of the script), so all I need to run it is Mojolicious and the script itself - nice, portable solution with low memory footprint when running. Yes, I could use Web Sockets, Comet or any similar technology (Mojolicious supports those out of the box anyway) but I didn't have time to play with it right then - I needed something that will work.<br />
<br />
Note to all Perl purists - I know you won't like the code because I call external (system) tail command to get log lines, but I didn't have time and honestly was too lazy to write it in pure Perl - will fix that in v2.0.<br />
<br />
To keep code listing short, I'll put placeholders for HTML and Javascript elements.<br />
<br />
<blockquote>
<i>#!/usr/bin/perl<br />use strict;<br />use warnings;<br />use Mojolicious::Lite;<br />use HTML::Entities;<br /><br /># logfile we want to see<br />my $logfile = '/var/log/syslog';<br /><br /># Route requests to templates in DATA section<br />get '/' => 'index';<br />get '/js/ajax.js' => 'ajax';<br />get '/js/logtail.js' => 'logtail';<br /><br /># RESTful interface - fixed tail size<br />get '/logdata' => sub {<br /> my $self = shift;<br /> open (IN, "tail -40 $logfile |");<br /> chomp(my @log = (<IN>));<br /> close (IN);</i><i><br /><br /> map { $_ = encode_entities($_) } @log;</i><br />
<i> $self->render(text => join("\n", reverse @log));<br />};<br /><br /># variable tail size<br />get '/tail-n/:N' => sub {<br /> my $self = shift;<br /> my $N = $self->param('N');<br /> if ($N =~ /\D/) {<br /> # command injection attempt?</i> <i><br /> $self->render(text => "Y U NO GIVE UP, NICE TRY!");<br /> } else {<br /> open (IN, "tail -$N $logfile |");<br /> chomp(my @log = (<IN>));<br /> close (IN);<br /> map { $_ = encode_entities($_) } @log;<br /> $self->render(text => join("<br/>", reverse @log));<br /> }<br />};<br /><br /># cookie encryption passphrase - no use here but if missing it produces warning :-)<br />app->secret('youcansafelyignorethisone');<br />app->start;<br /><br />__DATA__<br />@@ index.html.ep<br /><!-- here goes all the index.html contents --><br /><br />@@ ajax.js.ep<br /><!-- yes, you guessed it --><br /><br />@@ logtail.js.ep</i>/* an ajax log file tailer / viewer<br />
copyright 2007 john minnihan.<br />
<br />
http://freepository.com<br />
<br />
Released under these terms<br />
1. This script, associated functions and HTML code ("the code") may be used by you ("the recipient") for any purpose.<br />
2. This code may be modified in any way deemed useful by the recipient.<br />
3. This code may be used in derivative works of any kind, anywhere, by the recipient.<br />
4. Your use of the code indicates your acceptance of these terms.<br />
5. This notice must be kept intact with any use of the code to provide attribution.<br />
*/<br />
<i><!-- original disclaimer, the rest is as above --></i></blockquote>
<br />
That's it! Keep in mind that you have to customize a bit <i>logtail.js.ep</i> part - function <i>getLog</i> has <i>url</i> variable you need to point to <i>/logdata</i> provided by our script. You can also specify how often the AJAX call will be made to fetch log data - this is done in <i>startTail</i> function. I use 2000ms value and it's well enough, if not too often anyway - tune it so you won't get more than 40 lines in the log during this time... or tune for the maximum smoke - your call.<br />
<br />
<b><span style="font-size: 1.25em;">How it works?</span></b><br />
<br />
Built-in web server will respond to all paths defined with <i>get '<path>'</i> statement. Those that are routed to templates, will respond with templates (which can have dynamic content as well but that's out of scope here). Those with defined subroutines will get the code executed - no magic here.<br />
<br />
Index page pulls in two JavaScript files (all template based), <i>logtail.js</i> requests data from first subroutine responsible for <i>'/logdata'</i> and this one is refreshed as per timer in <i>startTail</i> function. <br />
<br />
Second subroutine is used to display static log chunk that won't refresh itself automatically - in case you are debugging something, the last thing you want are disappearing logs. This one is manually called by the user as <i>http://scrpt_url/tail-n/<lines to display></i>. Just in case someone had the idea to run script as root (command injection could be deadly!) the script will terminate if provided number of lines contains non-digits.<br />
<br />
<b><span style="font-size: 1.25em;">Running the app</span></b><br />
<br />
You can run it in many ways, but for small deployments (like mine) this is entirely enough:<br />
<br />
<blockquote>
./webtail-ajax.pl daemon</blockquote>
<br />
This will start listener on port 3000 (default, can be changed with command line parameter).<br />
<br />
<b><span style="font-size: 1.25em;">Security warning</span></b><br />
<br />
Logs can contain data that is not safe to be displayed via web interface as-is - think of XSS for example. At best, you will get popup, at worst... well, much worse. This is why I've added <i>encode_entities()</i> from <i>HTML::Entities</i> to the script - current version escapes at least the basic elements but you can decide which ones you want to encode - see module documentation for details.<br />
<br />
<b><span style="font-size: 1.25em;">Credits</span></b><br />
<br />
Big thank you goes to Sebastian Riedel (<a href="https://twitter.com/kraih">@kriah</a>) for his work on Mojolicious which simply rocks and John Minnihan who wrote the HTML and JavaScript I used... as well as and many others that gave me some ideas but the approach they proposed was sadly not acceptable in my usage scenario.<br />
<br />Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-79970178901042890832012-02-23T22:06:00.000+00:002013-09-22T21:25:20.327+01:00Secure backup of untrusted remote hostsI didn't blog for a long time, so it will be a long post caused by some nightmares I had about not doing proper backups on some of my hosts.<br />
<br />
Servers - all those small and big machines most of the geeks own, run or operate. As VPS pricing drops, we see more and more of those low-end, resource strapped servers. Organic growth usually means you start with empty server, some kind of definition what it will be doing and... from there it just goes downhill. How do you backup such VPS? Here is something I use myself.<br />
<br />
<span style="font-size: 1.25em;"><b>My backup requirements</b></span><br />
<br />
<ul>
<li>Automated - it has to run without supervision in roughly regular time intervals, if it's not automated it will never be done (read no backup)</li>
<li>Off-site - in case I loose the whole machine for some reason (because RAID is not backup and what fire doesn't destroy, water poured by firemen will)</li>
<li>No Cross-Backups - because they require trust relationship between machines and if you think about using cheap VPS'es for cross-backups, remember that you get what you pay for!</li>
<li>Automatically delete old backups - to save space, (my) time and money</li>
<li>Append only - machine can only write data to its own, designated backup volume but can not delete or modify other volumes (accidents and rogue users do happen)</li>
<li>Confidentiality - no unauthorized access backed up data</li>
<li>Availability - storage volume has to be highly available so I can not only write to it knowing it's there, but also access backups when I need them</li>
<li>Access controls - ability to define granular access rules and enforce append-only usage</li>
<li>Economy - it has to have reasonable cost</li>
</ul>
<b style="font-size: 1.25em;"></b><br />
<a name='more'></a><b style="font-size: 1.25em;">Proposed solution</b><br />
<br />
Server creates tarball with files I want to copy using simple shell script triggered from cron. File created is encrypted with GnuPG using the key of my backup user and the private key is stored off-line. Encrypted file is uploaded to off-site storage volume.<br />
<br />
As I used Amazon AWS before, this was my first choice. The company is big enough to do quality job, offers all the building blocks and pay-per-use is just what I need. By combining together services from Amazon I can satisfy most of the requirements out of the box and easily add what is missing.<br />
<br />
Amazon S3 is a storage solution that allows you to put your files into 'buckets' (think file shares) with globally unique names. Each bucket has series of properties - for example geographical location, so you can select where your data will reside (thinking of legal stuff and price differences across locations), object expiry time which will be our auto-delete mechanism for old data and finally ACLs. Because those ACLs are not enough for what I want to do (or rather how I want to have it done) I will be using IAM service that nicely integrates with S3 and many other AWS services, so let's get it set up.<br />
<br />
<span style="font-size: 1.25em;"><b>Setting up S3</b></span><br />
<br />
I create separate S3 bucket for each host, so I can select location and different expiry times easily. I decided to name buckets after hosts's FQDN and add '-backups' suffix, so for this blog post I have bucket called <i>aws-poc.home.lab-backups</i>. In bucket properties we are interested in the object expiry time configuration. Simply add the rule as seen below. If you leave prefix empty, it will affect all objects in the bucket - which is exactly what I want - retain backups for 180 days.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img border="0" height="170" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWBrqPV09B3XMb3FBwq5Jyl6Vxo6Jis3rTFEWBL6ivRYlLzvKolH5J6zBWRh5JF3AcHqz2JLxY_IObzKGHQw7k5ov6qS1NOoTehNpto5Talzs0XpVVfaiAzKYvoTKV0WNgxVWcz2pbxZc/s640/s3lifecycle.png" width="640" /></div>
<b style="font-size: 1.25em;">IAM configuration</b><br />
<br />
Uploading to S3 via web service requires providing user's Access Key ID and Secret Access Key. For each server I want to back up I need separate IAM user - this will allow me to tell them apart and revoke access to backup bucket if needed. IAM allows us to grant every IAM users and groups right to perform or deny certain actions, like 'allow to upload files only to bucket X, block all other bucket operations' - we do that below.<br />
<br />
After creating the user in IAM service (yes, IAM, not S3), remember to write down the access keys - they can't be displayed later - you will have to generate new keys (see user properties).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgcMbx8cH1o7MvDtmnQQSAoPz9QhznRbE4ckkJaGOm1jl69K_hnFfSyp6RcqVpZE-rbmtoad4hgdEoH1h15xGObE3gbuDx9S_-hHifDCd8Lek27SpIBAkyo7MrY0sXDwHPprJVzoHLHJG4/s1600/iam-keys.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="293" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgcMbx8cH1o7MvDtmnQQSAoPz9QhznRbE4ckkJaGOm1jl69K_hnFfSyp6RcqVpZE-rbmtoad4hgdEoH1h15xGObE3gbuDx9S_-hHifDCd8Lek27SpIBAkyo7MrY0sXDwHPprJVzoHLHJG4/s640/iam-keys.png" width="640" /></a></div>
<br />
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"></span>Now we need to define what the user can do. In user properties under <i>Permissions</i> tab, we select <i>Attach User Policy</i> and choose <i>Policy Generator</i>. To have append-only access to our S3 bucket we need to grant user access to <u><i>PutObject</i></u> action (and only this one) and specify ARN of our S3 bucket. This is the minimum we need to do.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img border="0" height="477" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtFUZ4aMJafaa56kM9jpAGAiLIgsHJ60dNfnzIrByFO4pIYmz7RBaLUXF0a0xp3SItvp7hA-ylaL0T9c5sL2mRwgIV7GbXNKrl6P98CmstMBJFi3mVquTDzctxLo1icFUmeOFoxAHnhuM/s640/iam-policy.png" width="640" /></div>
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><br /></span>
<span style="font-size: 1.25em;"><b>Backup and upload scripts</b></span><br />
<br />
Backups scripts are really easy - just tar and gzip directories as needed so they contain what is to be backed up, pipe that via gpg and save somewhere for a short time... Then upload to S3 and you can delete original encrypted tarball. For example it can be done this way:<br />
<br />
<blockquote>
#!/bin/bash<br />
#<br />
# this is updated version that adds file hash to the name<br />
# so once file was uploaded and source data changed, <br />
# potential attacker can't overwrite files already uploaded<br />
#<br />
WORKDIR=/tmp<br />
DATE=`date +%Y%m%d`<br />
HOSTNAME=`hostname --fqdn`<br />
cd $WORKDIR<br />
tar cf - /etc /var/backups 2>/dev/null | bzip2 -9 | gpg -e -r backups > tmpbackup<br />
SHA256=`sha256sum tmpbackup | awk '{ print $1; }'`<br />
BACKUPFILE=$DATE-$HOSTNAME-$SHA256.tar.bz2.gpg<br />
mv tmpbackup $BACKUPFILE<br />
s3upload.pl $HOSTNAME-backups $BACKUPFILE && rm $BACKUPFILE</blockquote>
<br />
That's all - the upload is done by s3upload.pl script:<br />
<br />
<blockquote>
#!/usr/bin/perl<br />
use strict;<br />
use warnings;<br />
use Net::Amazon::S3;<br />
<br />
# requires:<br />
# apt-get install libnet-amazon-s3-perl libwww-perl libxml-simple-perl<br />
<br />
if ($#ARGV < 1) {<br />
print "Usage:\n\t$0 <bucket name> <file name>\n";<br />
exit 1;<br />
}<br />
<br />
my $s3 = Net::Amazon::S3->new({ <br />
aws_access_key_id => "INSERT KEY ID HERE",<br />
aws_secret_access_key => "INSERT SECRET KEY HERE",<br />
});<br />
<br />
# upload or die<br />
my $bucket = $s3->bucket($ARGV[0]);<br />
$bucket->add_key_filename($ARGV[1], $ARGV[1]) or die $s3->err . ": " . $s3->errstr;<br />
exit 0;</blockquote>
<br />
<span style="font-size: 1.25em;"><b>Caveats</b></span><br />
<br />
To run gpg in the way I do above, importing the target key is not enough - you have to edit the imported key and set trust level to ULTIMATE or every time the script runs, you will have to interactively confirm that you are sure you want to encode data. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img border="0" height="160" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtGGy5Q2is0LPtp9WUUQrtke3mmV5JGlf_GVKSoUUPIBQTSaKAe4ibliIhPN1PWvqTssifdZVx0FWQzfPsUHwGSh6iEW-U11D_lZ6lhYrKoEmMUetIgkfa087Itokpu3Uz7x-vzt8S2dw/s640/gpg-key-trust.png" width="640" /></div>
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><br /></span>
To change trust level for the above key I did:<br />
<br />
<blockquote>
gpg --edit-key backups<br />
trust<br />
5 <== for ultimate trust<br />
quit</blockquote>
<br />
That's all, now the key has ultimate trust and the process can be fully automated - no more questions asked.<br />
<br />
<span style="font-size: 1.25em;"><b>Closing notes</b></span><br />
<br />
The old saying says there are two kinds of people - those who do backups and those who will do backups. In fact there is a third kind - those who test their backups... so please, test your backups, see if you can restore data, or otherwise you have just wasted your time and money to buy false sense of security.<br />
<br />
<b><span style="font-size: 1.25em;">UPDATE:</span></b><br />
As the <i>PutObject</i> permission allows to overwrite already existing files, it's desirable to have unique file names that can't be easily determined/guessed. I have updated the backup script above to have to so calculate SHA256 hash of encrypted backup file and add resulting hash to the file name. This is just a result of my paranoia - better be safe than sorry :-)<br />
Another update is for s3upload.pl - it is more generic right now, taking two parameters - bucket name and file name from the command line passed as parameters, so you can use it as well for uploading other things than backups and it will work ok.<br />
<br />Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-36231003444043629862011-02-17T09:26:00.000+00:002013-09-22T21:28:21.848+01:00How To Outrun A Lion?<i>You don't have to outrun a lion - it's enough you outrun the guy running next to you.</i><br />
<br />
Funny enough, the same stands for securing your IT infrastructure - if you are in the "low hanging fruit" category, you get owned for sure - possibly before you even notice anything shady going on behind your shiny website. When you raise the bar a bit and step out of the damned circle, most of the attackers will give up on you and move to find some other target that is easier to compromise.Of course that doesn't work for determined attackers that want YOU and nobody else, but that's a story for another time.<br />
<br />
<b>What's that smell?</b><br />
<br />
It's a smell of FAIL my friend...<br />
<br />
Just recently I was helping two of my friends and doing some forensics on their servers (or rather on what was left out of them) after they noticed something strange was going on. Long story short, the key part is that the attackers owned those boxes for months before they were discovered. They got in via path of the least resistance - badly written PHP web apps (there's so many of them!), dropped c99 or similar shell and owned the box to their liking.<br />
<br />
In general, we suck really bad if it takes us months to detect such hacks.<br />
<br />
<b>Here come the benefits of scale</b><br />
<br />
Wherever and whenever I look at any shared hosting providers, dedicated servers and alike, their default configuration is wide open by default. As long as the box is on-line and Nagios doesn't report issues, nobody is actually checking what's going on that box. Basically operators don't care - they provide functionality and they charge you for it. Oh yes, that's exactly what they do - charge you first and then provide a ton of stuff you don't need and don't use - unless you are an attacker that is :-) <br />
<br />
<br />
Plenty of dangerous PHP functions enabled, dumb/bad configuration of network services and often the networks itself, total lack of monitoring (except for Nagios)... and all of that provided by default, just in case a customer comes back and says 'oh, that breaks functionality I need'; all because that would mean they (operators) have to go back and spend some time on enabling it later. Sure, it's easier to blame it on the "bad hackers in my interwebz" - great business model guys! I believe, that if you build the security into your system from the start, your TCO will be lower than going with defaults (loss of clients due to compromise, cos of bringing system back in service, etc) but that's a business decision of course.<br />
<br />
Default configs are similar to default passwords.<br />
<br />
<b>Improving security posture</b><br />
<br />
If you are on a shared hosting platform, there's not much you can do really. It's a <i>shared </i>host, so you (or rather the operator) has to find the common denominator - something that will satisfy everybody using this particular host. It's about finding the weakest link and bringing everybody else down to the same level - not good.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img border="0" height="478" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrhgS0CuO2HW6Ya9sdVITpx5E533TP1XJp3FgwH_iDs4NhmyZbSeK9N-Z8w2w2j0MNPyBBNRDFNaE9fE-6MKv-HqTfbTmE5bKpiAtfFkEghd0sbHxBLxXIF4kZZmpDx5E4JcYC8XHQ6yM/s640/weakest-link.jpg" width="640" /></div>
<br />
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"></span>If you go with VPS or dedicated server, you can change a lot and it won't cost you a lot of money. Simple things can improve your posture and make it much harder for the attackers to run loose on your servers. Here are just three things you can do for free...<br />
<br />
<b>Egress filtering</b><br />
<br />
Do you have an outbound firewall policy set to DROP by default? Can you imagine that in datacenter environment? Can it work well or will be a huge PITA?<br />
<br />
Yup, easily doable and not that painful if you think about it. If we consider Linux, you can use <i>iptables</i> for that and I guess you already do have an <i>iptables</i> firewall of some sort that filters inbound packets. Let's extend it a bit - example below is for a simple web server:<br />
<br />
<blockquote>
<blockquote>
<span style="font-size: 1em;"># fail close - just in case</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;">/sbin/iptables -P OUTPUT DROP</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;">/sbin/iptables -F OUTPUT</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;"># allow responses - majority of traffic comes here so it's a first rule</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;">/sbin/iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;"># allow from self to self</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;">/sbin/iptables -A OUTPUT -d 127.0.0.1 -j ACCEPT</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;"># allow DNS servers listed in /etc/resolv.conf</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;">for DNS in `grep "^nameserver" /etc/resolv.conf`; do</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;"> if [ $DNS != "nameserver" ]; then</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;"> /sbin/iptables -A OUTPUT -p udp --dport 53 -d $DNS -j ACCEPT</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;"> fi</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;">done</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;"># allow SMTP out to email admins</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;">/sbin/iptables -A OUTPUT -p tcp --dport 25 -d $DOMAIN_MX -j ACCEPT</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;"># allow NTP outbound, local NTP is nice to have!</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;">/sbin/iptables -A OUTPUT -p udp --dport 123 -d $LOCAL_NTP -j ACCEPT</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;"># allow connections to our Linux repository mirror for updates</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;">/sbin/iptables -A OUTPUT -p tcp -d $LINUX_REPO -j ACCEPT</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;"># generic log and drop all</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;"># /etc/syslog.conf => kern.=debug /var/log/firewall</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;">/sbin/iptables -A OUTPUT -j LOG --log-level 7 --log-prefix "FW-DROP: "</span><span style="font-size: 1.25em;"><br /></span><span style="font-size: 1em;">/sbin/iptables -A OUTPUT -j REJECT --reject-with icmp-host-prohibited</span><span style="font-size: 1.25em;"><br /></span></blockquote>
</blockquote>
<br />
<i>Simple?</i> Yes!<br />
<i>Does it raise the bar?</i> Yes!<br />
<i>Do I have to write IP addresses everywhere?</i> No - iptables will resolve hostnames used in rules and I've noticed that for example if my <i>$LINUX_REPO</i> has several IP addresses, iptables actually created an entry for each of them.<br />
<i>But I can't do anything else!</i> That's exactly the point - you shouldn't do anything else on a web server, unless there is a justified need for that (say access SQL database on another host, etc).<br />
<br />
Wrapping up - all your web traffic (responses from web server and other services hosted here) will go into state matching rule, then you care for DNS, access to your own MX (only this one unless you have very good reason to do otherwise), NTP and distro updates are really nice to have, then drop all the other traffic. You could add rules for remote (off-site) logging, so you know when something tries to call out/pops your box.<br />
<br />
Now, when an attacker drops his php shell he is pretty much very limited (no call back home, no portscans, no IRC bots, etc), unless he escalates access to root, but hey - how about a network based firewall implementing above?<br />
<br />
<b>Server hardening</b><br />
<br />
Wow, you could write a book on that, but let's stick to the basics:<br />
Install only the software you really really need (do you need that gcc and all dev libraries to run your web server) - remove what was installed and is not needed - you can always put it back if you need it later!<br />
Turn off all services that shouldn't be running - my rule of thumb is to bring the system to the point, where I can run it entirely without any firewall, because there is no services to hide.<br />
Keep your software updated - cron is your friend (to see what updates are available)<br />
Kernel hardening - <a href="http://fedoraproject.org/wiki/SELinux">SELinux</a> and <a href="http://grsecurity.net/">Grsecurity</a> (+RBAC) seem to be the key candidates here. Yes, that can take a lot of time to set up, but in most cases it's well worth it.<br />
<br />
Just try to imagine how annoying it has to be for an attacker to own the box via web app, get root via local privilege escalation and not be able to install his rootkit (and hide) because kernel is monolithic (no loadable modules support) and has grsecurty baked in, with IP logging on resource overstep and other nice features it offers. <br />
<br />
BTW here's the funny note left in one of the toolkits I lifted from one of friend's servers - what you make out of it is up to you. Oh... and credit to Ingo Molnár for his exploits and awesome comments in their source code ;-)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9kB4-KFhaTR_ez6r4ZACD8mge2jc95CNXxiN8RLh8se_l5TtgUo0LUh76XKnU2MaZ8epKgt7NrhP8WWH8DRkwO35YslKkO0BuT6xRHOwkfXKdku8dkuhsvBFmp6BUIVqAl00RpWF6esg/s1600/funny.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="260" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9kB4-KFhaTR_ez6r4ZACD8mge2jc95CNXxiN8RLh8se_l5TtgUo0LUh76XKnU2MaZ8epKgt7NrhP8WWH8DRkwO35YslKkO0BuT6xRHOwkfXKdku8dkuhsvBFmp6BUIVqAl00RpWF6esg/s640/funny.jpg" width="640" /></a></div>
<br />
<b>Logging and monitoring</b><br />
<br />
Best things are free right? How about using syslog that comes with the system to send the logs off-site? Make a small box somewhere and simply pump it all out, so you have an off-site record in case of unwanted guests showing up.<br />
Not enough bandwidth you say? There's an app for that - pump logs out via OpenVPN using LZO compression with or without encryption (hint: you can set the cipher to none) and as my test show, this can drop your logging bandwidth by around 80% and on top of that you can do traffic shaping in OpenVPN itself.<br />
<br />
Now, having logs and not looking at them is a waste of resources, unless you are "checkbox security" organization and need it for compliance on paper... Depending on your pocket condition, you can use simple scripts to get what you need or get some free tools that sift through and visualize large amounts of data. For example <a href="http://www.splunk.com/">Splunk</a> has a free edition (up to 500MB raw log input per day) and there is many other (mostly paid for) products that you could use. Even "cloud based" services like <a href="http://www.loggly.com/">Loggly</a> (also offers free developer account) are available these days - simply pick something that works for you.<br />
<br />
<br />
It is not a rocket science - it's really about common sense, so calm down and carry on.<br />
<div>
<br /></div>
<div>
<br /></div>
<div>
<b>UPDATE:</b><br />
As <a href="http://twitter.com/denishowe">@denishowe</a> pointed out "it seems we need a checklist for dumb providers with the list of things to disable and another checklist for dumb users, so they can enable what they really need" - yes, that might just work :-)</div>
<br />Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-1420168758381772792010-11-24T22:15:00.000+00:002013-09-22T21:30:30.310+01:00Building cheap console serverThis time from the department of almost wasted time...<br />
<br />
We all know that serial ports come very handy when you need to (re)configure something like a switch/server/firewall or similar device. In theory you can do that over TCP/IP nowadays with one hint - you need to have connectivity. All would be ok if not the fact that those very switches/firewalls you want to reconfigure actually provide the connectivity you need :-)<br />
<b><br />The Idea</b><br />
<br />
Now... why spend hundreds of pounds/dollars on off-the shelf kit? Sure, it's cool, properly built and works unless you mess it up, but where's the fun part?! Today I needed a very very quick and cheap solution, so:<br />
<br />
<ol>
<li>SheevaPlug - £114.00</li>
<li>13-port USB hub - £19.99</li>
<li>USB-serial dongles (pl2303) - £14.99 each</li>
</ol>
This way I have fully networked console server with 4 ports just under £200 - acceptable, especially when the whole thing is running off DHCP and calls home via OpenVPN - very easy to deploy!<br />
<br />
<b>Tricky bits</b><br />
<br />
Generic Sheeva has one USB host port and hub has 13 of them - I want to send it off to remote location and have somebody plug it in and not mess up what's where. Trick is to write appropriate udev rules to detect adapters and give them <i>ttyUSBn</i> names according to physical port on the hub.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjiBqVt5SQPfxd2C2TsA-VJjcoOSEBlAOE9rrKw46i8CO98Cnj4KHW546DDfb2G5z1oCD6e9-5JK_qRBfT2_xQH-vwZcQ7bjdtSTKnqYwb9taTDSfZD5_QcnkI5dzvGtgquYx1qN1jhwyo/s1600/13x-usb-hub.jpg" /></div>
<br />
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"></span>All would be fine and easy if it worked as documented - sadly it doesn't. First problem was that <i>ATTRS{devpath}</i> (as returned by <i>udevadm info --attribute-walk -n /dev/ttyUSBn</i> that allows to distinguish usb ports) was used by rule in tests but wasn't propagated properly on none of my Debian or Ubuntu boxes. Then I tried to match <i>KERNELS</i> for parent devices - nope... if you go too far up the tree it doesn't see s**t :-/<br />
<br />
<br />
<b>The Solution</b><br />
<br />
Finally I got the working rule set - long story short, here it is:<br />
<br />
<span style="font-size: 0.8em;">KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.1:1.0", NAME="ttyUSB0"<br />KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.4.1:1.0", NAME="ttyUSB1"<br />KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.4.2:1.0", NAME="ttyUSB2"<br />KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.4.3:1.0", NAME="ttyUSB3"<br />KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.4.4:1.0", NAME="ttyUSB4"<br />KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.3.4:1.0", NAME="ttyUSB5"<br />KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.3.3:1.0", NAME="ttyUSB6"<br />KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.3.2:1.0", NAME="ttyUSB7"<br />KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.3.1:1.0", NAME="ttyUSB8"<br />KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.2.4:1.0", NAME="ttyUSB9"<br />KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.2.3:1.0", NAME="ttyUSB10"<br />KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.2.2:1.0", NAME="ttyUSB11"<br />KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.2.1:1.0", NAME="ttyUSB12"</span><br />
<br />
I had to use <i>KERNELS</i> match as above to have variables seen by the rule. I still don't know (and at this moment don't care any more) why it didn't work as documented...<br />
<br />
The bottom line is that it works, it can be done way cheaper than commercial solutions, literally at the fraction of cost - if you don't mind the spider-ish look of it :-)<br />
<br />
<br />
<b>Update:</b><br />
Hat tip to <a href="http://twitter.com/herkii">@herkii</a> for pointing out <a href="http://kitenet.net/~joey/blog/entry/random_tip:_per-port_naming_for_identical_USB_devices/">another approach</a>.<br />
<br />Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-52358468537297524292010-07-31T02:59:00.000+01:002013-08-20T22:05:18.114+01:00Making new friends with kippoLess than two weeks ago I've sent a tweet asking for honeypot recommendations. I wanted to play a bit with something new, something I never did before, mostly because I never had time for it (right, like I have it now). Anyway, thanks to all the great people that replied to my tweet I've learned a lot and found some great software. Now it's time to give something back to the community.<br /><br /><b>Kippo - simply amazing</b><br /><br />First honeypot I've reached for was <a href="http://code.google.com/p/kippo/">kippo</a>. It is a medium interaction SSH honeypot designed to log brute force attacks and log the whole session as it goes - including timings, typos, etc. The magic sauce is that you can play the session back (with typos!) and see what the attackers are made of. Believe me - playing back those session is totally amazing! Some samples are available on project's page.<br />There are also other features to like, like trapping sessions and not disconnecting them even if bad guys do logout, logging ssh client used (very easy to tell scanning bots apart from real people), quite nice interaction and most of all easy way to extend your honeypot it with your own commands.<br /><br/><br/><b>Installing kippo</b><br /><br/><br /><br/>For the base platform I used one of my Debian hosts and started with<br/>kippo 0.4. It was good to see how to run it, but options are limited,<br/>so go full steam ahead and get SVN version - it is well worth it!<br /><br /><br/>By default kippo runs on port 2222 but I wanted it on port 22 as normal<br/>SSH would be (running as unprivileged user), so I've set it up on one<br/>of my unused IP addresses - the setup was very easy.<br /><br/><br /><br/>Before you grab the latest version from SVN repo, you should install<br/>required python packages (dependencies will be pulled in<br/>automagically): <br /><br/><br /><br/><blockquote><tt>apt-get install python-twisted</tt><br /><tt><br/>svn checkout http://kippo.googlecode.com/svn/trunk/ kippo-read-only</tt><br /></blockquote><br/><br /><br/>Main benefit of SVN version is that it can use MySQL to log events<br/>(alongside the regular log file) and that it can actually bind to given<br/>IP address - version 0.4 binds to all available addresses which is a<br/>bummer for me when I want to spawn totally fake host and have normal<br/>ssh working as well.<br /><br/><br /><br/><b>Honeypots - rule #1</b><br /><br/><br /><br/><font color="red">DO NOT run honeypot as root!</font><br /><br/><br /><br/>Remember that honeypots are software components, they may (and most<br/>likely do) have their own bugs. Of course you have to be root<br/>to bind to port <1024, or do you?<br /><br/><br /><br/><b>Configuration</b><br /><br/><br /><br/>Couldn't be easier... create unprivileged, regular user account to run<br/>your honeypot (I called it honeytrap), create your own <tt>kippo.cfg</tt> using<br/><tt>kippo.cfg.dist</tt> as template, set MySQL parameters, honeypot hostname<br/>(attackers will see it after they log in), IP address to bind to and<br/>port. If you don't want to use MySQL - your call... it may come very<br/>handy for reporting. That's it - you are ready to go.<br /><br/><br /><br/>Now the trick is to get it running on port 22. There is obviously more<br/>than one way to do it. If you have only one IP address available, you<br/>should most likely go to kippo's Wiki page that describes how to <a href="http://code.google.com/p/kippo/wiki/MakingKippoReachable">make<br/>kippo reachable through port 22</a> but if you<br/>have spare IP address... =B-]<br /><br/><br /><br/>Now, how do I bind to port 22 as regular user? Somebody must have solved that problem before, right?<br/>Sure, and they even created a package that solves this issue! It's called authbind and it's amazingly easy to use.<br /><br/><br /><br/><blockquote><tt>apt-get install authbind</tt><br /><tt><br/>touch /etc/authbind/byport/22</tt><br /><tt><br/>chown honeytrap:honeytrap /etc/authbind/byport/22</tt><br /><tt><br/>chmod +x /etc/authbind/byport/22</tt><br /></blockquote><br/><br /><br/>Authbind works almost like sudo, except for the ports - not the<br/>commands. In kippo's directory you will find start.sh script - add<br/><tt>authbind</tt> in front of the startup command and you are good to go.<br /><br/><br /><br/><b>Ready, set, go!</b><br /><br/><br /><br/>Ok - don't do my mistake... test your install - ssh into the honeypit and<br/>see if you can log in. The root password is in kippo.cfg. Testing setup<br/>is important - if sql database is gone, then you won't log in and in<br/>kippo.log it will say that the root password was incorrect, when in<br/>fact the problem is disconnected sql log backend.<br /><br/><br /><br/><b>Observations</b><br /><br/><br /><br/>Kippo is really great tool to learn what the bad guys are up to. If<br/>they add user, they can log in as that user later. If they change root<br/>password, it will be there for them when they return. The best part is<br/>that of course you can see those passwords and suddenly you will have<br/>new 'accounts' added and new, correct and active at the same time root<br/>passwords (yes, more than one correct password!). <br /><br />I've spent some time<br/>watching the sessions recorded so far - there's so much to see, laugh<br/>and cry, but I'll leave that for another post...<br /><br />BTW, I blame Andrew (<a href="http://twitter.com/Infosanity">@Infosanity</a>) for all of that - he got me back to honeypots topic, then other great tweeps came back with advice (much appreciated), so make sure to visit <a href="http://blog.infosanity.co.uk/category/honeypot/">his blog</a> ;-)<br/>Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-48101877067831851222010-05-25T11:45:00.000+01:002013-09-22T21:31:53.939+01:00Coder vs Security - friend or foe?Certainly 140 characters is not enough to express all the thoughts around recent CSRF flaw in OpenCart and how it was handled (in my humble opinion it even deserves nomination for <a href="http://pwnies.com/nominations/">Pwnie Awards</a>), although some people had a good go at Daniel Kerr.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjk_66FLhlVqjkg5F3nFP2jw6ErSsFImn5mskRj_yxMVmKPinJSZxvV5amxrmMXlkWRJwsnkCOfPIDL67uUu2WmFBAIaVeyXDYrI9MCUvMmNzTIassvvz5MW1kQMCzHbwPe5ieKPineIZc/s1600/ocart-fail.png" /></div>
<br />
Above is just a selection of comments that you can find on Twitter and in all of this negative karma there is some good thing going on. This incident got quite a lot of people to write some really good posts about the incident. Some of my favorite posts are <a href="http://coffeetocode.net/2010/05/humble-helps/">Humble Helps</a> and <a href="http://h20597.www2.hp.com/securitysoftware/blogs/rafal/archive/2010/05/24/psychology-of-quot-secure-code-quot.aspx">Psychology of "Secure Code"</a> - definitely worth reading.<br />
<br />
Although I'm not an expert in either coding or security (but I did quite a lot of both) I think there is also a bit more to it.<br />
<br />
<br />
I used to work with many coders (people that write code) - some<br />
extremely good and some extremely bad. When I look back I would happily<br />
say that I was at some point doing a bit of both - at least in my own<br />
opinion.<br />
<br />
<span style="font-size: 1.25em;"><b>Developers vs code-slingers</b></span><br />
<br />
There<br />
is a significant distinction between those two groups at least as far<br />
as I can tell. Developers do their job and write apps the best way they<br />
can, they are proud of their job because they know they did the best<br />
they could. Code-slingers, well... get it done, whatever... Usability is<br />
something they may or may not understand (if it works, it's usable,<br />
right?), quality and elegance rings the bell somewhere but that's not<br />
in their church so nothing to worry about... and security is often<br />
totally unheard of. Sad, isn't it?<br />
<br />
I think we've all been there<br />
and done more or less of that - it takes time to learn and even more<br />
time to understand. That requires patience and a lot of energy, and<br />
more than anything it requires a person to say "<i>I want to do it right, I want to understand</i>".<br />
<br />
Recently in one of the emails I've found an anonymous quote by a person that was training newly hired staff - he said "<i>I can teach them just about anything, but I can't give them a basic<br />sense of curiosity</i>". I couldn't describe it better!<br />
<br />
<span style="font-size: 1.25em;"><b>Developers vs Infosec</b></span><br />
<br />
In<br />
my opinion the real virtue of a good developer is aiming for perfection<br />
and taking criticism as a chance to improve. It is sometime painful (I<br />
know from my own experience) but we all make mistakes and no matter how<br />
good we are, there will be someone better that will prove we are wrong.<br />
Real developers know how to deal with it because they want their code<br />
to be beautiful in all possible aspects and they are curious people.<br />
Some of those can be real inspiration and you enjoy every second you<br />
spend with them.<br />
<br />
<span style="font-size: 1.25em;"><b>Code-slingers vs Infosec</b></span><br />
<br />
Mostly<br />
not as skilled as developers, often with bad habits, etc - you can say<br />
'developers in training' and that is ok. The first shock of getting<br />
something that actually does the job is hopefully passing by and they<br />
want more - or they don't... they are so happy that their code works<br />
that nothing else matters - that's where the problems come from.<br />
<br />
It's<br />
not a problem of skill, it's a problem of attitude. You can spend a lot<br />
of time with them trying to explain, demonstrate or even send to some<br />
training that deals with secure coding... still they couldn't care less -<br />
oblivious, ignorant, often arrogant and portraying you (the infosec<br />
person) as their biggest enemy because you prevent them from doing<br />
their job. Yes, I've been there and worked with such people.<br />
<br />
<span style="font-size: 1.25em;"><b>Free vs 'for money'</b></span><br />
<br />
Your<br />
options vary depending on environment. In 'for money' space you have<br />
tools to deal with that - you can and should mentor such people to help<br />
them understand. Sometimes a cup of coffee, friendly chat at the<br />
whiteboard going through the requirements and proposed solutions or<br />
ideas can really make huge difference. If you are not so friendly then<br />
get your company to pay for some good training that will give those<br />
people some good base to do their work (get rid of bad habits, don't<br />
post code snippets on forum with URL to the product, etc) so they get<br />
the carrot. <br />
If that doesn't help, go for the stick - at the end of<br />
the day, that code-slinger or his/her supervisor or their supervisor<br />
will have to face a dilemma of signing off a code for production - it's<br />
a business decision. If you can't block it and don't communicate your<br />
security concerns it will be your fault if things go wrong and your<br />
head on the chopping block. Brutal but simple - isn't it?<br />
<br />
In the<br />
free software world, where coder does something for little money (let's<br />
say donations) or no money at all, what is the carrot and what is the<br />
stick? If people care, they get good ratings, maybe more donations,<br />
good publicity and are praised for their work, but if they don't give a<br />
s**t... Oh, hi Daniel!<br />
<br />
Free software users will complain, do a lot<br />
of bad PR and a lot of them will go away migrating to other products,<br />
but hold on... in this particular case THERE IS a commercial support<br />
for OpenCart. What will the paid customers do with such response like<br />
we've seen? How do they feel? They pay for support that they clearly<br />
don't get :-(<br />
<br />
<span style="font-size: 1.25em;"><b>Lessons to take away</b></span><br />
<br />
<span style="font-size: 1em;">If you are so called code-slinger</span><br />
try to understand that writing code that works is not all you have to<br />
do. You should create solutions - not problems. Don't behave like a<br />
little kid, put your pride aside for a few minutes and listen to what<br />
people have to say about your work - it really helps, even if it will<br />
ruin your day.<br />
<br />
If you are developer, please, be a mentor to the<br />
code-slingers so they understand the beauty of the code and what it is<br />
all about. Be a role model - calm, patient, their best friend and<br />
inspiration - that's how miracles happen.<br />
<br />
If you are the infosec<br />
person please remember that saying 'no, you do it wrong' doesn't get<br />
you anywhere. You have to be patient more than ever, explain why you<br />
said 'no' and help find a solution - otherwise you have just created<br />
another problem.<br />
<br />
Whoever you are - remember that people<br />
sometimes get frustrated, they have a bad day, they say things they<br />
later regret - it's a design flaw we all have. Daniel had just<br />
demonstrated it and it got public. Simple 'sorry' can clear the<br />
atmosphere and create a place to work together and solve the problem. <br />
<br />
<br />
At the end of the day, we should all be friends, not foes... so I'll better shut up before I say something I will regret :-)<br />
<br />
<br />
<br />Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-81632384035006328272009-12-15T22:42:00.000+00:002013-09-29T18:16:54.637+01:00AirView2 Spectrum AnalyzerRecently I had some serious problems with wi-fi at home - especially one of the laptops was dropping off and couldn't come back. Quick survey using Kismet and other tools to scan what's flying around has proven that my network is in less populated part of the spectrum (at least here) but still, problems are getting worse and worse.<br />
<br />
I was fully aware of <a href="http://www.metageek.net/">Wi-Spy by Metageek</a>, seen it in action previously but never had a chance to buy one. Part of the decision was the price back then, maybe now it would be another game, but anyway - I got myself another device, made by well known wi-fi vendor <a href="http://ubnt.com/">Ubiquiti</a> and it's called <a href="http://ubnt.com/airview/">AirView2</a>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgs0I4bV7J9pbksD1-zPuQ8edVnmEIzb56a0EMe1p9Y23sIDJKPxQ5AGqD66B3-wzw40Mo1wBttXZjnSQc9gDerQKnBEgbmf-8EXjrTkO_WKu4-YLDZkZPi0ViTnau5VYhYcsmdnu2wFx8/s1600/AirView2ext-osx.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="386" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgs0I4bV7J9pbksD1-zPuQ8edVnmEIzb56a0EMe1p9Y23sIDJKPxQ5AGqD66B3-wzw40Mo1wBttXZjnSQc9gDerQKnBEgbmf-8EXjrTkO_WKu4-YLDZkZPi0ViTnau5VYhYcsmdnu2wFx8/s640/AirView2ext-osx.png" width="640" /></a></div>
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><br /></span>
<b>What's so special about this one? Why it's better than Wi-Spy?</b><br />
<br />
First of all I didn't say it's better. It's different, woks with Linux, Mac OS X and Windows, has a nice price tag and does pretty much the same as Wi-Spy. Let's have a closer look then, shall we?<br />
<br />
<br />
<b>It's different</b><br />
<br />
Well, obviously it is... it comes from different vendor... and this post is not a sales pitch - it's just what I've experienced myself. On a bit more serious note, it's smaller than all the Wi-Spy models I've seen so far. Smaller is good, right? Yes - takes up less space, No - easier to loose (looks almost like USB stick).<br />
<br />
<b>Works with Linux, Mac and Windows</b><br />
<br />
Yes, it does... better or worse but it does and it's not a matter of hardware or bundled software, but clearly it depends on the host OS and Java. You got it right - Java!<br />
Software is written in Java to be really cross platform, but those that are not Java developers but use it a bit know quite well what a pain in the rear Java can be. Same is here - Ubiquiti warns about compatibility issues, there are long posts on the forums why this particular version of AirView software doesn't work (mostly on Mac OS X) and how to fix it, etc.<br />
<br />
<i>Windows</i> - OK, even inside VM with USB passed through to the guest VM (tested VirtualBox, VMWare Workstation and Fusion - all with Windows 7 and latest Java). It was all very slow, loosing connection with the device and re-initializing it all the time, but worked. In native mode with Windows 7 on bare metal box worked like a charm (tested on a netbook PC).<br />
<br />
<i>Linux</i> - didn't try, not enough time - sorry.<br />
<i><br />Mac OS X</i> - yeah... that sucked! If you have the latest patches installed most likely the software will hang on detecting the device. Of course the reason is Java + OS X (I'm on 10.6.2 as of now with Java 1.6.0_17 in 64-bit mode).<br />
<br />
<blockquote>
<i>java.lang.UnsatisfiedLinkError: /Library/Java/Extensions/librxtxSerial.jnilib: no suitable image found. Did find: /Library/Java/Extensions/librxtxSerial.jnilib: no matching architecture in universal wrapper thrown while loading gnu.io.RXTXCommDriver<br /><br />Exception in thread "AirViewer-Initializer" java.lang.UnsatisfiedLinkError: /Library/Java/Extensions/librxtxSerial.jnilib: no suitable image found. Did find: /Library/Java/Extensions/librxtxSerial.jnilib: no matching architecture in universal wrapper</i></blockquote>
WTF?! File not found... but found? Never mind - luckily the solution is very simple - AirView comes with it's own version of <i>librxtxSerial.jnilib</i> so the one that came with OS X needs to be disabled temporarily and problem will go away. That can be done very easily with one command in the terminal:<br />
<br />
<blockquote>
<i>mv /Library/Java/Extensions/librxtxSerial.jnilib{,-disabled}</i></blockquote>
That's it, now it works :-)<br />
<br />
<b>Price tag</b><br />
<br />
This argument is obviously quite important. Is it that much cheaper? I'm not so sure... of course you can get the basic Wi-Spy for about £65+VAT so even if AirView2 would be equal to it in hardware terms, it would cost a bit more - £69+VAT... There is one catch to it though - AirView2 comes in several versions. I bought the AirView2-EXT for £64+VAT and this one has MMCX connector for external antenna (because it doesn't have a built-in one) and with clip-on omnidirectional antenna (~3-5dBi I guess) in the package. To buy Wi-Spy with RP-SMA connector you would have to spend at least £120+VAT which is almost double the price of AirView2-EXT.<br />
<br />
HINT: Wi-Spy with RP-SMA is 2nd generation - faster with better scan resolution than the first generation of the device. So far I didn't have enough time to grab the full spec for AirView2 and compare them side by side - that would be very interesting (it's already on my TODO list).<br />
<br />
<b>Does it do the same stuff?</b><br />
<br />
I would say YES based on what I can see, but as most of those devices are SDRs (Software Defined Radio), they can do all the software allows them to do and I didn't have a chance to compare recent version of Wi-Spy software to the AirView one, so please take my words here with a grain of salt and look for other sources to confirm that.<br />
<br />
<b>Conclusions</b><br />
<br />
The device worked for me like a charm - it turned out that the signal from my AP was attenuated by temporary objects that came in the way (books - whole piles of them) and as the amount of networks around at least doubled in the last 12 months, somebody put up some very messy device that is transmitting all the time with a very wide signal, exactly in the area of channel I was on - so here come the interference!<br />
Quick look at the graphs and it was clear, that simple channel change should cut down on the interference and moving books a bit will improve signal strength in a place where this unlucky laptop is used most of the time - it worked very well, no more problems!<br />
<br />
Looking at the bottom line, for me that's a money very well spent! I was asking myself a question 'how often I will use this thing' and now I really appreciate the power of seeing something that Kismet and similar tools won't see. <br />
<br />
Using spectrum analyzer like AirView or Wi-Spy (doesn't really matter which one - pick one that suits your needs) is like reading between the lines - there is a lot of valuable information out there... if only you can see it!<br />
<br />Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-39307517865263704262009-12-05T09:32:00.000+00:002013-09-29T18:19:38.781+01:00The Hex Factor at SANS London 2009The competition is now officially over and I have to say it was AWESOME!<br />
<br />
Those that made it to <a href="http://www.brucon.org/">BruCON</a> had a chance to play it, those that came to <a href="http://www.sans.org/london09/">SANS London 2009 </a>also had their fun, all the rest of you - bad luck :-/ maybe next time.<br />
<br />
The Hex Factor was run for four evenings/nights at <a href="http://foxbars.com/excel/index.html">The Fox</a> Bar and Restaurant located literally next to the Excel center where SANS courses were hosted. What can be better than beer, hacking and a spirit of competition?!<br />
<br />
Tasks set by the authors were varied in difficulty and topics they covered. One category was about history and culture of hacking with a bit of general teaser tasks and was called <b>Once Upon A Time</b>, like finding a name of candy shop at <street name>, so that was a soft introduction.<br />
<br />
My favorite category was <b>Out Of The Box</b> category (also known as <b>Pure Leetness</b>), where questions were really 'out of the box' and solving them was the best fun I had for a long time! First 100 points for finding a number 'hidden' in the message was really simple and <a href="http://www.youtube.com/watch?v=qkLClG0FBBw">here's how I did it</a>:<br />
<br />
<center>
<object height="480" width="640"><param name="movie" value="http://www.youtube.com/v/qkLClG0FBBw&hl=en_US&fs=1&rel=0&hd=1" /><param name="allowFullScreen" value="true" /><param name="allowscriptaccess" value="always" /><embed src="http://www.youtube.com/v/qkLClG0FBBw&hl=en_US&fs=1&rel=0&hd=1" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" height="480" width="640"><a style="left: 611px ! important; top: 791.167px ! important;" title="Click here to block this object with Adblock Plus" class="ksfsknelulljisbfpftk" href="http://www.youtube.com/v/qkLClG0FBBw&hl=en_US&fs=1&rel=0&hd=1"></a><a class="ksfsknelulljisbfpftk" href="http://www.youtube.com/v/qkLClG0FBBw&hl=en_US&fs=1&rel=0&hd=1"></a><a class="ksfsknelulljisbfpftk" href="http://www.youtube.com/v/qkLClG0FBBw&hl=en_US&fs=1&rel=0&hd=1"></a><a class="ksfsknelulljisbfpftk" href="http://www.youtube.com/v/qkLClG0FBBw&hl=en_US&fs=1&rel=0&hd=1"></a></object></center>
<br />
<br />
I didn't have time to do the one for 200 points, but finally after some time I managed to solve the 300 points one - finding a secret number hidden in the PDF file - hats off to Didier Stevens for this task - it was amazing! <a href="http://blog.didierstevens.com/">Didier's blog</a> was a great guide and help in the process.<br />
<br />
<br />
Third category was <b>Pwned</b> and consisted of physical box with<br />
sensors you had to trigger in the right order to get the code and two<br />
systems to be penetrated. As I said, the difficulty was varied and so<br />
were the nominal point values for each task, from 100 to 300, but you<br />
could also get the partial points if you did only part of the task<br />
properly. Of course during competition like this one you are never alone... Hello brotha!<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVKiCzAQ5I4gvHrr8DV0qmZ_CEnDXECBS3BNNnsbKen5-koXUCovdJE_FXWV5ZbbRxqWN_-b290tbBwyAoc_RVLFPtY1FgPqUEzGqZRmlkZ6sVhQsGo4l5vNoqYJPhNVleP8DrKoyECkE/s1600/hexfactorhello.jpg" /></div>
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><br /></span>
Anyway, it was all very very friendly competition - beer infused with brains hurting after the classes (typical for 'SANS Fire Hose Syndrome'). <br />
<br />
Third category of tasks was <b>Binary fu</b> where you had to work your way through programs delivered as .exe files and get the secret codes out of them. First one was easy, but again I had no time to go through the remaining two. That is the reason why our team (I was working with <a href="http://blog.c22.cc/">Chris Riley</a>, better known as <a href="http://twitter.com/ChrisJohnRiley">@ChrisJohnRiley</a>) was called <i>Drunk and going home</i>. <br />
<br />
At the end we were #4 at <a href="http://www.thehexfactor.org/home/2009_sanslondon">the leader board</a> but as it turned out, two teams of the first three were the same people, so kind of we are #3, so here we are - two of three winning teams, already in The Hex Factor t-shirts!<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQNmOJeiX0ggACFb7GDSvb0R-qNHXGoMxxpxybGh1SWUGDqWL05CTsiWlISAsc-IDPtAhhrVOm0zrkrkcTR062zTXeBoBKOjLrYTmGSQNF66SqIs72XCHdO99uYqYPNZORABge8RTlLzM/s1600/hexfactorwinners.jpg" /></div>
<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><br /></span>
I'd like to say <b>THANK YOU</b> to all the people behind The Hex Factor - it was really awesome experience and great fun, so I hope it's not the last time we see The Hex Factor. See you next time!<br />
<br />Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0tag:blogger.com,1999:blog-1630534171443998118.post-24390955329672208092009-11-28T09:45:00.000+00:002013-08-20T22:05:18.100+01:00CONFidence09.02 - post mortemWell... my plan to blog live from the CONFidence was good but still remained to be more of a plan than a reality. Twitting went much better (possibly because you can twitt between chats with people, drinks, etc) so I'll wrap up what happened and how it went.<br /><br />The conference was great - I really liked the lectures (those I actually made to), loved the chat with speakers and it was awesome to meet some old friends and make some new contacts. Overall, if you didn't come to Warsaw for CONFidence09.02 you missed quite a lot.<br /><br /><b>Day 1 summary</b><br />There was very nice presentation by Felix "FX" Lindner on how 'awesome' Cisco IOS is, Claudio Criscone (<a href="http://twitter.com/paradoxengine">@paradoxengine</a>) talked about security in virtualization environments, Frank Breedijk renamed hist AutoNessus to <a href="http://seccubus.org/">Seccubus</a> (new twitter feed at <a href="http://twitter.com/seccubus">@seccubus</a>), Leonardo NVE Egea showed us how you can use the satellites to work as your downlink (and it seemed much easier than actually you would think), Pavol Luptak pretty much owned the RFID there (yes, the basic cloning kit is just €30), Elisa dropped the pressure a bit with Power Point Karaoke where Felix "FX" Lindner was presenting about detecting unknown alcohols, Raoul Chiesa gave great presentation about knitting (yes, knitting) and I was rolled into a presentation about IT slang/acronyms and there was something about insulting someone :-) and that was just the first day.<br /><br /><b>Day 2 summary</b><br />For those that survived the 'afterparty' on the evening/night/morning you had a chance to see nice explanation of the cold boot attack given by Nadia Heninger, Nick DePetrillo discussed 'what could go wrong' with intelligent power grids and believe me... there's a lot! Jacob Applebaum (<a href="http://twitter.com/ioerror">@ioerror</a>) gave us some TOR love and a lot of TOR laptop stickers. Alessio "mayhem" Penasilico (<a href="http://twitter.com/mayhemspp">@mayhemspp</a>) and Raoul Chiesa gave nice presentation on history of hacking telcos - there was some good info there... just before Raoul killed it all with final presentation dissecting the underground economy (with some slides show just after the cameras and other recording equipment was turned off). That was a really good one...<br /><br />Finishing off, Frank has posted a bunch of posts about presentations we saw in Warsaw. They are:<br /><ul><li><a href="http://www.cupfighter.net/index.php/2009/11/confidence0902-threat-feeds/">Fusing 3rd party threat feeds to obtain better threat intelligence - Eddie Schwartz</a></li><li><a href="http://www.cupfighter.net/index.php/2009/11/confidence0902-router-exploitation/">Router Exploitation - Felix "FX" Lindner</a></li><li><a href="http://www.cupfighter.net/index.php/2009/11/confidence-seccubus-slides/">My Seccubus slide deck</a> (slides from Frank's presentation)</li><li><a href="http://www.cupfighter.net/index.php/2009/11/confidence-tls-renegotiation/">My TLS renegotiation vulnerability slides</a> (Frank's lightning talk during one of the breaks)</li><li><a href="http://www.cupfighter.net/index.php/2009/11/confidence-mifare/">Mifare Classic anaysis - Pavol Luptak</a></li><li><a href="http://www.cupfighter.net/index.php/2009/11/confidence-power-hungy-people-%e2%80%93-nick-depetrillo/">Power Hungy People - Nick DePetrillo</a></li><li><a href="http://www.cupfighter.net/index.php/2009/11/confidence-tor/">The Tor Project - Jacob Appelbaum</a></li><li><a href="http://www.cupfighter.net/index.php/2009/11/confidence-cybercrime/">Underground economy - Raoul Chiesa</a><br /></li></ul><br />That's it for now - just make sure you get there next time :P<br /><br/><br/>Tomasz Miklashttp://www.blogger.com/profile/00221642679288385721noreply@blogger.com0