Wednesday, October 29, 2014

The Little Unattended Monster - Openstack deployment


First compute node is kind of special. Since The Little Unattended Monster has limited resources
one of compute nodes is also used to host two special virtual machines created before Openstack deployment - Controller node and Network node.


Interface p2p1 is used for management
Interface p4p2 is used for
  • external network on VLAN 10
  • internal network-GRE on VLAN 20
  • internal network-VLAN on VLANs 100-200
Interface p4p1 is used for cinder volumes/iSCSI on VLAN 30

Compute 2 & 3

All compute nodes have Nova-Compute and Cinder-Volume services

The Little Unattended Monster - Physical wiring diagram

Saturday, October 11, 2014

The Little Unattended Monster

3 motherboards, 9 network interfaces, 1x8 ports & 1x5 ports gigabit switches + power meter. Installs itself from 3 USB sticks and becomes self-aware :)

Wednesday, September 24, 2014

Unattended Ubuntu installer USB drive, headless server

Goal: to create a USB stick that can be used to automatically install Ubuntu server on a headless machine. Installation should configure machine for most basic passwordless root SSH access. 

The idea is to be able to remotely return a physical machine into "original" state. Set up boot order preference to 
  1. HDD
  2. UEFI USB  
Permanently attach a USB stick to the machine, and when you need to return it to original state - remove partitions on the HDD so it falls back to USB boot.

*** For motherboards that support  UEFI boot
*** Initially I thought of simulating this on VirtualBox VM, but it did not work. Though works well on physical machine.

Step 1: Copy all files from Ubuntu ISO to USB

Somehow on Mac I was not able to mount Ubuntu ISO, it gives an error "no mountable file system", so I did it on Ubuntu VM, tar/gzip, scp'ed to Mac and untarred onto USB.

NOTE: it is important to untar directly to USB, if you untar to a folder and then copy to USB - OSX changes something in files, and then Ubuntu thinks that installation media is corrupted.

# download iso 

# mount it
mkdir /media/iso
mount -o loop ubuntu-14.04.1-server-amd64.iso /media/iso

mkdir -p /tmp/iso
cp -rT /media/iso /tmp/iso
cd /tmp/iso/

Step 2: Now prepare it for automatic install with Kickstart

Add ks.preseed,  ks.cfg, and edit boot/grub/grub.cfg files. All path are relative to ISO's root folder. 
Also not needed for USB/UEFI, but needed to boot a VM from ISO: change isolinux/lang, isolinux/isolinux.cfg and isolinux/txt.cfg
  - ks.cfg
  - ks.preseed
  - boot/
    - grub/
      - grub.cfg
  - isolinux/
    - lang
    - isolinux.cfg
    - txt.cfg

Add ks.preseed

1) setting of user-setup/allow-password-weak to true,  if you do not want installation to be stopped because you set "weak" password
2) setting cdrom-detect/eject to false to prevent USB drive from being ejected after installation. Otherwise you will have to manually power cycle machine or re-insert USB drive, to be able to re-install OS
d-i partman-auto/method string lvm
d-i partman-auto-lvm/guided_size        string max
d-i partman-auto/choose_recipe          select atomic
d-i partman-auto/disk                   string /dev/sda
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/confirm_write_new_label     boolean true
d-i partman/choose_partition            select  finish
d-i partman/confirm_nooverwrite         boolean true
d-i partman/confirm                     boolean true
d-i partman-auto/purge_lvm_from_device  boolean true
d-i partman-lvm/device_remove_lvm       boolean true
d-i partman-lvm/confirm                 boolean true
d-i partman-lvm/confirm_nooverwrite     boolean true
d-i partman-auto/init_automatically_partition select Guided - use entire disk and set up LVM
d-i partman/choose_partition            select Finish partitioning and write changes to disk
d-i partman-auto-lvm/no_boot            boolean true
d-i partman-md/device_remove_md         boolean true
d-i partman-md/confirm                  boolean true
d-i partman-md/confirm_nooverwrite      boolean true
d-i user-setup/allow-password-weak      boolean true
d-i cdrom-detect/eject                  boolean false

Add ks.cfg

I end up setting root password :( 
There was a problem - if you set it to add an initial user, somehow it does not exist when %post section executed, so you cannot "chown" files to that user. That's necessary if you setting authorized_keys. Alternatively you can set root password to something complex/encrypted and "forget" it. Then for reason of simplicity I set ssh key for "root" user. 
Anyway the goal of this automatic installer was to provide bare minimum passwordless root SSH access to the server, to do further configuration remotely.

Network device p2p1 - this is how it called on my motherboard instead of eth0. How to find it? 
You need to boot it, and run "ifconfig --all" and look what network devices you have. Also you can put --device=link, but it did not work on a physical machine for me, only on VM
lang en_US
langsupport en_US
keyboard us
timezone America/Los_Angeles
rootpw ubuntu
user --disabled
# rootpw --disabled
# user ubuntu --fullname "ubuntu" --password "ubuntu"
auth  --useshadow  --enablemd5
network --bootproto=dhcp --device=p2p1
firewall --disabled

@ ubuntu-server

mkdir -p /root/.ssh
chmod 700 /root/.ssh
cat >> /root/.ssh/authorized_keys << EOF
ssh-rsa ...

chmod 600 /root/.ssh/authorized_keys

# to make sure it boots when power is turned on
echo "GRUB_RECORDFAIL_TIMEOUT=0" >> /etc/default/grub
Edit boot/grub/grub.cfg
Add highlighted parts to set it automatically execute first option with Kickstart configuration files

Important: do not set timeout to 0. In this case it will wait forever until you press a key. You do not want it on headless system.
if loadfont /boot/grub/font.pf2 ; then
        set gfxmode=auto
        insmod efi_gop
        insmod efi_uga
        insmod gfxterm
        terminal_output gfxterm

set menu_color_normal=white/black
set menu_color_highlight=black/light-gray
set timeout=5
set default=0

menuentry "Install Ubuntu Server" {
        set gfxpayload=keep
        linux   /install/vmlinuz  file=/cdrom/preseed/ubuntu-server.seed quiet ks=cdrom:/ks.cfg preseed/file=/cdrom/ks.preseed --
        initrd  /install/initrd.gz
menuentry "OEM install (for manufacturers)" {
        set gfxpayload=keep
        linux   /install/vmlinuz  file=/cdrom/preseed/ubuntu-server.seed quiet oem-config/enable=true --
        initrd  /install/initrd.gz
menuentry "Multiple server install with MAAS" {
        set gfxpayload=keep
        linux   /install/vmlinuz  modules=maas-enlist-udeb vga=788 initrd=/install/initrd.gz quiet --
        initrd  /install/initrd.gz
menuentry "Check disc for defects" {
        set gfxpayload=keep
        linux   /install/vmlinuz  MENU=/bin/cdrom-checker-menu quiet --
        initrd  /install/initrd.gz
menuentry "Rescue a broken system" {
        set gfxpayload=keep
        linux   /install/vmlinuz  rescue/enable=true --
        initrd  /install/initrd.gz

Add isolinux/lang

echo en >isolinux/lang

Edit isolinux/isolinux.cfg

Change timeout to 10, if it set to 0 it will wait forever 
# D-I config version 2.0
include menu.cfg
default vesamenu.c32
prompt 0
timeout 10
ui gfxboot bootlogo

Edit isolinux/txt.cfg

Add highlighted 
default install
label install
  menu label ^Install Ubuntu Server
  kernel /install/vmlinuz
  append  file=/cdrom/preseed/ubuntu-server.seed vga=788 initrd=/install/initrd.gz quiet ks=cdrom:/ks.cfg preseed/file=/cdrom/ks.preseed --
label cloud
  menu label ^Multiple server install with MAAS
  kernel /install/vmlinuz
  append   modules=maas-enlist-udeb vga=788 initrd=/install/initrd.gz quiet --
label check
  menu label ^Check disc for defects
  kernel /install/vmlinuz
  append   MENU=/bin/cdrom-checker-menu vga=788 initrd=/install/initrd.gz quiet --
label memtest
  menu label Test ^memory
  kernel /install/mt86plus
label hd
  menu label ^Boot from first hard disk
  localboot 0x80

Create ISO

Make sure you are in the /tmp/iso folder
# install genisoimage
apt-get install genisoimage

mkisofs -r -V "UBUNTU1404" -cache-inodes -J -l -b isolinux/isolinux.bin -c isolinux/ -no-emul-boot -boot-load-size 4 -boot-info-table -o ../ubuntu-14.04.1-server-amd64-autoinstall.iso .

Make a USB from this new ISO

# unmount original Ubuntu ISO
umount /media/iso

# mount newly created ISO
mount -o loop /tmp/ubuntu-14.04.1-server-amd64-autoinstall.iso /media/iso

# tar/gzip it 
cd /media/iso/
tar cvzf /tmp/ubuntu-14.04.1-server-amd64-autoinstall.tar.gz .

# mount a USB drive
mkdir -p /media/usb
mount /dev/sdc1 /media/usb

# and untar to USB
cd /media/usb/
tar xvf /tmp/ubuntu-14.04.1-server-amd64-autoinstall.tar.gz 

Note, that I am doing tar/untar but not copy. That's the easiest way to preserve all files exactly as-is. With "cp" command you need to specify some options, but I was not able to find a good working combination (too lazy)

Return system to original state - remove all HDD partitions

Initially I tried to do it with a simple "dd" command, erasing MBR like below:
dd if=/dev/zero of=/dev/sda bs=512 count=1
But it corrupts HDD the way so it still tries to boot from it, and then stuck with some error. I figured out that if I do it more clean way, by removing all partitions with parted - it correctly fails over to next boot device - USB drive after reboot:
root@ubuntu:~# parted /dev/sda p rm 1 y i rm 2 y i rm 3 i p
Model: ATA KINGSTON SV300S3 (scsi)
Disk /dev/sda: 120GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End    Size   File system  Name                  Flags
 1      1049kB  538MB  537MB  fat32        EFI System Partition  boot
 2      538MB   794MB  256MB  ext2
 3      794MB   120GB  119GB                                     lvm

Warning: Partition /dev/sda1 is being used. Are you sure you want to continue?
Error: Partition(s) 1 on /dev/sda have been written, but we have been unable to inform the
kernel of the change, probably because it/they are in use.  As a result, the old partition(s)
will remain in use.  You should reboot now before making further changes.
Warning: Partition /dev/sda2 is being used. Are you sure you want to continue?
Error: Partition(s) 1, 2 on /dev/sda have been written, but we have been unable to inform the
kernel of the change, probably because it/they are in use.  As a result, the old partition(s)
will remain in use.  You should reboot now before making further changes.
Error: Partition(s) 1, 2, 3 on /dev/sda have been written, but we have been unable to inform
the kernel of the change, probably because it/they are in use.  As a result, the old
partition(s) will remain in use.  You should reboot now before making further changes.
Model: ATA KINGSTON SV300S3 (scsi)
Disk /dev/sda: 120GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End  Size  File system  Name  Flags

Information: You may need to update /etc/fstab.                           

root@ubuntu:~# reboot

Broadcast message from root@ubuntu
 (/dev/pts/0) at 8:27 ...

The system is going down for reboot NOW!

What's happen if you accidentally boot from this USB?

Will it wipe a system? No - installer will stop when it will discover that disk is partitioned and ask to confirm whether you really want to erase everything.

Wednesday, February 5, 2014

Do it yourself security camera system

How you can build a security camera system that has a live video preview, motion detection, email notifications and uploads snapshots to the cloud?

Though you easily can go and purchase one for something under $100, if you know a bit of Linux you can build one yourself. In fact, it will be cheaper and will give you power of customization - so you can make it better than one out-of-the-box.

Pogoplug E02         $15 
USB Wifi adaptor     $10
USB Flash drive 8G   $10
USB HD Webcam***     $20
Total                $55

*** Here you can make it more expensive, or less: you can buy non-HD camera for $5 or have something with superior quality starting at $40. Also you can connect two cameras to the system directly, or even more if you add a powered USB hub.


REALLY helpful tutorial on Pogoplug, I used a lot of information from it for this blog post: Qui's techNOLOGY Blog. The blog post is based on Arch Linux ARM project Pogoplug v2 (Pink/Gray)

What is Pogoplug

ARM based computer, you can install Arch Linux on it.

CPU: 1.2GHz
RAM: 256Mb
4 USB ports
1 Ethernet port. 

And... much cheaper than Raspberry Pi, you can buy one for just $15 dollars with free shipping (link above)

Let's start

Better start at small home network, which you can easily scan with tools like nmap. Because when you connect Pogoplug to the network there is no way to find its address other than just scan network for it.

So you can network before and after you connect Pogoplug, and check for difference. For example if your network is I have this as default using my Comcast router
nmap -sn

Once you find a new IP, you pretty much need to follow tutorial at Qui's blog.
Let's say IP is, ssh as root, default password is "ceadmin"

ssh root@
password: ceadmin

# Stop the Pogoplug software
killall hbwd

# Install UBoot - E02 only
cd /tmp
chmod +x

# Would you like to disable the Pogoplug services? 
# answer 'Y'

Now plug in USB drive to the slot next to Ethernet, it should appear as /dev/sda
You need at least 4 GB, but it will only be enough to play with it. Better to have 16 GB.

Partitioning and formatting flash drive

I suggest to create two partitions, root and swap. Qui's blog does not say so. But since the unit has just 256Mb RAM, I thought it would be good to have 1GB swap partition.

/sbin/fdisk /dev/sda

Check manual how to use it to create partitions.
Here is the result. 
Disk /dev/sda: 16.0 GB, 16008609792 bytes
255 heads, 63 sectors/track, 1946 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sda1               1        1820    14619118+ 83 Linux
/dev/sda2            1821        1946     1012095  82 Linux swap

Note file file system types 83 and 82. 
also root partition should be first /dev/sda1, otherwise device will not boot.

If you want to partition in non-interactive mode, you can do something like this:
echo -e "o\nn\np\n1\n\n+15G\nn\np\n2\n\n\nt\n2\n82\nw\np\n" | fdisk /dev/sda

Unfortunately there is no better tool available for Pogoplug OS

Now formatting the drive, you will need to download mke2fs to do this
cd /tmp
chmod 755 mke2fs
./mke2fs -L ROOTFS -j /dev/sda1
mkdir -p /tmp/usb
mount /dev/sda1 /tmp/usb

Installing Arch Linux

Basically all what this does - you download a pre-baked "image" onto USB drive. Then you boot from it and upgrade to latest version:
cd /tmp/usb
tar -xzvf ArchLinuxARM-armv5te-*.tar.gz
rm ArchLinuxARM-armv5te-*.tar.gz
cd ..
umount /tmp/usb
busybox reboot

After reboot, generally the unit should come up on the same IP address. Otherwise you will need to scan your network again using nmap like at the beginning.

Login as root, default password is "root"

ssh root@
password: root

If you were able to do this, congratulations! I was also so glad when I first did this and it worked!
Now you need to upgrade to the latest version.

pacman -Sy linux-kirkwood linux-kirkwood-headers ntp yaourt base-devel nano wget

Accept defaults, but when it asks "Remove linux?" select "y"
After it complete proceed with most basic necessary steps

Full log

Enable swap

mkswap /dev/sda2
swapon /dev/sda2
echo "/dev/sda2 none swap defaults 0 0" >> /etc/fstab 

set timezone, use "timedatectl list-timezones" to get available values

timedatectl set-timezone America/Los_Angeles

Pogoplug has no hardware clock, to every time you reboot the time will be set to some date in 1969. Enable ntpd to fix time. By default it is not enabled.

systemctl enable ntpd
systemctl start ntpd

Couple more tricks is needed:

Check /usr/lib/systemd/system/ntpd.service. Edit it to have -qg in the ntpd startup line. Refer to this post I found about it.

ExecStart=/usr/bin/ntpd -qg -u ntp:ntp -p /run/

Also as per Qui's blog you can set it to sync time on network start:

echo "ExecUpPost='/usr/bin/ntpdate -u'" >> /etc/netctl/eth0

Next steps are as as per Qui's blog, check it for more details and variants. Below are commands that I executed on my Pogoplug E02

#Add ROOTFS to fstab so fsck will run when needed
echo "LABEL=ROOTFS / ext3 rw,noatime 0 1" >> /etc/fstab

pacman -Sy uboot-env
echo "/dev/mtd0 0xc0000 0x20000 0x20000">>/etc/fw_env.config

#Have uboot mount partition as ext3 
#safer in power outages
#E02 only, V4 defaults to ext3 already
fw_setenv usb_rootfstype ext3
#Correct machid - make LED Green versus Orange
#Only for E02
fw_setenv machid dd6


Now basic installation is complete. From this step you have a unit that you can shape to your need. I decided to make it a security system with two cameras

Making it a security camera system

This is how is looks now
- front side with Wifi adaptor:

 - back side with USB flash drive

Wifi setup

(here I get into some troubleshooting)

I remember that I purchased some cheap wifi USB adaptor at Fry's several years back, and almost did not use it. I was lucky I did not lose this small thingy, that's so easy if it's not plugged in somewhere.

Hoping for the best I plugged it in, followed "Wireless Configuration" section of Qui's blog post 

#install netctl and related packages
pacman -Sy wireless_tools netctl ifplugd wpa_actiond dialog

#Connect to wireless network and create wireless profile
cd /etc/netctl

and.... It did not work right away. Apparently system did not recognize it as wifi adaptor :(

But it does see it as a USB device, this is better:

# lsusb
Bus 001 Device 005: ID 0411:014f BUFFALO INC. (formerly MelCo., Inc.) 
Bus 001 Device 003: ID 13fe:3623 Kingston Technology Company Inc. 
Bus 001 Device 002: ID 05e3:0608 Genesys Logic, Inc. USB-2.0 4-Port HUB
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Googling of 0411:014f, Buffalo WLI-UC-GN revealed a potential chipset it uses - Ralink 2870, and Arch Linux has a driver for it - rt2800usb. But the driver just does not know about device ID: 0411:014f.

Here is how it can be fixed:

# load needed driver
modprobe rt2800usb 

# add ID of wifi adaptor 0411:014f from lsusb command output
echo 0411 014f > /sys/bus/usb/drivers/rt2800usb/new_id

# check how it worked

# start wifi
systemctl start netctl-auto@wlan0.service

I guess this technique can be used to make almost any adaptor to work. Most hard thing here will be find correct chipset and driver name

Wireless configuration, it can be generated using "wifi-menu" wizard like in Qui's blog, but you also can just create a config file, like this way:

cat > /etc/netctl/wlan0-your_network_name << EOF
Description='Wifi profile for your_network_name'

Also patch for wifi driver does not persist reboot. So you will need to execute it at boot time. Here is what I end up with - created a service:

cat > /usr/lib/systemd/system/buffalo-wifi.service << EOF


And script for fixing driver, used by service. You also can put all these commands directly into service file, but I did not spend much time fixing syntax for it:

cat > /usr/bin/buffalo-wifi << EOF
/usr/bin/modprobe rt2800usb 
/usr/bin/echo 0411 014f > /sys/bus/usb/drivers/rt2800usb/new_id
/usr/bin/systemctl start netctl-auto@wlan0.service

Set correct permissions and enable service to start at boot time:

chmod +x /usr/bin/buffalo-wifi
systemctl start buffalo-wifi
systemctl enable buffalo-wifi

Install httpd

pacman -S apache
systemctl enable httpd
systemctl start httpd

Some additional configuration in next section

Motion WebCam

pacman -S motion x264 fswebcam

Configuration is in /etc/motion/motion.conf, here is what I changed from defaults:

# Restrict webcam connections to localhost only (default: on)
webcam_localhost off

# Image width (pixels). Valid range: Camera dependent, default: 352
width 1280

# Image height (pixels). Valid range: Camera dependent, default: 288
height 720

# TCP/IP port for the http server to listen on (default: 0 = disabled)
control_port 0

# The mini-http server listens to this port for requests (default: 0 = disabled)
webcam_port 8090

# set palette to MJPEG, turn on compression to use less bandwidth of USB hub
v4l2_palette 2

# disable storing video fragments (does not work properly from my point of view)
# Use ffmpeg to encode mpeg movies in realtime (default: off)
ffmpeg_cap_new off

# Target base directory for pictures and films
# Recommended to use absolute path. (Default: current working directory)
target_dir /srv/http/motion/camera

Create folders:

mkdir -p /var/run/motion
mkdir -p /srv/http/motion/camera

Auto start Motion:

cat > /etc/systemd/system/motion.service << EOF
Description=motion camera

ExecStartPre=/usr/bin/fswebcam -r 320x240 -d /dev/video0 -v /dev/null
ExecStart=/usr/bin/motion -c /etc/motion/motion.conf


systemctl enable motion

Try starting it manually to see that there are no errors:

fswebcam -r 320x240 -d /dev/video0 -v /dev/null
motion -n -c /etc/motion/motion.conf 

Start service:

systemctl start motion

httpd configuration to proxy camera stream at 8090 as /motion/video0/ on port 80

cat >> /etc/httpd/conf/httpd.conf << EOF 
ProxyPass /motion/video0/ http://localhost:8090/

Add some password protection to folder "/motion", for example basic authentication. Look apache docs for more details 

<Directory "/srv/http/motion">
    AuthType Basic
    AuthName "Motion"
    AuthBasicProvider file
    AuthUserFile /etc/httpd/conf/passwd
    Require user motion

Add user "motion" with a password:

htpasswd -c /etc/httpd/conf/passwd motion
New password: 
Re-type new password: 
Adding password for user motion

Create a basic HTML page, that shows video and links to folder with pictures

cat > /srv/http/motion/index.html << EOF
<a href="camera/?C=M;O=D"><img src="video0/"/></a>

Restart httpd to pick our configuration changes

systemctl restart httpd

Check how it works:

http://<Pogoplug IP>/motion/