Unattended Ubuntu installer USB drive, headless server

Goal: to create a USB stick that can be used to automatically install Ubuntu server on a headless machine. Installation should configure machine for most basic passwordless root SSH access. 

The idea is to be able to remotely return a physical machine into "original" state. Set up boot order preference to 

1. HDD
2. UEFI USB  

Permanently attach a USB stick to the machine, and when you need to return it to original state - remove partitions on the HDD so it falls back to USB boot.

*** For motherboards that support  UEFI boot

*** Initially I thought of simulating this on VirtualBox VM, but it did not work. Though works well on physical machine.

Step 1: Copy all files from Ubuntu ISO to USB

Somehow on Mac I was not able to mount Ubuntu ISO, it gives an error "no mountable file system", so I did it on Ubuntu VM, tar/gzip, scp'ed to Mac and untarred onto USB.

NOTE: it is important to untar directly to USB, if you untar to a folder and then copy to USB - OSX changes something in files, and then Ubuntu thinks that installation media is corrupted.

# download iso 
wget http://releases.ubuntu.com/trusty/ubuntu-14.04.1-server-amd64.iso

# mount it
mkdir /media/iso
mount -o loop ubuntu-14.04.1-server-amd64.iso /media/iso

mkdir -p /tmp/iso
cp -rT /media/iso /tmp/iso
cd /tmp/iso/

Step 2: Now prepare it for automatic install with Kickstart

Add ks.preseed,  ks.cfg, and edit boot/grub/grub.cfg files. All path are relative to ISO's root folder. 

Also not needed for USB/UEFI, but needed to boot a VM from ISO: change isolinux/lang, isolinux/isolinux.cfg and isolinux/txt.cfg

/tmp/iso/ 
  |
  - ks.cfg
  - ks.preseed
  - boot/
    |
    - grub/
      |
      - grub.cfg
  - isolinux/
    - lang
    - isolinux.cfg
    - txt.cfg

Add ks.preseed

Notes
1) setting of user-setup/allow-password-weak to true,  if you do not want installation to be stopped because you set "weak" password
2) setting cdrom-detect/eject to false to prevent USB drive from being ejected after installation. Otherwise you will have to manually power cycle machine or re-insert USB drive, to be able to re-install OS

d-i partman-auto/method string lvm
d-i partman-auto-lvm/guided_size        string max
d-i partman-auto/choose_recipe          select atomic
d-i partman-auto/disk                   string /dev/sda
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/confirm_write_new_label     boolean true
d-i partman/choose_partition            select  finish
d-i partman/confirm_nooverwrite         boolean true
d-i partman/confirm                     boolean true
d-i partman-auto/purge_lvm_from_device  boolean true
d-i partman-lvm/device_remove_lvm       boolean true
d-i partman-lvm/confirm                 boolean true
d-i partman-lvm/confirm_nooverwrite     boolean true
d-i partman-auto/init_automatically_partition select Guided - use entire disk and set up LVM
d-i partman/choose_partition            select Finish partitioning and write changes to disk
d-i partman-auto-lvm/no_boot            boolean true
d-i partman-md/device_remove_md         boolean true
d-i partman-md/confirm                  boolean true
d-i partman-md/confirm_nooverwrite      boolean true
d-i user-setup/allow-password-weak      boolean true
d-i cdrom-detect/eject                  boolean false

Add ks.cfg

I end up setting root password :( 

There was a problem - if you set it to add an initial user, somehow it does not exist when %post section executed, so you cannot "chown" files to that user. That's necessary if you setting authorized_keys. Alternatively you can set root password to something complex/encrypted and "forget" it. Then for reason of simplicity I set ssh key for "root" user. 
Anyway the goal of this automatic installer was to provide bare minimum passwordless root SSH access to the server, to do further configuration remotely.

Network device p2p1 - this is how it called on my motherboard instead of eth0. How to find it? 
You need to boot it, and run "ifconfig --all" and look what network devices you have. Also you can put --device=link, but it did not work on a physical machine for me, only on VM

lang en_US
langsupport en_US
keyboard us
mouse
timezone America/Los_Angeles
rootpw ubuntu
user --disabled
# rootpw --disabled
# user ubuntu --fullname "ubuntu" --password "ubuntu"
reboot
text
install
cdrom
auth  --useshadow  --enablemd5
network --bootproto=dhcp --device=p2p1
firewall --disabled
skipx

%packages
@ ubuntu-server
openssh-server

%post
mkdir -p /root/.ssh
chmod 700 /root/.ssh
cat >> /root/.ssh/authorized_keys << EOF
ssh-rsa ...
EOF

chmod 600 /root/.ssh/authorized_keys

# to make sure it boots when power is turned on
echo "GRUB_RECORDFAIL_TIMEOUT=0" >> /etc/default/grub
update-grub

Edit boot/grub/grub.cfg

Add highlighted parts to set it automatically execute first option with Kickstart configuration files

Important: do not set timeout to 0. In this case it will wait forever until you press a key. You do not want it on headless system.

if loadfont /boot/grub/font.pf2 ; then
        set gfxmode=auto
        insmod efi_gop
        insmod efi_uga
        insmod gfxterm
        terminal_output gfxterm
fi

set menu_color_normal=white/black
set menu_color_highlight=black/light-gray
set timeout=5
set default=0

menuentry "Install Ubuntu Server" {
        set gfxpayload=keep
        linux   /install/vmlinuz  file=/cdrom/preseed/ubuntu-server.seed quiet ks=cdrom:/ks.cfg preseed/file=/cdrom/ks.preseed --
        initrd  /install/initrd.gz
}
menuentry "OEM install (for manufacturers)" {
        set gfxpayload=keep
        linux   /install/vmlinuz  file=/cdrom/preseed/ubuntu-server.seed quiet oem-config/enable=true --
        initrd  /install/initrd.gz
}
menuentry "Multiple server install with MAAS" {
        set gfxpayload=keep
        linux   /install/vmlinuz  modules=maas-enlist-udeb vga=788 initrd=/install/initrd.gz quiet --
        initrd  /install/initrd.gz
}
menuentry "Check disc for defects" {
        set gfxpayload=keep
        linux   /install/vmlinuz  MENU=/bin/cdrom-checker-menu quiet --
        initrd  /install/initrd.gz
}
menuentry "Rescue a broken system" {
        set gfxpayload=keep
        linux   /install/vmlinuz  rescue/enable=true --
        initrd  /install/initrd.gz
}

Add isolinux/lang

echo en >isolinux/lang

Edit isolinux/isolinux.cfg

Change timeout to 10, if it set to 0 it will wait forever 

# D-I config version 2.0
include menu.cfg
default vesamenu.c32
prompt 0
timeout 10
ui gfxboot bootlogo

Edit isolinux/txt.cfg

Add highlighted 

default install
label install
  menu label ^Install Ubuntu Server
  kernel /install/vmlinuz
  append  file=/cdrom/preseed/ubuntu-server.seed vga=788 initrd=/install/initrd.gz quiet ks=cdrom:/ks.cfg preseed/file=/cdrom/ks.preseed --
label cloud
  menu label ^Multiple server install with MAAS
  kernel /install/vmlinuz
  append   modules=maas-enlist-udeb vga=788 initrd=/install/initrd.gz quiet --
label check
  menu label ^Check disc for defects
  kernel /install/vmlinuz
  append   MENU=/bin/cdrom-checker-menu vga=788 initrd=/install/initrd.gz quiet --
label memtest
  menu label Test ^memory
  kernel /install/mt86plus
label hd
  menu label ^Boot from first hard disk
  localboot 0x80

Create ISO

Make sure you are in the /tmp/iso folder

# install genisoimage
apt-get install genisoimage

mkisofs -r -V "UBUNTU1404" -cache-inodes -J -l -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -o ../ubuntu-14.04.1-server-amd64-autoinstall.iso .

Make a USB from this new ISO

# unmount original Ubuntu ISO
umount /media/iso

# mount newly created ISO
mount -o loop /tmp/ubuntu-14.04.1-server-amd64-autoinstall.iso /media/iso

# tar/gzip it 
cd /media/iso/
tar cvzf /tmp/ubuntu-14.04.1-server-amd64-autoinstall.tar.gz .

# mount a USB drive
mkdir -p /media/usb
mount /dev/sdc1 /media/usb

# and untar to USB
cd /media/usb/
tar xvf /tmp/ubuntu-14.04.1-server-amd64-autoinstall.tar.gz 

Note, that I am doing tar/untar but not copy. That's the easiest way to preserve all files exactly as-is. With "cp" command you need to specify some options, but I was not able to find a good working combination (too lazy)

Return system to original state - remove all HDD partitions

Initially I tried to do it with a simple "dd" command, erasing MBR like below:

dd if=/dev/zero of=/dev/sda bs=512 count=1
reboot

But it corrupts HDD the way so it still tries to boot from it, and then stuck with some error. I figured out that if I do it more clean way, by removing all partitions with parted - it correctly fails over to next boot device - USB drive after reboot:

root@ubuntu:~# parted /dev/sda p rm 1 y i rm 2 y i rm 3 i p
Model: ATA KINGSTON SV300S3 (scsi)
Disk /dev/sda: 120GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End    Size   File system  Name                  Flags
 1      1049kB  538MB  537MB  fat32        EFI System Partition  boot
 2      538MB   794MB  256MB  ext2
 3      794MB   120GB  119GB                                     lvm

Warning: Partition /dev/sda1 is being used. Are you sure you want to continue?
Error: Partition(s) 1 on /dev/sda have been written, but we have been unable to inform the
kernel of the change, probably because it/they are in use.  As a result, the old partition(s)
will remain in use.  You should reboot now before making further changes.
Warning: Partition /dev/sda2 is being used. Are you sure you want to continue?
Error: Partition(s) 1, 2 on /dev/sda have been written, but we have been unable to inform the
kernel of the change, probably because it/they are in use.  As a result, the old partition(s)
will remain in use.  You should reboot now before making further changes.
Error: Partition(s) 1, 2, 3 on /dev/sda have been written, but we have been unable to inform
the kernel of the change, probably because it/they are in use.  As a result, the old
partition(s) will remain in use.  You should reboot now before making further changes.
Model: ATA KINGSTON SV300S3 (scsi)
Disk /dev/sda: 120GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End  Size  File system  Name  Flags

Information: You may need to update /etc/fstab.                           

root@ubuntu:~# reboot

Broadcast message from root@ubuntu
 (/dev/pts/0) at 8:27 ...

The system is going down for reboot NOW!

What's happen if you accidentally boot from this USB?

Will it wipe a system? No - installer will stop when it will discover that disk is partitioned and ask to confirm whether you really want to erase everything.