Linux on Random notes from mg

Booting ISO images, 2020 edition

marius@gedmin.as (Marius Gedminas) — Fri, 22 May 2020 18:57:21 +0300

Now that Ubuntu 20.04 LTS is out, it was time to update my bootable USB drive with a bunch of ISO images. Except I got tired of editing grub.cfg by hand. So I wrote a script.

Now all I have to do is plug in my USB drive, download a new ISO into it, run make in the /boot/grub subdirectory, and presto! It works.

What the script does:

finds all ISO files in ../../ubuntu/*.iso
extracts /boot/grub/grub.cfg from inside the ISO image (yes, I wrote an ISO 9660 filesystem parser in Python)
extracts the kernel command line
adds the iso-scan/filename=… argument to make it boot from an ISO image
generates a new grub.cfg with a menu listing all Ubuntu versions and all image variants (desktop vs server)

This works for all Ubuntu ISO images based on casper, which is a tool Ubuntu uses to produce their LiveCD images.

Notably it doesn’t work for older Ubuntu server images based on debian-installer.

I don’t know about other distributions and Ubuntu variants. Ubuntu variants ought to work, but my script might generate wrong titles for them.

Ansible is great

marius@gedmin.as (Marius Gedminas) — Fri, 16 Sep 2016 13:59:19 +0300

I haven’t used configuration management tools in a very long time because I didn’t know which tool to pick (Chef? Puppet? Ansible? Salt? cfengine?). Now I use Ansible and am very happy with it.

Here’s how you start – with an empty git repository:

$ mkdir my-infra
$ cd my-infra
$ git init .

Let’s start by making sure a couple of Ubuntu servers will have unattended upgrades enabled. First we tell Ansible what the servers are

$ vim hosts
$ cat hosts
debesis
ranka

Let’s make sure Ansible can ssh to those hosts:

$ ansible -i hosts all -m ping
ranka | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
debesis | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

I’ve an ~/.ssh/config that expands these nicknames to the full hostnames, and I’ve a private SSH key that allows me access without providing my password every time. The key is passphrase-protected and I use ssh-agent to avoid typing my passphrase every time. Notice a trend? I like convenience.

Speaking of convenience, it’s annoying to have to specify -i hosts every time (since the default inventory file is /etc/ansible/hosts and I never ever want that), so

$ vim ansible.cfg
$ cat ansible.cfg
[defaults]
inventory = hosts

Now ansible all -m ping should work. All this needs on the servers is SSH and Python (2 or 3).

We can do more useful ad-hoc stuff too, like apply any pending security updates:

$ ansible all -m apt -a 'update_cache=yes upgrade=dist' -u root
debesis | SUCCESS => {
    "changed": false, 
    "msg": "Reading package lists...\nBuilding dependency tree...\nReading state information...\n0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.\n", 
    "stderr": "", 
    "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\n0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.\n", 
    "stdout_lines": [
        "Reading package lists...", 
        "Building dependency tree...", 
        "Reading state information...", 
        "0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded."
    ]
}
ranka | SUCCESS => {
    "changed": false, 
    "msg": "Reading package lists...\nBuilding dependency tree...\nReading state information...\n0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.\n", 
    "stderr": "", 
    "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\n0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.\n", 
    "stdout_lines": [
        "Reading package lists...", 
        "Building dependency tree...", 
        "Reading state information...", 
        "0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded."
    ]
}

This runs apt-get update && apt-get dist-upgrade, so it needs root access. You can ssh directly as root (-u root), or you can use sudo (-b -K). I prefer ssh’ing as root because I don’t want to be typing my sudo password every time I run ansible. Let’s make this the default then:

$ vim ansible.cfg
$ cat ansible.cfg
[defaults]
inventory = hosts
remote_user = root

Now, back to unattended-upgrades. Let’s create a playbook:

$ vim setup.yml
$ cat setup.yml
---
- hosts: all
  tasks:
    - name: install unattended-upgrades
      apt: name=unattended-upgrades state=present

    - name: enable unattended-upgrades
      copy:
        dest: /etc/apt/apt.conf.d/50unattended-upgrades-local
        content: |
          APT::Periodic::Update-Package-Lists "1";
          APT::Periodic::Unattended-Upgrade "1";
          Unattended-Upgrade::Mail "root";
          Unattended-Upgrade::Remove-Unused-Dependencies "true";
          Unattended-Upgrade::Automatic-Reboot "true";
          Unattended-Upgrade::Allowed-Origins {
            "${distro_id}:${distro_codename}-security";
            "${distro_id}:${distro_codename}-updates";
          };

It does two things: install a package and set up a config file. Let’s see what it would do:

$ ansible-playbook setup.yml -CD

PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
ok: [ranka]
ok: [debesis]

TASK [install unattended-upgrades] *********************************************
ok: [ranka]
ok: [debesis]

TASK [enable unattended-upgrades] **********************************************
changed: [ranka]
--- before
+++ after: /tmp/tmpcIWt6O
@@ -0,0 +1,9 @@
+APT::Periodic::Update-Package-Lists "1";
+APT::Periodic::Unattended-Upgrade "1";
+Unattended-Upgrade::Mail "root";
+Unattended-Upgrade::Remove-Unused-Dependencies "true";
+Unattended-Upgrade::Automatic-Reboot "true";
+Unattended-Upgrade::Allowed-Origins {
+  "${distro_id}:${distro_codename}-security";
+  "${distro_id}:${distro_codename}-updates";
+};

changed: [debesis]
--- before
+++ after: /tmp/tmpePUnG4
@@ -0,0 +1,9 @@
+APT::Periodic::Update-Package-Lists "1";
+APT::Periodic::Unattended-Upgrade "1";
+Unattended-Upgrade::Mail "root";
+Unattended-Upgrade::Remove-Unused-Dependencies "true";
+Unattended-Upgrade::Automatic-Reboot "true";
+Unattended-Upgrade::Allowed-Origins {
+  "${distro_id}:${distro_codename}-security";
+  "${distro_id}:${distro_codename}-updates";
+};


PLAY RECAP *********************************************************************
debesis                    : ok=3    changed=1    unreachable=0    failed=0   
ranka                      : ok=3    changed=1    unreachable=0    failed=0

-CD stands for --check --diff, which tells Ansible not to make any changes, but show what it would do, with full unified diffs. Ansible’s killer feature IMHO. Probably the main reason I why switched from Fabric to Ansible.

Everything looks fine, so we can apply the changes – but let’s be cautious and only do it for only one machine for now

$ ansible-playbook setup.yml --limit debesis

PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
ok: [debesis]

TASK [install unattended-upgrades] *********************************************
ok: [debesis]

TASK [enable unattended-upgrades] **********************************************
changed: [debesis]

PLAY RECAP *********************************************************************
debesis                    : ok=3    changed=1    unreachable=0    failed=0

And we’re done!

$ git add .
$ git commit -m "Set up unattended-upgrades"

Ansible has great documentation, if you’d like to learn more.

Updated 2020-05-23 to remove the bit about requiring Python 2.x specifically. Python 3 works too nowadays, and Python 2 is EOL.

Rackspace OSS

marius@gedmin.as (Marius Gedminas) — Thu, 21 Nov 2013 10:47:20 +0200

I've a Jenkins server running tests for most of my open-source projects now: https://jenkins.gedmin.as/. It was not too difficult to set up:

Send an email to Jesse Noller about that free Rackspace Cloud hosting for OSS projects, expecting to be rebuffed because all my projects are small and insignificant.
Receive a positive response with the instructions. Yay!
Sign up for an account, type in all my credit card details and personal phone numbers, get it verified by a phone call (I hate phone calls, but apparently it prevents fraud or something). Send the account name/ID back to Jesse.
Log in to the MyCloud account control panel/dashboard/thingie, create a server, ask for Ubuntu 12.04 LTS.
Wait a couple of minutes, ssh as root, do the usual setup (locale-gen, adduser, ~/.ssh/authorized_keys, dotfiles, etckeeper, postfix, my PPA for sysadmin automation stuff).
There was an amusing interlude where I tried to set up the Rackspace Cloud Monitoring agent according to their instructions, failed, had to open a support ticket, give their techs a login with root, then wait less than 24 hours until it was resolved. I still don't know what went wrong, but it works now.
Get and install a free SSL certificate from StartSSL.
Install and configure Jenkins (latest upstream version, just in case).
Point my DNS to the server's IP.

It's been running this way for a month now, with no problems. So, thanks, Rackspace!

Grand plans for the future: have Jenkins do daily/weekly mirrors of all my GitHub repos, look for commits made since the last release tag, filter out boring ones ("bump version number"), and send me reminders that "project X needs a new release: you've new features sitting there unreleased for X days now". Or maybe just a web page with a table of "X commits since last release" linking to GitHub history.

P. S. I can't preview this since PyBlosxom fails to run on my laptop with a cryptic error. I'm debating debugging this versus migrating to a static blog compiler.

Booting ISO images from a USB drive

marius@gedmin.as (Marius Gedminas) — Mon, 03 May 2010 20:24:31 +0300

Dear lazyweb, I would like to download an arbitrary ISO image (say, a Ubuntu 10.04 Desktop CD) into a directory of a USB flash drive, and then make that USB flash drive boot that ISO image. I do not want to

re-partition or re-format the flash drive (this eliminates usb-creator, AFAIU)
extract the contents of the ISO image into the root of the USB drive (this eliminates unetbootin)
skip the ISO's bootloader and directly boot the kernel+initramfs from the ISO (eliminates this recipe, and this recipe)

I just want a bootloader on the USB to read the VFAT filesystem, mount the ISO image as a loop device, then chain-load the bootloader from that ISO. Bonus points for having a menu letting me choose one of several ISO images. Running a script to edit a text file (say, grub's config) to get that menu is fine.

Is this even possible? If not, can I at least have two out of three (no partition/extraction, but skipping intrinsic bootloader is fine)?

Solution that I finally chose (from Ubuntu forums):

Plug in USB key. Find out the mount point (/media/disk) and the device name (/dev/sdx) of the USB key with

$ mount|grep /media

Install GRUB 2 into the USB key with

$ sudo grub-install --root-directory=/media/usbdisk /dev/sdx

If it says something about embedding being impossible and falling back to UNRELIABLE blocklist-based setup, run

$ sudo grub-install --root-directory=/media/usbdisk /dev/sdx --force

Download a CD image, let's say ubuntu-10.10-desktop-i386.iso. Put it into /media/usbdisk/ubuntu/.

Create a text file /media/usbdisk/boot/grub/grub.cfg with

menuentry "Ubuntu 10.10 (x86 desktop livecd)" {
    set isofile="/ubuntu/ubuntu-10.10-desktop-i386.iso"
    loopback loop $isofile
    linux (loop)/casper/vmlinuz boot=casper iso-scan/filename=$isofile quiet splash noprompt --
    initrd (loop)/casper/initrd.lz
}

You can have as many ISO images as you want, just make sure to add a menuentry for each. There's no need to run grub-install again after adding or removing a .iso file. Oh, and if you want to use an ISO file for a different distribution, you'll have to figure out the correct linux and initrd lines somehow.

Update: Ubuntu 13.04 changed the name of the kernel -- use (loop)/casper/vmlinuz.efi instead of (loop)/casper/vmlinuz.

I tested it with the following images

ubuntu-10.04-desktop-i386.iso
ubuntu-10.04-server-i386.iso
ubuntu-10.04-server-amd64.iso
ubuntu-10.10-desktop-i386.iso
ubuntu-11.04-desktop-i386.iso
ubuntu-11.10-desktop-i386.iso
ubuntu-12.04-desktop-i386.iso
ubuntu-12.04-desktop-amd64.iso
ubuntu-12.04-server-i386.iso
ubuntu-12.04-server-amd64.iso
ubuntu-12.10-desktop-i386.iso
ubuntu-12.10-desktop-amd64.iso
ubuntu-13.04-desktop-amd64.iso

This solution skips the CD-ROM's boot menu. I haven't found a better one.

Update: I wrote a script to generate a grub.cfg for me so I don't have to do it by hand any more.

Unix is an IDE, or my Vim plugins

marius@gedmin.as (Marius Gedminas) — Wed, 09 Dec 2009 01:23:53 +0200

Unix is an IDE. I do my development (Python web apps mostly) with Vim with a bunch of custom plugins, shell (in GNOME Terminal: tabs rule!), GNU make, ctags, find + grep, svn/bzr/hg/git.

The current working directory is my project configuration/state. I run tests here (bin/test), I search for code here (vim -t TagName, find + grep), I run applications here (make run or bin/appname). I can multitask freely, for example, if I'm in the middle of typing an SVN commit message, I can hit Ctrl+Shift+T, get a new terminal tab in the same working directory, and look something up. No aliases/environment variables/symlinks. I can work on multiple projects at the same time. I can work remotely (over ssh).

Gary Bernhardt's screencasts on Vimeo show how productive you can get if you learn Vim and tailor it to your needs. I have Vim scripts that let me

See the name of the class and function that I'm editing in the statusbar, even if the class/function definition is offscreen: pythonhelper.vim.
See all pyflakes warnings and errors in a list as soon as I press F2 to save the file: python_check_syntax.vim.
Add a "from foo.bar import Something" line at the top of the file if I press F5 when my cursor is on Something, looking up the package and module from ctags: python-imports.vim.
Switch between production code and unit tests with a single key if the project uses one of several conventions for tests (e.g. ./foo.py <-> ./tests/test_foo.py): py-test-switcher.vim.
Generate a command line for running one particular unit test (the one my cursor is inside) and copy it into the system clipboard, so I can run that test by Alt-Tabbing into my terminal window and pasting. py-test-runner.vim.
Open the right file and move the cursor to the right line if I triple-click a line of traceback in a shell (or an email) then press F7 in my gvim window: py-test-locator.vim.
Compare my version of the code with the pristine version in source control in an interactive side-by-side diff that lets me revert bits I no longer want: vcscommand.vim.
Highlight which lines of the source are covered by my tests, if I have coverage information in trace.py format: py-coverage-highlight.vim.
Show the signature of a function/class's __init__ when I type the name of that class/function and an open parenthesis (looked up from tags): py-function-signature.vim.
Fold code into an outline so I only see names of methods or classes instead of their full bodies: vimrc, function PythonFoldLevel.
Fold diff files so I can see whole hunks/files and can delete those with a single key (well, two keys -- dd). Useful for reviewing large diffs (tens of thousands of lines): vimrc, function DiffFoldLevel.

Some of these come from www.vim.org, some I've written myself, some I've taken and modified a little bit to avoid an irritating quirk or add a missing feature. Some things I don't have (and envy Emacs or IDE users for having -- like an integrated debugger for Python apps, and, generally, integration with other tools, running in the background).

It's been my plan for a long time to polish my plugins, release them somewhere (github? bitbucket? launchpad?) and upload to vim.org, but as it doesn't seem to be happening, I thought I'd at least put an svn export of my ~/.vim on the web.

Escaping hotel firewalls with ssh over port 80

marius@gedmin.as (Marius Gedminas) — Sun, 11 Oct 2009 22:09:51 +0200

I booked a stay at a particular hotel because the web page said "Free WiFi". It didn't say "all outgoing ports firewalled except for port 80 and a few other (useless) ones". Not having SSH access is most painful. Luckily, there's a solution.

You need a web server running Apache and SSH. Enable mod_proxy and mod_proxy_connect and add this to the first (i.e. default) virtual host configuration:

<VirtualHost whatever:80>
...

  # allow ssh to localhost over http proxy
  ProxyRequests on
  AllowCONNECT 22
  <Proxy localhost>
    Order allow,deny
    Allow from all
  </Proxy>

</VirtualHost>

Reload Apache configuration. The setup is done. (Instructions based on Tunneling SSH over HTTP(S) by Dag Wieers.)

On the client side you need proxytunnel. Sadly, it's not packaged for Ubuntu yet, but compiling from sources is trivial. Edit ~/.ssh/config and add an entry for your proxied ssh connection:

Host pmyservername
ProxyCommand proxytunnel -q -p myserver.mydomain.com:80 -d localhost:22

That's it. Now you can ssh pmyservername. (The p prefix is a reminder that I'm using a proxied connection: ssh fridge versus ssh pfridge. Also it reminds me of Terry Pratchett's Pyramids.).

For extra fun (e.g. IRC) use ssh's built-in SOCKS5 proxy: ssh -D 1080 pmyservername. Then tell the apps to use a SOCKS5 proxy on localhost. Since telling each app to use a proxy (and then, later, telling it to stop using it) is a big *pain*, and some apps (e.g. ssh) don't support proxies directly, a wrapper like tsocks is handy. Edit /etc/tsocks.conf and set the default socks server to 127.0.0.1, then use it to run apps:

$ tsocks xchat-gnome
$ tsocks bzr push lp:myprojectname

tsocks is packaged for Ubuntu.

If your hotel doesn't have free WiFi, a prepaid SIM card with 3G access could be cheaper than roaming charges. Apparently you can get one with a virtually unlimited (for a short stay, anyway) data plan for 27 EUR in Amsterdam.