Introducing dqsub

I've been using qsub for a while now on the cluster here at the IGB at UofI. qsub is a command line program which is used to submit jobs to a scheduler to eventually be run on one (or more) nodes of a cluster.

Unfortunately, qsub's interface is horrible. It requires that you write a shell script for every single little thing you run, and doesn't do simple things like providing defaults or running multiple jobs at once with slightly different arguments. I've dealt with this for a while using some rudimentary shell scripting, but I finally had enough.

So instead, I wrote a wrapper around qsub called dqsub.

What used to require a complicated invocation like:

echo -e '#!/bin/bash\nmake foo'| \
 qsub -q default -S /bin/bash -d $(pwd) \
  -l mem=8G,nodes=1:ppn=4 -;

can now be run with

dqsub --mem 8G --ppn 4 make foo;

Want to run some command in every single directory which starts with SRX? That's easy:

ls -1 SRX*|dqsub --mem 8G --ppn 4 --array chdir make bar;

Want instead to behave like xargs but do the same thing?

ls -1 SRX*|dqsub --mem 8G --ppn 4 --array xargs make bar -C;

Now, this wrapper isn't complete yet, but it's already more than enough to do what I require, and has saved me quite a bit of time already.

You can steal dqsub for yourself

Feel free to request specific features, too.

Adding a Table of Contents to PDFs from R

I routinely generate very large PDFs from R which have hundreds (or thousands) of pages, and navigating these pages can be very difficult. Unfortunately, neither R's pdf() nor its cairopdf() drivers support creating Table of Contents (or Index) while plots are being written out. In the case of cairo, the underlying library doesn't support it either, so this isn't something that can easily be added to R directly. I had been thinking about sitting down for months and writing the support into cairo and R's cairo package... but real life kept getting in the way.

Fast forward to a week ago, when I realized that pdftk does support dumping the table of contents and updating the table of contents using dump_data_utf8 and update_info_utf8! Armed with that knowledge, and a bit of hackery, we can save an index, and then update the pdf once it's been closed.

The R code then looks like the following:

 ..device.set.up <- FALSE <<- 0

 save.bookmark <- function(text,bookmarks=list(),level=1,page=NULL) {
     if (!..device.set.up) {
         Cairo.onSave(device = dev.cur(),
                 <<- page
         ..device.set.up <<- TRUE
     if (missing(page)|| is.null(page)) {
         page <-
     bookmarks[[length(bookmarks)+1]] <-

 write.bookmarks <- function(pdf.file,bookmarks=list()) {
     pdf.bookmarks <- ""
     for (bookmark in 1:length(bookmarks)) {
         pdf.bookmarks <-
                    "BookmarkTitle: ",bookmarks[[bookmark]]$text,"\n",
                    "BookmarkLevel: ",bookmarks[[bookmark]]$level,"\n",
                    "BookmarkPageNumber: ",bookmarks[[bookmark]]$page,"\n")
     temp.pdf <- tempfile(pattern=basename(pdf.file)) <- tempfile(pattern=paste0(basename(pdf.file),"info_utf8"))
     if (file.exists(temp.pdf)) {
     } else {
         warning("unable to properly create bookmarks")

and can be used like so:

 bookmarks <- list()
 bookmarks <- save.bookmark("First plot",bookmarks)
 bookmarks <- save.bookmark("Second plot",bookmarks)

et voila. Bookmarks and a table of contents for PDFs.

This basic methodology can be extended to any language which writes PDFs and does not have a built-in method for generating a Table of Contents. Currently, the usage of Cairo.onSave is a horrible hack, and may conflict with anything else which uses the onSave hook, but hopefully R will report the current page number from Cairo in the future.

Adding a newcomer (⎈) tag to the BTS

Some of you may already be aware of the gift tag which has been used for a while to indicate bugs which are suitable for new contributors to use as an entry point to working on specific packages. Unfortunately, some of us (including me!) were unaware that this tag even existed.

Luckily, Lucas Nussbaum clued me in to the existence of this tag, and after a brief bike-shed-naming thread, and some voting using pocket_devotee we decided to name the new tag newcomer, and I have now added this tag to the BTS documentation, and tagged all of the bugs which were user tagged "gift" with this tag.

If you have bugs in your package which you think are ideal for new contributors to Debian (or your package) to fix, please tag them newcomer. If you're getting started in Debian, and working on bugs to fix, please search for the newcomer tag, grab the helm, and contribute to Debian.

Virginia King selected for Debbugs FOSS Outreach Program for Women

I'm glad to announce that Virginia King has been selected as one of the three interns for this round of the FOSS Outreach Program for women. Starting December 9th, and continuing until March 9th, she'll be working on improving the documentation of Debian's bug tracking system.

The initial goal is to develop a Bug Triager Howto to help new contributors to Debian jump in and help existing teams triage bugs. We'll be getting in touch with some of the larger teams in Debian to help make this document as useful as possible. If you're a member of a team in Debian who would like this howto to address your specific workflow, please drop me an e-mail, and we'll keep you in the loop.

The secondary goals for this project are to:

  • Improve documentation under
  • Document of bug-tags and categories
  • Improve upstream debbugs documentation
ErgoDox keyboard assembly

I routinely use a Kinesis Advantage Pro keyboard, which is a split, ergonomic keyboard with thumb clusters that uses brown cherryMX switches. Over the thirteen years that I've been using it, I've become a huge fan of this style of keyboard. However, I have two major annoyances with the Kinesis. First, while the firmware is good, remapping the keys is complicated and producing more complicated keyboard layouts with layers and keycodes that are not present in the original layout is not possible. Secondly, the interconnect between the main key wells and the controller board in the middle occasionally fails, and requires disassembly and occasional re-tinning of the circuit board interconnect connector.

1 About a year ago, I became aware of the ErgoDox keyboard, which is a keyboard design which mimics the kinesis to some degree, but with completely separated key halves (useful, because I'm substantially bigger than the average human), programmable firmware (so I can finally have the layers and missing keys) and with slightly more elegant interconnects (TRRS cables). Unfortunately, at the time I first heard about it (and other custom keyboards), making it required sourcing circuit boards, parts, and finding someone to cut a case for the keyboard. Then, a few months ago, I learned about MassDrop, a company who puts together groups of people to do buys of products at near-wholesale level prices, and their offer of all of the parts to build an ErgoDox. After waiting for a group buy of the keyboard to become available, I put in an order, and received the parts two months later.

Over a few hours yesterday, I learned how to do surface mount soldering of the 78 diodes (one for each key), and finished assembling and flashing the firmware. This morning, I fixed up the few key bindings that I needed to be productive, and viola, my laptop at home now has a brand new ergonomic keyboard.

Dropbox Recursive Downloader

I'm working on some analyses for the Genetic Analysis Workshop #19, which has placed it's data on Dropbox. Unfortunately, Dropbox doesn't allow for people to download zip archives larger than 1GB, and the data was made available in an unpacked structure with more than a hundred files. Some searching indicated that no one had written a recursive downloader for Dropbox, so 30 minutes of hacking with WWW::Mechanize later, I wrote a simple recursive downloader for Dropbox.

Two hours later, all of the files had downloaded.

Debian Booth at Scale 12x

I spent the weekend at SCALE 12x running the Debian booth. SCALE is one of the best conferences that I get to attend every year; it has a great mix of commercial exhibitors and community groups, and routinely gets great speakers. As I've done for quite some time, I organized a Debian booth there, and talked to lots of people about Debian.

If you're in the Southern California area, or have a chance to give a talk for SCALE 13x, you should do so! Thanks again to Matt Kraai and Paul Hardy for helping out in the Debian booth all weekend!

Plotting Embedded Bitmap in Vector Plot in R

Recently, one of my collaborators complained that one of my plots took forever to render on his machine. The particular plot in question had a few thousand points, many of which were overlapping. Ideally, R would be able to simplify the vector image which was drawn to avoid drawing points which were occluded by other points, but this is difficult to do properly, and R currently doesn't do it.

However, R is able to plot to a bitmap, and bitmap images have the nice property of automatically handling this for you. Furthermore, raster images have recently been made far less clunky in R, so it's pretty easy to shove an arbitrary bitmap image anywhere. With dev.capture in Cairo coupled with grid.raster in grid, we have everything we need to solve this problem:


start.rasterplot <- function(width=NULL,height=NULL) {
    x.y.ratio <- convertX(unit(1,"npc"),"mm",valueOnly=TRUE)/
    width.points <- as.numeric(convertX(unit(1,"npc"),"points"))
    dpi <- as.numeric(convertX(unit(1,"inch"),"point"))
    if (is.null(width) && is.null(height)) {
        width <- 1024
    if (is.null(width)) {
        width <- height*x.y.ratio
    if (is.null(height)) {
        height <- width/x.y.ratio

stop.rasterplot <- function(plot=TRUE) {
    raster.image <- dev.capture(native=TRUE)
    if (plot) {
    } else {

Now we can do the following:


and our PDF will contain a raster image, and will load in seconds instead of taking forever to plot the file.

Working with Org-mode: Committing Changes Everywhere

I'm a huge fan of Org-mode, and I keep all of my org-mode files in git repositories which are under myrepos control.

However, because I often make lots of changes to my agenda and notes, I hate having to manually visit each individual project and make changes to it. [And it's also annoying when I forget to commit a specific change and then have to try to get my laptop and desktop back into sync.]

Luckily, myrepos can easily run a command in parallel in all of the repositories! The following "update_org_files" command will update all of my org-file containing repositories in parallel:


ORG_GREP='-e .org$ -e .org_archive$ -e .org_done$'

if [ "x$1" == "xdoit" ]; then
    if git status --porcelain -z | grep -z '^ M' | grep -zq $ORG_GREP; then
        git status --porcelain -z | grep -z '^ M' | grep -z $ORG_GREP | \
            sed -z 's/^ M//g' | \
            xargs -0 git commit -m'update org files'
        git push;
    emacsclient -n -e '(org-save-all-org-buffers)' >/dev/null 2>&1
    mr -d ~ -j5 run update_org_files doit;

An updated version of this lives in my git repository

Biblatex format for Genes & Immunity

Here's a biblatex format for Genes & Immunity (a Nature imprint) which I needed recently:

The following code in the preamble does almost all of the hard lifting:

% from
% from
% remove commas between authors and first inits
% from
% make et al. /et al./
\renewbibmacro*{name:andothers}{% Based on name:andothers from biblatex.def
    test {\ifnumequal{\value{listcount}}{\value{liststop}}}
    test \ifmorenames
% \renewbibmacro{journal}{#1}%
% from
    test {\iffieldundef{title}}
    test {\iffieldundef{subtitle}}

This blog is powered by ikiwiki.