From: W. Trevor King <wking@drexel.edu>
Date: Thu, 18 Nov 2010 12:51:21 +0000 (-0500)
Subject: Add Abax post.
X-Git-Url: http://git.tremily.us/?a=commitdiff_plain;h=577123a27512afe1ef495291b42d4f6aa08267d9;p=mw2txt.git

Add Abax post.
---

diff --git a/.gitignore b/.gitignore
index aa3a10e..365fa1e 100644
--- a/.gitignore
+++ b/.gitignore
@@ -3,6 +3,12 @@
 /recentchanges
 
 # built by Makefiles inside posts/
+posts/Abax/*.pdf
+posts/Abax/*/main.aux
+posts/Abax/*/main.log
+posts/Abax/*/main.out
+posts/Abax/*/main.pdf
+posts/Abax/*/main.toc
 posts/drexel-thesis/*.pdf
 posts/drexel-thesis/drexel-thesis/
 /posts/Hooke/README.rst
diff --git a/Makefile b/Makefile
index 32dee29..6141a51 100644
--- a/Makefile
+++ b/Makefile
@@ -1,7 +1,7 @@
 # Build everything necessary to compile the wiki.
 
-SUBDIRS = posts/drexel-thesis posts/Hooke posts/problempack posts/rel-vcs \
-	posts/sawsim posts/Thesis posts/yacc2dot posts/Xmodmap \
+SUBDIRS = posts/Abax posts/drexel-thesis posts/Hooke posts/problempack \
+	posts/rel-vcs posts/sawsim posts/Thesis posts/yacc2dot posts/Xmodmap \
 	posts/XSLT/chapter posts/XSLT/code
 TARGETS = all clean
 
diff --git a/posts/Abax.mdwn b/posts/Abax.mdwn
new file mode 100644
index 0000000..58192cd
--- /dev/null
+++ b/posts/Abax.mdwn
@@ -0,0 +1,52 @@
+[[!meta  title="Notes on using and managing clusters"]]
+
+Using
+-----
+
+Our cluster runs the open source [Torque][]/[Maui][] [[portable batch
+scheduling system|PBS]] (PBS).  A batch scheduler takes user submitted
+jobs, and distributes them across the the cluster in an intelligent
+manner, so users don't need to worry about sharing resources fairly or
+[[ssh]]ing into compute nodes to start their jobs.  Users submit jobs
+to the queue using `qsub`.  I've compiled my own [[brief
+intro|pbs_queues.pdf]] to `qsub`, and there are lots more floating
+about the internet.
+
+While PBS queues are great for distributing embarassingly parallel
+jobs across the cluster, your application may need processes running
+on seperate compute nodes to share data.  A common approach is to use
+the [[Message Passing Interface|MPI]] (MPI).  Our cluster uses the
+[mpich2][] implementation.  Cluster-aware applications written in MPI
+can be started through Torque using an [alternate mpiexec][mpiexec]
+from the [Ohio Supercomputer Center][OSC].  There is a nice, brief
+[introduction][] by Kristina Wanous at the [University of Northern
+Iowa][UNI].
+
+Managing
+--------
+
+Our cluster (9 dual-core nodes) runs [Debian][].  The compute nodes
+all boot to [[NFS roots|nfs_root.pdf]] off the server node.  Once that
+hurdle was passed, setting up Torque, Maui, mpich2, and mpiexec was
+pretty simple, mostly the usual:
+
+    wget ...
+    tar ...
+    configure ...
+    make
+    make install
+
+with a bit of configuring for our setup.  I'll put up some more
+detailed notes and our config options when I get the time.
+
+[[!tag tags/linux]]
+[[!tag tags/hardware]]
+
+[Torque]: http://www.clusterresources.com/pages/products/torque-resource-manager.php
+[Maui]: http://www.clusterresources.com/pages/products/maui-cluster-scheduler.php
+[mpich2]: http://www.mcs.anl.gov/research/projects/mpich2/
+[mpiexec]: http://www.osc.edu/~djohnson/mpiexec/
+[OSC]: http://www.osc.edu/~djohnson/mpiexec/
+[introduction]: http://debianclusters.cs.uni.edu/index.php/MPICH_with_Torque_Functionality
+[UNI]: http://www.uni.edu/
+[Debian]: http://www.debian.org/
diff --git a/posts/Abax/Makefile b/posts/Abax/Makefile
new file mode 100644
index 0000000..a91cba3
--- /dev/null
+++ b/posts/Abax/Makefile
@@ -0,0 +1,20 @@
+all: nfs_root.pdf pbs_queues.pdf
+
+clean:
+	rm -rf nfs_root.pdf pbs_queues.pdf
+	$(MAKE) -C nfs_root clean
+	$(MAKE) -C pbs_queues latex-clean pdf-clean  # no Torque/Maui on webserver
+
+nfs_root.pdf: nfs_root/main.pdf
+	cp $< $@
+
+.PHONY: nfs_root/main.pdf
+nfs_root/main.pdf:
+	$(MAKE) -C nfs_root main.pdf
+
+pbs_queues.pdf: pbs_queues/main.pdf
+	cp $< $@
+
+.PHONY: pbs_queues/main.pdf
+pbs_queues/main.pdf:
+	$(MAKE) -C pbs_queues main.pdf
diff --git a/posts/Abax/nfs_root/Makefile b/posts/Abax/nfs_root/Makefile
new file mode 100644
index 0000000..955d8ab
--- /dev/null
+++ b/posts/Abax/nfs_root/Makefile
@@ -0,0 +1,14 @@
+all : main.pdf
+
+view : main.pdf
+	xpdf main.pdf &
+
+semi-clean : 
+	rm -f main.aux main.log main.out main.toc
+
+clean : semi-clean
+	rm -f main.pdf
+
+main.pdf : main.tex
+	pdflatex main
+	pdflatex main
diff --git a/posts/Abax/nfs_root/main.tex b/posts/Abax/nfs_root/main.tex
new file mode 100644
index 0000000..8ab880e
--- /dev/null
+++ b/posts/Abax/nfs_root/main.tex
@@ -0,0 +1,740 @@
+\documentclass{article}
+
+\usepackage{url}
+\usepackage[colorlinks]{hyperref}
+\usepackage{fancyvrb}
+\usepackage{fullpage}
+
+%\renewcommand{\c}[1]{{\tt #1}}
+%\newcommand{\url}[1]{{\tt #1}}
+%\newenvironment{code}{\begin{verbatim}}{\end{verbatim}}
+\DefineVerbatimEnvironment{code}{Verbatim}{}
+\DefineVerbatimEnvironment{file}{Verbatim}{frame=single}
+% \begin{file}[label=/etc/example/file]
+% # some _ crazy \ contents
+% \end{file}
+\DefineVerbatimEnvironment{shell}{Verbatim}{}
+\newcommand{\inputfile}[1]{\VerbatimInput[frame=single,label=#1]{#1}}
+
+\newcommand{\Con}{\DefineShortVerb{\|}}
+\newcommand{\Coff}{\UndefineShortVerb{\|}}
+
+\author{W.\ Trevor King\thanks{Drexel University}}
+\date{\today}
+\title{Diskless Debian Etch}
+
+\begin{document}
+\maketitle
+\tableofcontents
+\Con
+
+\section{Overview}
+
+This HOWTO details the procedure I used to set up the abax cluster for NFS-rooted network booting.
+The system is useful in this case because it centralizes the installation in the head node (server), which makes maintaining, upgrading, or altering the computational nodes (clients) easier.
+
+This procedure follows mainly Tim Brom's Microwulf configuration notes\cite{microwulf} with two major differences.
+
+\begin{itemize}
+ \item Microwulf uses Ubuntu (gutsy?), and I'm using Debian etch.
+ \item Microwulf has a seperate partition for each client's root, populated with an independent installation from CD.
+  I'm using a single partition for all of my clients, with the base system created using debootstrap (no CD).
+\end{itemize}
+
+For guidance in my deviations, I'm indebted to Bart Trojanowski's pxeboot and nfsroot notes\cite{debian-nfsroot} and Falko Timme's notes on kernel compilation in Debian Etch\cite{kernel}.
+
+
+\subsection{Physical setup}
+
+Our cluster has one server with eight clients.
+The server has two network cards, |eth0| and |eth1|.
+|eth1| is connected to the outsize world (WAN).
+All of the clients have one network card, |eth0|.
+All of the |eth0|s are connected together through a gigabit switch (LAN).
+
+
+\subsection{Notation}
+
+Throughout this HOWTO, I will use |#| as the prompt for root, |$| as the prompt for an unpriveledged user, and |chroot#| as the prompt for a root in a |chroot|ed environment.
+File contents will be framed with the full path listed labeling the frame
+\begin{file}[label=/path/to/file]
+Contents of file
+\end{file}
+All files are complete with the exception of lines containing |...|, in which case the meaning of the example should be clear from the context.
+
+
+\section{Basic server setup}
+
+\subsection{Installing the OS}
+
+Boot the server with the Debian installation kernel following one of the options in the Debian installation guide\cite{install}.
+I netbooted my server from one of the client nodes following this procedure to set up the DHCP and TFTP servers on the client and untarring \url{http://http.us.debian.org/debian/dists/etch/main/installer-i386/current/images/netboot/netboot.tar.gz} in my |tftpboot| directory.
+After netbooting from a client, don't forget to take that client down so you woln't have DHCP conflicts once you set up a DHCP server on your server.
+
+Install Debian in whatever manner seems most appropriate to you.  I partitioned my 160 GB drive manually according to
+\begin{center}
+\begin{tabular}{l l l l}
+ Mount point   & Type & Size \\
+ |/|         & ext3 & 280 MB \\
+ |/usr|      & ext3 & 20 GB \\
+ |/var|      & ext3 & 20 GB \\
+ |/swap|     & swap & 1 GB \\
+ |/tmp|      & ext3 & 5 GB \\
+ |/diskless| & ext3 & 20 GB \\
+ |/home|     & ext3 & 93.7 GB
+\end{tabular}
+\end{center}
+I went with a highly partitioned drive to ease mounting, since I will be sharing some partitions with my clients.
+To understand why partitioning is useful, see the Partition HOWTO\cite{partition}.
+
+You can install whichever packages you like, but I went with just the standard set (no Desktop, Server, etc.).  
+You can adjust your installation later with any of (not an exhaustive list)
+\begin{itemize}
+ \item |tasksel| : command line, coarse-grained package control\cite{tasksel}.
+ \item |apt-get| : command line, fine-grained package control.
+ \item |aptitude| : curses frontend for |apt-get|.
+ \item |synaptic| : gtk+ frontend for |apt-get|.
+ \item |dpkg| : command line, package-management without dependency checking.
+\end{itemize}
+
+The base install is pretty bare, but I don't need a full blown desktop, so I flesh out my system with
+\begin{shell}
+# apt-get install xserver-xorg fluxbox fluxconf iceweasel xterm xpdf
+# apt-get install build-essentials emacs21-nox
+\end{shell}
+which gives me a bare-bones graphical system (fire it up with |startx|), and
+a bunch of critical build tools (|make|, |gcc|, etc.).
+
+
+\subsection{Configuring networking}
+
+We need to set up our server so that |eth1| assumes it's appropriate static IP on the WAN, and |eth0| assumes it's appropriate static IP on the LAN.
+We achieve this by changing the default |/etc/network/interfaces| to
+\begin{file}[label=/etc/network/interfaces]
+# This file describes the network interfaces available on your system
+# and how to activate them. For more information, see interfaces(5).
+
+# The loopback network interface
+auto lo
+iface lo inet loopback
+
+allow-hotplug eth0           # start on boot & when plugged in
+iface eth0 inet static       # static LAN interface
+	address 192.168.2.100
+	netmask 255.255.255.0
+	broadcast 192.168.2.255	
+
+allow-hotplug eth1           # start on boot & when plugged in
+#iface eth1 inet dhcp        # WAN DHCP interface (not used)
+iface eth1 inet static       # WAN static interface
+	address XXX.XXX.YYY.YYY
+	netmask 255.255.128.0
+	broadcast XXX.XXX.127.255
+	gateway XXX.XXX.ZZZ.ZZZ
+\end{file}
+where I've censored our external IPs for privacy.
+The netmask selects which addresses belong to which networks.
+The way we've set it up, all 192.168.2.xxx messages will be routed out |eth0|, and everything else will go through |eth1| to it's gateway.
+See the Net-HOWTO\cite{net-howto} for more details.
+
+
+\section{Remote booting}
+
+\subsection{Server services}
+
+The clients will boot remotely using the Pre eXecution Environment (PXE).
+The boot procedure is
+\begin{enumerate}
+ \item Client powers on.
+ \item Client BIOS comes up, detects attached devices, and looks for a DHCP server for advice on network booting.
+ \item DHCP server gives client an IP address, domain name, host name, the IP address of the TFTP server, and the location of the bootloader on the TFTP server.
+ \item Client gets bootloader from TFTP server.
+ \item BIOS hands over control to bootloader.
+ \item Bootloader gets kernel and initial ramdisk from TFTP server.
+ \item Bootloader hands over control to kernel
+ \item Kernel starts up the system, mounting root via NFS.
+ \item \ldots after this point, it's just like a normal boot process.
+\end{enumerate}
+We can see that we need to set up DHCP, TFTP, and NFS servers (not necessarily on the same server, but they are in our case).
+
+\subsubsection{pxelinux}
+
+The bootloader can be obtained with
+\begin{shell}
+# apt-get install syslinux
+\end{shell}
+which installs it to |/usr/lib/syslinux/pxelinux.0| along with a manual\cite{pxe} and some other |syslinux| tools.
+
+\subsubsection{DHCP}
+
+Install a server with
+\begin{shell}
+# apt-get install dhcp
+\end{shell}
+
+Configure the server with
+\begin{file}[label=/etc/dhcpd.conf]
+allow bootp;  # maybe?
+allow booting;# maybe?
+option domain-name "your.domain.com";
+option domain-name-servers XXX.XXX.XXX.XXX,YYY.YYY.YYY.YYY;
+
+subnet 192.168.2.0 netmask 255.255.255.0 {
+	range 192.168.2.150 192.168.2.200; # non-static IP range
+	option broadcast-address 192.168.2.255;
+	option routers 192.168.2.100; # Gateway server
+	next-server 192.168.2.100;    # TFTP server
+	filename "pxelinux.0";        # bootloader
+
+	host n1 {
+		hardware ethernet ZZ:ZZ:ZZ:ZZ:ZZ:ZZ;
+		fixed-address 192.168.2.101;
+		option root-path "192.168.2.100:/diskless/n1";
+		option host-name "n1";
+	}
+	... more hosts for other client nodes ...
+}
+\end{file}
+This assigns the client a static hostname, domain name, and IP address according to it's ethernet address (aka MAC address).
+It also tells all the clients to ask the TFTP server on 192.168.2.100 for the bootloader |pxelinux.0|.
+For extra fun, it tells the clients to send packets to the router at 192.168.2.100 if they can't figure out where they should go, and to use particular DNS servers to resolve domain names to IP addresses.
+This gives them access to the outside WAN.
+I don't know yet if the booting options are necessary, since I don't know what they do.
+
+We also need to ensure that the DHCP server only binds to |eth0|, since starting a DHCP server on your WAN will make you unpopular with your ISP.
+\begin{file}[label=/etc/default/dhcp]
+INTERFACES="eth0"
+\end{file}
+
+Once the DHCP server is configured, you can start it with
+\begin{shell}
+# /etc/init.d/dhcp restart
+\end{shell}
+
+Check that the server is actually up with
+\begin{shell}
+# ps -e | grep dhcp
+\end{shell}
+and if it is not, look for error messages in
+\begin{shell}
+# grep -i dhcp /var/log/syslog
+\end{shell}
+
+
+\subsubsection{TFTP}
+
+There are several TFTP server packages.
+We use |atftpd| here, but |tftp-hpa| is also popular.
+Install |atftpd| with
+\begin{shell}
+# apt-get install atftpd xinetd
+\end{shell}
+where |xinetd| is a super-server (replacing |inetd|, see |man xinetd| for details).
+Configure |atftpd| with
+\begin{file}[label=/etc/xinetd.d/atftpd]
+service tftp
+{
+	disable     = no
+	socket_type = dgram
+	protocol    = udp
+	wait        = yes
+	user        = nobody
+	server      = /usr/sbin/in.tftpd
+	server_args = --tftpd-timeout 300 --retry-timeout 5 --bind-address 192.168.2.100
+		--mcast-port 1758 --mcast-addr 239.239.239.0-255 --mcast-ttl 1
+		--maxthread 100 --logfile /var/log/atftpd.log --verbose=10
+		/diskless/tftpboot
+}
+\end{file}
+Note that the |server_args| should all be on a single, long line, since I haven't been able to discover if |xinetd| recognizes escaped endlines yet.
+This configuration tells |xinetd| to provide TFTP services by running |in.tftpd| (the daemon form of |atftpd|) as user |nobody|.
+Most of the options we pass to |in.tftpd| involve multicasting, which I believe is only used for MTFTP (which |pxelinux.0| doesn't use).
+|--logfile /var/log/atftpd.log --verbose=10| logs lots of detail to |/var/log/atftpd.log| if it exists.
+You can create it with
+\begin{shell}
+# touch /var/log/atftpd.log
+# chown nobody.nogroup /var/log/atftpd.log
+\end{shell}
+
+The most important argument is |/diskless/tftpboot|, which specifies the root of the TFTP-served filesystem (feel free to pick another location if you would like).
+This is where we'll put all the files that the TFTP will be serving.
+It needs to be read/writable by |nobody|, so create it with
+\begin{shell}
+# mkdir \tftpboot
+# chmod 777 tftpboot
+\end{shell}
+(TODO: possibly set the sticky bit, remove writable?)
+
+Finally, we need to restart the |xinetd| server so it notices the new |atftpd| server.
+\begin{shell}
+# /etc/init.d/xinetd restart
+\end{shell}
+
+Check that the |xinetd| server is up with
+\begin{shell}
+# ps -e | grep xinetd
+\end{shell}
+and look for error messages in either of
+\begin{shell}
+# grep -i dhcp /var/log/syslog
+\end{shell}
+Just having |xinetd| up cleanly doesn't prove that |atftpd| is working though, it just shows that the |atftpd| configuration file wasn't too bungled.
+To actually test |atftpd| we need to wait to Section \ref{sec.synthesis} when we actually have files to test-transfer.
+
+
+\subsubsection{NFS}
+
+Install the NFS utilities on the server with
+\begin{shell}
+# apt-get install nfs-common nfs-kernel-server
+\end{shell}
+We go with the kernel server because we want fast NFS, since we'll be doing a lot of it.
+Set the NFS server up to export the root file systems and the user's home directories with
+\begin{file}[label=/etc/exports]
+/diskless/n1    192.168.2.0/24(rw,no_root_squash,sync,no_subtree_check)
+... other node root exports ...
+/diskless       192.168.2.0/24(rw,no_root_squash,sync,no_subtree_check) # unnecessary
+/home           192.168.2.0/24(rw,no_root_squash,sync,no_subtree_check)
+/usr            192.168.2.0/24(rw,no_root_squash,sync,no_subtree_check)
+\end{file}
+Then let the NFS server know we've changed the |exports| file with
+\begin{shell}
+# exportfs -av  # TODO: -r?
+\end{shell}
+
+Test that the NFS server is working properly by |ssh|ing onto one of the clients and running
+\begin{shell}
+client# mkdir /mnt/n1
+client# mount 192.168.2.100:/diskless/n1 /mnt/n1
+client# ls /mnt/n1
+... some resonable contents ...
+client# umount /mnt/n1
+client# rmdir /mnt/n1
+\end{shell}
+
+
+\subsection{Client setup}
+
+The only client setup that actually happens on the client is changing the BIOS boot order to preferentially boot from the network.
+Consult your motherboard manual for how to accomplish this.
+It should be simple once you get into the BIOS menu, which you generally do by pressing |del|, |F2|, |F12|, or some such early in your boot process.
+\emph{Everything else happens on the server}.
+
+
+\subsubsection{Root file system}
+
+We want to install a basic Debian setup on our clients.
+Since each client doesn't have it's own, private partition, we need to install Debian using |debootstrap|.
+\begin{shell}
+# apt-get install debootstrap
+# mkdir /diskless/n1
+# debootstrap --verbose --resolve-deps etch /diskless/n1
+# chroot /diskless/n1
+chroot# tasksel install standard
+chroot# dpkg-reconfigure locales
+chroot# apt-get install kernel-image-2.6-686 openssh-server nfs-client
+\end{shell}
+TODO: what get's installed with standard?
+See |/usr/share/tasksel/debian-tasks.desc| for a list of possible tasks and \url{http://d-i.alioth.debian.org/doc/internals/} for details on how a full installation from CD or netboot.
+
+We can also add a few utilities so we can work in our |chroot|ed environment
+\begin{shell}
+chroot# apt-get install emacs21-nox
+\end{shell}
+
+
+\subsubsection{Configuring etc}
+
+The client will be getting its hostnames from the DHCP server, so remove the default
+\begin{shell}
+# rm /diskless/n1/etc/hostname
+\end{shell}
+
+We also need to setup the |fstab| to mount |/home| and |/usr| from the server
+\begin{file}[label=/diskless/n1/etc/fstab]
+# /etc/fstab: static file system information.
+#
+# <file system> <mount point>   <type>  <options>       <dump>  <pass>
+# automatically mount nfs root and proc through other means
+192.168.2.100:/home /home 	nfs	defaults,nolock        0       0
+192.168.2.100:/usr /usr 	nfs	defaults,nolock        0       0
+# we're diskess so we don't need to mount the hard disk sda :)
+#/dev/sda1       /               ext3    defaults,errors=remount-ro 0       1
+/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
+/dev/fd0        /media/floppy0  auto    rw,user,noauto  0       0
+\end{file}
+
+
+\subsubsection{Kernel and initial ramdisk}
+\label{sec.kernel}
+
+The kernel version number shows up often in this section.
+You can determine your kernel version number (in my case 2.6.18-6-686) with |uname -r|.
+Because kernel versions change fairly frequently, I'll use |KERNEL_VERSION| to denote the kernel version string.
+
+Your kernel must be compiled with NFS root support if it's going to have an NFS root\cite{nfsroot}.
+You can determine whether your kernel supports NFS roots with
+\begin{shell}
+# grep 'ROOT_NFS' /diskless/n1/boot/config-KERNEL-VERSION
+\end{shell}
+I didn't have it in my default debian etch 2.6.18-6-686 kernel, so I had to recompile my kernel (see Appendix \ref{app.kernel} and Falko's notes\cite{kernel}).
+My compiled kernel had a version string |2.6.18-custom|.
+
+Most kernels boot using an initial ramdisk (a compressed root filesytem that lives in RAM\cite{initrd}.
+This ramdisk contains the necessary programs and scripts for booting the kernel.
+We need to create a ramdisk that can handle an NFS root, so |chroot| into your client filesystem and install some tools
+\begin{shell}
+chroot# apt-get install initramfs-tools 
+\end{shell}
+Configure future ramdisks for NFS mounting with
+\begin{file}[label=/etc/initramfs-tools/initramfs.conf]
+# Configuration file for mkinitramfs(8). See initramfs.conf(5).
+...
+BOOT=nfs    # was BOOT=local
+...
+\end{file}
+Compile a new |initrd| with
+\begin{shell}
+chroot# update-initramfs -u
+\end{shell}
+If you compiled your own kernel as in Appendix \ref{app.kernel} after setting up |initramfs.conf|, an appropriate ramdisk should have been created automatically.
+
+You can examine the contents of your ramdisk with
+\begin{shell}
+$ cp /diskless/n1/boot/initrd.img-2.6.18-6-686 initrd.img.gz
+$ gunzip initrd.img.gz
+$ mkdir initrd
+$ cd initrd/
+$ cpio -i --make-directories < ../initrd.img 
+\end{shell}
+
+
+\subsection{Synthesis}
+\label{sec.synthesis}
+
+To configure PXE, we need to bring |pxelinux.0| into our new |tftpboot| directory
+\begin{shell}
+# cp /usr/lib/syslinux/pxelinux.0 /diskless/tftpboot/
+\end{shell}
+
+We also need to bring in our kernel image and initial ramdisk
+\begin{shell}
+# cd /diskless/tftpboot
+# ln -s /diskless/n1/boot/initrd.img-2.6.18-custom
+# ln -s /diskless/n1/boot/vmlinuz-2.6.18-custom
+\end{shell}
+|atftpd| handles the symbolic links, but if your TFTP server doesn't, you'll have to copy the image and ramdisk over instead.
+
+At this point you should test your TFTP server with test transfers.
+Install the atftp client
+\begin{shell}
+# apt-get install atftp
+\end{shell}
+And attempt to transfer the important files.
+\begin{shell}
+$ atftp 192.168.2.100
+tftp> status
+Connected:  192.168.2.100 port 69
+Mode:       octet
+Verbose:    off
+Trace:      off
+Options
+ tsize:     disabled
+ blksize:   disabled
+ timeout:   disabled
+ multicast: disabled
+mtftp variables
+ client-port:   76
+ mcast-ip:      0.0.0.0
+ listen-delay:  2
+ timeout-delay: 2
+Last command: quit
+tftp> get pxelinux.0
+tftp> get initrd.img-2.6.18-custom
+tftp> get vmlinuz-2.6.18-custom
+tftp> quit
+$ ls -l
+...
+-rw-r--r-- 1 sysadmin sysadmin 4297523 2008-05-30 09:27 initrd.img-2.6.18-custom
+-rw-r--r-- 1 sysadmin sysadmin   13480 2008-05-30 09:26 pxelinux.0
+-rw-r--r-- 1 sysadmin sysadmin 1423661 2008-05-30 09:27 vmlinuz-2.6.18-custom
+...
+\end{shell}
+If this doesn't work, look for errors in |/var/log/syslog| and |/var/log/atftpd.log| and double check your typing in the |atftpd| configuration file.
+
+The last stage is to configure the |pxelinux.0| bootloader.
+Create a configuration directory in |tftboot| with
+\begin{shell}
+# mkdir /diskless/tftpboot/pxelinux.cfg
+\end{shell}
+When each client loads |pxelinux.0| during the boot, they look for a configuration file in |pxelinux.cfg|.
+The loader runs through a sequence of possible config file names, as described in |pxelinux.doc|.
+We'll have different rood directories for each of our nodes, so we need a seperate config for each of them.
+In order to make our configs machine-specific, we'll use the ethernet (MAC) address file-name scheme.
+That is, for a machine with MAC address AA:BB:CC:DD:EE:FF, we make the file |pxelinux.cgf/01-aa-bb-cc-dd-ee-ff|.
+TODO: base config on IP address.
+\begin{file}[label=/diskless/tftpboot/pxelinux.cfg/01-aa-bb-cc-dd-ee-ff]
+default linux
+
+label linux
+  kernel vmlinuz-2.6.18-custom
+  append root=/dev/nfs initrd=initrd.img-2.6.18-custom
+         nfsroot=192.168.2.100:/diskless/n1,tcp ip=dhcp rw
+\end{file}
+Note that the |append|ed args should all be on a single, long line, since I haven't been able to discover if |pxelinux| recognizes escaped endlines yet.
+This file is basically like a |grub| or |lilo| config file, and you can get fancy with a whole menu, but since this is a cluster and not a computer lab, we don't need to worry about that.
+Note that this file was only for our first node (|n1|).
+You have to make copies for each of your nodes, with the appropriate file names and |nfsroot|s.
+
+The kernel options are fairly self explanatory except for the |tcp| for the |nfsroot| option, which says the client should mount the root directory using TCP based NFS.
+Traditional NFS uses UDP, which is faster, but possibly less reliable for large files (like our kernel and initrd).
+However I'm having trouble tracking down a reliable source for this.
+For now, consider the |tcp| a voodoo incantation to be attempted if the NFS booting isn't working.
+
+You're done!
+Plug a monitor into one of the clients and power her up.
+Everything should boot smoothly off the server, without touching the client's harddrive.
+
+
+\section{Adding clients}
+
+To add a new client node |nX| to the cluster, we need to do the following (which can be combined into an |add-client| script).
+First, we need to create a root directory for the new client
+\begin{shell}
+# cd /diskless/
+# cp -rp n1 nX
+\end{shell}
+Now we need to export that directory
+\begin{shell}
+# echo '/diskless/nX	192.168.2.0/24(rw,no_root_squash,sync,no_subtree_check)' \
+    >> /etc/exports
+# exportfs -av
+\end{shell}
+Finally, we need to set up the booting and DHCP options
+\begin{shell}
+# cd /diskless/tftpboot
+# sed 's/\/diskless\/n1/\/diskless\/nX/' 01-xx-xx-xx-xx-xx-xx > 01-yy-yy-yy-yy-yy-yy
+# echo '	host n8 {
+		hardware ethernet YY:YY:YY:YY:YY:YY;
+		fixed-address 192.168.2.10X;
+		option root-path "192.168.2.100:/diskless/nX/";
+		option host-name "nX";
+	}' >> /etc/dhcpd.conf
+# /etc/init.d/dhcp restart
+\end{shell}
+
+
+
+\phantomsection
+\addcontentsline{toc}{section}{Appendix}
+\appendix
+
+
+\section{Compiling a kernel}
+\label{app.kernel}
+
+See Falko's notes\cite{kernel} for and excellent introduction, and the NFS-root mini-HOWTO\cite{nfsroot-mini} for NSF root particulars.
+
+First, grab a bunch of useful compilation tools
+\begin{shell}
+chroot# apt-get install wget bzip2 kernel-package
+chroot# apt-get install libncurses5-dev fakeroot build-essential initramfs-tools 
+\end{shell}
+Some of these (e.g.\ |wget|) should already be installed, but apt-get will realize this, so don't worry about it.
+Configure |initramfs| for building NFS root-capable initial ramdisks by setting up \\ %HACK, force newline
+|/etc/initramfs-tools/initramfs.conf| as explained in Section \ref{sec.kernel}.
+For NSF root, your kernel needs the following options\cite{nfsroot-mini}:\\
+\begin{tabular}{l l}
+  |IP_PNP_DHCP| &  Networking \\
+	          &  $\quad\rightarrow$ Networking support (|NET [=y]|) \\ 
+		  &  $\quad\quad\rightarrow$ Networking options \\
+		  &  $\quad\quad\quad\rightarrow$ TCP/IP networking (|INET [=y]|) \\
+		  &  $\quad\quad\quad\quad\rightarrow$ IP: kernel level autoconfiguration (|IP_PNP =y|) \\
+  |ROOT_NFS|(|NET && NFS_FS=y && IP_PNP|)    &  File systems \\
+	          &  $\quad\rightarrow$ Network File Systems
+\end{tabular}
+
+I also used the build-in NFS client instead of the module.
+Here is a |diff| of the original debian etch conf vs.\ mine:
+\begin{file}[label=diff /diskless/n1/boot/config-2.6.18-6-686 .config]
+4c4
+< # Sun Feb 10 22:04:18 2008
+---
+> # Thu May 29 23:59:47 2008
+402c402,405
+< # CONFIG_IP_PNP is not set
+---
+> CONFIG_IP_PNP=y
+> CONFIG_IP_PNP_DHCP=y
+> CONFIG_IP_PNP_BOOTP=y
+> CONFIG_IP_PNP_RARP=y
+3314c3317
+< CONFIG_NFS_FS=m
+---
+> CONFIG_NFS_FS=y
+3325c3328,3329
+< CONFIG_LOCKD=m
+---
+> CONFIG_ROOT_NFS=y
+> CONFIG_LOCKD=y
+3328c3332
+< CONFIG_NFS_ACL_SUPPORT=m
+---
+> CONFIG_NFS_ACL_SUPPORT=y
+3330,3332c3334,3336
+< CONFIG_SUNRPC=m
+< CONFIG_SUNRPC_GSS=m
+< CONFIG_RPCSEC_GSS_KRB5=m
+---
+> CONFIG_SUNRPC=y
+> CONFIG_SUNRPC_GSS=y
+> CONFIG_RPCSEC_GSS_KRB5=y
+3485c3489
+< CONFIG_CRYPTO_DES=m
+---
+> CONFIG_CRYPTO_DES=y
+\end{file}
+
+Compile your shiny, new kernel with 
+\begin{shell}
+chroot# make-kpkg clean
+chroot# fakeroot make-kpkg --initrd --append-to-version=-custom kernel_image kernel_headers
+\end{shell}
+The new kernel packages are in the src directory
+\begin{shell}
+chroot# cd /usr/src
+chroot# ls -l
+\end{shell}
+
+Install the packages with
+\begin{shell}
+chroot# dpkg -i linux-image-2.6.18-custom_2.6.18-custom-10.00.Custom_i386.deb
+\end{shell}
+
+\section{Troubleshooting}
+
+Getting
+\begin{shell}
+IP-Config: No network devices available.
+\end{shell}
+messages during the boot (\emph{after} the kernel is successfully loaded!).
+According to \url{http://www.linuxquestions.org/questions/linux-networking-3/ip-config-no-network-device-available-591273/},
+the problem is due to a missing kernel driver.
+
+So I figured out what card I had:
+\begin{shell}
+# lspci
+...
+03:03.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller
+03:04.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller
+...
+\end{shell}
+
+The ethernet HOWTO (\url{http://tldp.org/HOWTO/Ethernet-HOWTO-4.html#ss4.24}) claimed that the |e1000| drivers were required for Intel gigabit cards, and indeed I had the e1000 module mounted on my server:
+\begin{shell}
+# lsmod | less
+...
+e1000                 108480  0 
+...
+\end{shell}
+
+I reconfigured my kernel with (new vs. old):
+\begin{file}[label=diff .config .config\_mod\_e1000]
+3,4c3,4
+< # Linux kernel version: 2.6.18
+< # Fri May 30 22:21:29 2008
+---
+> # Linux kernel version: 2.6.18-custom
+> # Fri May 30 00:13:47 2008
+1542c1542
+< CONFIG_E1000=y
+---
+> CONFIG_E1000=m
+\end{file}
+
+After which I recompiled and reinstalled the kernel as in Appendix \ref{app.kernel}.
+
+
+\subsection{Waiting for /usr/}
+
+On booting a client, I noticed a |Waiting for /usr/: FAILED| message just
+before entering runlevel 2.
+I attribute the error to a faulty boot order on the client not mounting it's
+fstab filesystems before trying to run something in |/usr/|.
+There don't seem to be any serious side effects though, since the wait times
+out, and by the time I can log in to the node, |/usr/| is mounted as it should
+be.
+
+\begin{thebibliography}{99}
+\addcontentsline{toc}{section}{Bibliography}
+
+\bibitem{microwulf}
+  Tim Brom.
+  {\it Microwulf Software and Network Configuration Notes}.
+  May 16, 2008.\\
+  \url{http://www.calvin.edu/~adams/research/microwulf/sys/microwulf_notes.pdf}
+
+\bibitem{debian-nfsroot}
+  Bart Trojanowski.
+  {\it pxeboot and nfsroot with debian}.
+  March 29, 2007.\\
+  \url{http://www.jukie.net/~bart/blog/nfsroot-on-debian}
+
+\bibitem{kernel}
+  Falko Timme.
+  {\it How To Compile A Kernel - Debian Etch}.
+  June 12, 2007.\\
+  \url{http://www.howtoforge.com/kernel_compilation_debian_etch}
+
+\bibitem{install}
+  Debian Installer team.
+  {\it Debian GNU/Linux 4.0 -- Installation Guide}.\\
+  \url{http://www.debian.org/releases/stable/installmanual}
+
+\bibitem{partition}
+  Anthony Lissot et al.
+  {\it Linux Partition HOWTO}.\\
+  |/usr/share/doc/HOWTO/en-txt/Partition.gz|\\
+  \url{http://www.tldp.org/HOWTO/text/Partition}
+
+\bibitem{tasksel}
+  Debian installer appendix with |tasksel| description.
+  {\it D.3. Installing Debian GNU/Linux from a Unix/Linux System} \\
+  \url{http://www.debian.org/releases/stable/i386/apds03.html.en}
+
+\bibitem{net-howto}
+  Joshua Drake.
+  {\it Linux Networking HOWTO}. \\
+  \url{http://www.faqs.org/docs/Linux-HOWTO/Net-HOWTO.html}
+
+\bibitem{pxe}
+  H.\ Peter Anvin.
+  {\it PXELINUX}.\\
+  |/usr/share/doc/syslinux/pxelinux.doc.gz| \\
+  \url{http://syslinux.zytor.com/pxe.php}
+
+\bibitem{nfsroot}
+ Gero Kuhlmann et al.
+ {\it Mounting the root filesystem via NFS (nfsroot)}.
+ 2006.\\
+ |/usr/share/doc/linux-doc-2.6.18/Documentation/nfsroot.txt.gz| in |linux-doc-2.6.18| package\\
+ \url{http://www.kernel.org/doc/Documentation/filesystems/nfsroot.txt}.
+
+\bibitem{initrd}
+  M.\ Tim Jones.
+  {\it Linux initial RAM disk (initrd) overview}.
+  IBM Linux DeveloperWorks:
+  July 31, 2006.\\
+  \url{http://www.ibm.com/developerworks/linux/library/l-initrd.html}
+
+\bibitem{nfsroot-mini}
+  Andreas Kostyrka et al.
+  {\it NFS-Root mini-HOWTO}.
+  September 20, 2002.\\
+  |/usr/share/doc/HOWTO/en-txt/NFS-Root.gz|\\
+  \url{http://www.tldp.org/HOWTO/text/NFS-Root}
+
+
+\end{thebibliography}
+
+\end{document}
diff --git a/posts/Abax/pbs_queues/A_out b/posts/Abax/pbs_queues/A_out
new file mode 100644
index 0000000..bd68091
--- /dev/null
+++ b/posts/Abax/pbs_queues/A_out
@@ -0,0 +1 @@
+we're in
diff --git a/posts/Abax/pbs_queues/B_out b/posts/Abax/pbs_queues/B_out
new file mode 100644
index 0000000..cfc1a1d
--- /dev/null
+++ b/posts/Abax/pbs_queues/B_out
@@ -0,0 +1 @@
+/home/sysadmin/howto/cluster/pbs_queues
diff --git a/posts/Abax/pbs_queues/C_out b/posts/Abax/pbs_queues/C_out
new file mode 100644
index 0000000..3cc67da
--- /dev/null
+++ b/posts/Abax/pbs_queues/C_out
@@ -0,0 +1,2 @@
+we're in
+/home/sysadmin/howto/cluster/pbs_queues
diff --git a/posts/Abax/pbs_queues/Makefile b/posts/Abax/pbs_queues/Makefile
new file mode 100644
index 0000000..25dd052
--- /dev/null
+++ b/posts/Abax/pbs_queues/Makefile
@@ -0,0 +1,30 @@
+# each MetaPost graphic has it's own file,
+# so the basic definitions should get bundled out into an external file
+# (encourages reuse anyway, so it's good for you :p)
+
+all: main.pdf
+
+view: all
+	xpdf main.pdf &
+
+latex-clean:
+	rm -f *.1 *.log *.mp *.mpx *.aux *.out *.cut
+
+semi-clean: latex-clean
+	rm -f command_list *.sh_out *.[eo][0-9]* *_out *_err comE comO
+
+pdf-clean:
+	rm -f *.pdf
+
+clean: semi-clean pdf-clean
+
+%.pdf: %.tex *.tex dummy.sh_out
+	pdflatex $<
+	pdflatex $<
+
+command_list: make_command_list.sh submitting.tex querying.tex altering.tex
+	./$^ > $@
+
+dummy.sh_out: make_com_out.sh command_list
+	./$<
+	echo "dummy dependency for makefile" > $@
diff --git a/posts/Abax/pbs_queues/STDIN.e2683 b/posts/Abax/pbs_queues/STDIN.e2683
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2687 b/posts/Abax/pbs_queues/STDIN.e2687
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2689 b/posts/Abax/pbs_queues/STDIN.e2689
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2690 b/posts/Abax/pbs_queues/STDIN.e2690
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2691 b/posts/Abax/pbs_queues/STDIN.e2691
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2692 b/posts/Abax/pbs_queues/STDIN.e2692
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2693 b/posts/Abax/pbs_queues/STDIN.e2693
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2694 b/posts/Abax/pbs_queues/STDIN.e2694
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2695 b/posts/Abax/pbs_queues/STDIN.e2695
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2696 b/posts/Abax/pbs_queues/STDIN.e2696
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2697 b/posts/Abax/pbs_queues/STDIN.e2697
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2698 b/posts/Abax/pbs_queues/STDIN.e2698
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2699-1 b/posts/Abax/pbs_queues/STDIN.e2699-1
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2699-2 b/posts/Abax/pbs_queues/STDIN.e2699-2
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2699-3 b/posts/Abax/pbs_queues/STDIN.e2699-3
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2699-4 b/posts/Abax/pbs_queues/STDIN.e2699-4
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2699-5 b/posts/Abax/pbs_queues/STDIN.e2699-5
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2700-1 b/posts/Abax/pbs_queues/STDIN.e2700-1
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2700-2 b/posts/Abax/pbs_queues/STDIN.e2700-2
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2700-3 b/posts/Abax/pbs_queues/STDIN.e2700-3
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2700-4 b/posts/Abax/pbs_queues/STDIN.e2700-4
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2700-5 b/posts/Abax/pbs_queues/STDIN.e2700-5
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2701 b/posts/Abax/pbs_queues/STDIN.e2701
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2702 b/posts/Abax/pbs_queues/STDIN.e2702
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2703 b/posts/Abax/pbs_queues/STDIN.e2703
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2704 b/posts/Abax/pbs_queues/STDIN.e2704
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2705 b/posts/Abax/pbs_queues/STDIN.e2705
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2709 b/posts/Abax/pbs_queues/STDIN.e2709
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2711 b/posts/Abax/pbs_queues/STDIN.e2711
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2712 b/posts/Abax/pbs_queues/STDIN.e2712
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2713 b/posts/Abax/pbs_queues/STDIN.e2713
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2714 b/posts/Abax/pbs_queues/STDIN.e2714
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2715 b/posts/Abax/pbs_queues/STDIN.e2715
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2716 b/posts/Abax/pbs_queues/STDIN.e2716
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2717 b/posts/Abax/pbs_queues/STDIN.e2717
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2718 b/posts/Abax/pbs_queues/STDIN.e2718
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2719 b/posts/Abax/pbs_queues/STDIN.e2719
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2720 b/posts/Abax/pbs_queues/STDIN.e2720
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2721-1 b/posts/Abax/pbs_queues/STDIN.e2721-1
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2721-2 b/posts/Abax/pbs_queues/STDIN.e2721-2
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2721-3 b/posts/Abax/pbs_queues/STDIN.e2721-3
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2721-4 b/posts/Abax/pbs_queues/STDIN.e2721-4
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2721-5 b/posts/Abax/pbs_queues/STDIN.e2721-5
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2722-1 b/posts/Abax/pbs_queues/STDIN.e2722-1
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2722-2 b/posts/Abax/pbs_queues/STDIN.e2722-2
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2722-3 b/posts/Abax/pbs_queues/STDIN.e2722-3
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2722-4 b/posts/Abax/pbs_queues/STDIN.e2722-4
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2722-5 b/posts/Abax/pbs_queues/STDIN.e2722-5
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2723 b/posts/Abax/pbs_queues/STDIN.e2723
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2724 b/posts/Abax/pbs_queues/STDIN.e2724
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2725 b/posts/Abax/pbs_queues/STDIN.e2725
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2726 b/posts/Abax/pbs_queues/STDIN.e2726
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.e2727 b/posts/Abax/pbs_queues/STDIN.e2727
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.o2683 b/posts/Abax/pbs_queues/STDIN.o2683
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2683
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2687 b/posts/Abax/pbs_queues/STDIN.o2687
new file mode 100644
index 0000000..e229dd4
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2687
@@ -0,0 +1,2 @@
+/home/sysadmin
+/home/sysadmin/howto/cluster/pbs_queues
diff --git a/posts/Abax/pbs_queues/STDIN.o2688 b/posts/Abax/pbs_queues/STDIN.o2688
new file mode 100644
index 0000000..9883ddc
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2688
@@ -0,0 +1,5 @@
+Running a job...
+
+real	0m30.010s
+user	0m0.000s
+sys	0m0.000s
diff --git a/posts/Abax/pbs_queues/STDIN.o2689 b/posts/Abax/pbs_queues/STDIN.o2689
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2689
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2690 b/posts/Abax/pbs_queues/STDIN.o2690
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.o2691 b/posts/Abax/pbs_queues/STDIN.o2691
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.o2693 b/posts/Abax/pbs_queues/STDIN.o2693
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2693
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2694 b/posts/Abax/pbs_queues/STDIN.o2694
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2694
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2695 b/posts/Abax/pbs_queues/STDIN.o2695
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2695
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2696 b/posts/Abax/pbs_queues/STDIN.o2696
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2696
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2697 b/posts/Abax/pbs_queues/STDIN.o2697
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2697
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2698 b/posts/Abax/pbs_queues/STDIN.o2698
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2698
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2699-1 b/posts/Abax/pbs_queues/STDIN.o2699-1
new file mode 100644
index 0000000..c07fd2c
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2699-1
@@ -0,0 +1 @@
+Running job $PBS_ARRAYID...
diff --git a/posts/Abax/pbs_queues/STDIN.o2699-2 b/posts/Abax/pbs_queues/STDIN.o2699-2
new file mode 100644
index 0000000..c07fd2c
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2699-2
@@ -0,0 +1 @@
+Running job $PBS_ARRAYID...
diff --git a/posts/Abax/pbs_queues/STDIN.o2699-3 b/posts/Abax/pbs_queues/STDIN.o2699-3
new file mode 100644
index 0000000..c07fd2c
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2699-3
@@ -0,0 +1 @@
+Running job $PBS_ARRAYID...
diff --git a/posts/Abax/pbs_queues/STDIN.o2699-4 b/posts/Abax/pbs_queues/STDIN.o2699-4
new file mode 100644
index 0000000..c07fd2c
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2699-4
@@ -0,0 +1 @@
+Running job $PBS_ARRAYID...
diff --git a/posts/Abax/pbs_queues/STDIN.o2699-5 b/posts/Abax/pbs_queues/STDIN.o2699-5
new file mode 100644
index 0000000..c07fd2c
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2699-5
@@ -0,0 +1 @@
+Running job $PBS_ARRAYID...
diff --git a/posts/Abax/pbs_queues/STDIN.o2700-1 b/posts/Abax/pbs_queues/STDIN.o2700-1
new file mode 100644
index 0000000..cfc1a1d
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2700-1
@@ -0,0 +1 @@
+/home/sysadmin/howto/cluster/pbs_queues
diff --git a/posts/Abax/pbs_queues/STDIN.o2700-2 b/posts/Abax/pbs_queues/STDIN.o2700-2
new file mode 100644
index 0000000..cfc1a1d
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2700-2
@@ -0,0 +1 @@
+/home/sysadmin/howto/cluster/pbs_queues
diff --git a/posts/Abax/pbs_queues/STDIN.o2700-3 b/posts/Abax/pbs_queues/STDIN.o2700-3
new file mode 100644
index 0000000..cfc1a1d
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2700-3
@@ -0,0 +1 @@
+/home/sysadmin/howto/cluster/pbs_queues
diff --git a/posts/Abax/pbs_queues/STDIN.o2700-4 b/posts/Abax/pbs_queues/STDIN.o2700-4
new file mode 100644
index 0000000..cfc1a1d
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2700-4
@@ -0,0 +1 @@
+/home/sysadmin/howto/cluster/pbs_queues
diff --git a/posts/Abax/pbs_queues/STDIN.o2700-5 b/posts/Abax/pbs_queues/STDIN.o2700-5
new file mode 100644
index 0000000..cfc1a1d
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2700-5
@@ -0,0 +1 @@
+/home/sysadmin/howto/cluster/pbs_queues
diff --git a/posts/Abax/pbs_queues/STDIN.o2702 b/posts/Abax/pbs_queues/STDIN.o2702
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2702
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2703 b/posts/Abax/pbs_queues/STDIN.o2703
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2703
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2704 b/posts/Abax/pbs_queues/STDIN.o2704
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2704
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2705 b/posts/Abax/pbs_queues/STDIN.o2705
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2705
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2709 b/posts/Abax/pbs_queues/STDIN.o2709
new file mode 100644
index 0000000..e229dd4
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2709
@@ -0,0 +1,2 @@
+/home/sysadmin
+/home/sysadmin/howto/cluster/pbs_queues
diff --git a/posts/Abax/pbs_queues/STDIN.o2710 b/posts/Abax/pbs_queues/STDIN.o2710
new file mode 100644
index 0000000..7040cea
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2710
@@ -0,0 +1,5 @@
+Running a job...
+
+real	0m30.013s
+user	0m0.000s
+sys	0m0.000s
diff --git a/posts/Abax/pbs_queues/STDIN.o2711 b/posts/Abax/pbs_queues/STDIN.o2711
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2711
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2712 b/posts/Abax/pbs_queues/STDIN.o2712
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.o2713 b/posts/Abax/pbs_queues/STDIN.o2713
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/STDIN.o2715 b/posts/Abax/pbs_queues/STDIN.o2715
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2715
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2716 b/posts/Abax/pbs_queues/STDIN.o2716
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2716
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2717 b/posts/Abax/pbs_queues/STDIN.o2717
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2717
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2718 b/posts/Abax/pbs_queues/STDIN.o2718
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2718
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2719 b/posts/Abax/pbs_queues/STDIN.o2719
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2719
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2720 b/posts/Abax/pbs_queues/STDIN.o2720
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2720
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2721-1 b/posts/Abax/pbs_queues/STDIN.o2721-1
new file mode 100644
index 0000000..c07fd2c
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2721-1
@@ -0,0 +1 @@
+Running job $PBS_ARRAYID...
diff --git a/posts/Abax/pbs_queues/STDIN.o2721-2 b/posts/Abax/pbs_queues/STDIN.o2721-2
new file mode 100644
index 0000000..c07fd2c
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2721-2
@@ -0,0 +1 @@
+Running job $PBS_ARRAYID...
diff --git a/posts/Abax/pbs_queues/STDIN.o2721-3 b/posts/Abax/pbs_queues/STDIN.o2721-3
new file mode 100644
index 0000000..c07fd2c
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2721-3
@@ -0,0 +1 @@
+Running job $PBS_ARRAYID...
diff --git a/posts/Abax/pbs_queues/STDIN.o2721-4 b/posts/Abax/pbs_queues/STDIN.o2721-4
new file mode 100644
index 0000000..c07fd2c
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2721-4
@@ -0,0 +1 @@
+Running job $PBS_ARRAYID...
diff --git a/posts/Abax/pbs_queues/STDIN.o2721-5 b/posts/Abax/pbs_queues/STDIN.o2721-5
new file mode 100644
index 0000000..c07fd2c
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2721-5
@@ -0,0 +1 @@
+Running job $PBS_ARRAYID...
diff --git a/posts/Abax/pbs_queues/STDIN.o2722-1 b/posts/Abax/pbs_queues/STDIN.o2722-1
new file mode 100644
index 0000000..cfc1a1d
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2722-1
@@ -0,0 +1 @@
+/home/sysadmin/howto/cluster/pbs_queues
diff --git a/posts/Abax/pbs_queues/STDIN.o2722-2 b/posts/Abax/pbs_queues/STDIN.o2722-2
new file mode 100644
index 0000000..cfc1a1d
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2722-2
@@ -0,0 +1 @@
+/home/sysadmin/howto/cluster/pbs_queues
diff --git a/posts/Abax/pbs_queues/STDIN.o2722-3 b/posts/Abax/pbs_queues/STDIN.o2722-3
new file mode 100644
index 0000000..cfc1a1d
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2722-3
@@ -0,0 +1 @@
+/home/sysadmin/howto/cluster/pbs_queues
diff --git a/posts/Abax/pbs_queues/STDIN.o2722-4 b/posts/Abax/pbs_queues/STDIN.o2722-4
new file mode 100644
index 0000000..cfc1a1d
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2722-4
@@ -0,0 +1 @@
+/home/sysadmin/howto/cluster/pbs_queues
diff --git a/posts/Abax/pbs_queues/STDIN.o2722-5 b/posts/Abax/pbs_queues/STDIN.o2722-5
new file mode 100644
index 0000000..cfc1a1d
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2722-5
@@ -0,0 +1 @@
+/home/sysadmin/howto/cluster/pbs_queues
diff --git a/posts/Abax/pbs_queues/STDIN.o2724 b/posts/Abax/pbs_queues/STDIN.o2724
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2724
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2725 b/posts/Abax/pbs_queues/STDIN.o2725
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2725
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2726 b/posts/Abax/pbs_queues/STDIN.o2726
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2726
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/STDIN.o2727 b/posts/Abax/pbs_queues/STDIN.o2727
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/STDIN.o2727
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/altering.tex b/posts/Abax/pbs_queues/altering.tex
new file mode 100644
index 0000000..a4d8d59
--- /dev/null
+++ b/posts/Abax/pbs_queues/altering.tex
@@ -0,0 +1,14 @@
+\section{Altering and deleting jobs}
+\label{sec.qalter}
+
+Minor glitches in submitted jobs can be fixed by altering the job with |qalter|.
+For example, incorrect dependencies may be causing a job to hold in the queue forever.
+We can remove these invalid holds with
+\qcom{qalter_depend_echo.sh}
+|qalter| is a Swiss-army-knife-style command, since it can change many aspects of a job.
+The specific hold-release case above could also have been handled with the |qrls| command.
+There are a number of |q*| commands which provide a 
+
+If you decide a job is beyond repair, you can kill it with |qdel|.
+For obvious reasons, you can only kill your own jobs, unless your an administrator.
+\qcom{qdel_echo.sh}
diff --git a/posts/Abax/pbs_queues/array_depend_echo.sh b/posts/Abax/pbs_queues/array_depend_echo.sh
new file mode 100755
index 0000000..db833a3
--- /dev/null
+++ b/posts/Abax/pbs_queues/array_depend_echo.sh
@@ -0,0 +1,16 @@
+JOBID=`echo "cd \\\$PBS_O_WORKDIR && sleep 30 && pwd && echo 1 > val\\\${PBS_ARRAYID}_out" | qsub -t 1-5`
+sleep 2 # give the Job a second to load in...
+#echo "cd \\\$PBS_O_WORKDIR && awk 'START{s=0}{s+=\$0}END{print s}' val*_out" | \
+#    qsub -o sum_out -W depend=afterany:$JOBID
+JOBNUM=`echo $JOBID | cut -d. -f1`
+#echo "cd \\\$PBS_O_WORKDIR && awk 'START{s=0}{s+=\$0}END{print s}' val*_out" | \
+#    qsub -o sum_out -W depend=afterany:$JOBNUM
+COND="depend=afterany"
+i=1
+while [ $i -le 5 ]
+do
+  COND="$COND:$JOBNUM-$i"
+  let "i += 1"
+done
+echo "cd \$PBS_O_WORKDIR && awk 'START{s=0}{s+=\$0}END{print s}' val*_out" | \
+    qsub -o sum_out -W $COND
diff --git a/posts/Abax/pbs_queues/array_depend_echo.sh_out b/posts/Abax/pbs_queues/array_depend_echo.sh_out
new file mode 100644
index 0000000..6a8eabd
--- /dev/null
+++ b/posts/Abax/pbs_queues/array_depend_echo.sh_out
@@ -0,0 +1,17 @@
+$ JOBID=`echo "cd \\\$PBS_O_WORKDIR && sleep 30 && pwd && echo 1 > val\\\${PBS_ARRAYID}_out" | qsub -t 1-5`
+$ sleep 2 # give the Job a second to load in...
+$ #echo "cd \\\$PBS_O_WORKDIR && awk 'START{s=0}{s+=\$0}END{print s}' val*_out" | \
+$ #    qsub -o sum_out -W depend=afterany:$JOBID
+$ JOBNUM=`echo $JOBID | cut -d. -f1`
+$ #echo "cd \\\$PBS_O_WORKDIR && awk 'START{s=0}{s+=\$0}END{print s}' val*_out" | \
+$ #    qsub -o sum_out -W depend=afterany:$JOBNUM
+$ COND="depend=afterany"
+$ i=1
+$ while [ $i -le 5 ]
+$ do
+>   COND="$COND:$JOBNUM-$i"
+>   let "i += 1"
+> done
+$ echo "cd \$PBS_O_WORKDIR && awk 'START{s=0}{s+=\$0}END{print s}' val*_out" | \
+>     qsub -o sum_out -W $COND
+2723.n0.physics.drexel.edu
diff --git a/posts/Abax/pbs_queues/array_depend_echo_out_err b/posts/Abax/pbs_queues/array_depend_echo_out_err
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/array_depend_echo_out_out b/posts/Abax/pbs_queues/array_depend_echo_out_out
new file mode 100644
index 0000000..bdd1cff
--- /dev/null
+++ b/posts/Abax/pbs_queues/array_depend_echo_out_out
@@ -0,0 +1,2 @@
+$ cat sum_out
+5
diff --git a/posts/Abax/pbs_queues/array_echo.sh b/posts/Abax/pbs_queues/array_echo.sh
new file mode 100755
index 0000000..385cefb
--- /dev/null
+++ b/posts/Abax/pbs_queues/array_echo.sh
@@ -0,0 +1 @@
+echo "sleep 30 && echo 'Running job \$PBS_ARRAYID...'" | qsub -t 1-5
diff --git a/posts/Abax/pbs_queues/array_echo.sh_out b/posts/Abax/pbs_queues/array_echo.sh_out
new file mode 100644
index 0000000..2f2e6c4
--- /dev/null
+++ b/posts/Abax/pbs_queues/array_echo.sh_out
@@ -0,0 +1,2 @@
+$ echo "sleep 30 && echo 'Running job \$PBS_ARRAYID...'" | qsub -t 1-5
+2721.n0.physics.drexel.edu
diff --git a/posts/Abax/pbs_queues/command_list b/posts/Abax/pbs_queues/command_list
new file mode 100644
index 0000000..23274da
--- /dev/null
+++ b/posts/Abax/pbs_queues/command_list
@@ -0,0 +1,45 @@
+QCOM simple_echo.sh
+COM qstat_simple_echo qstat
+QCOM simple_script.sh
+sleep 40
+QCOM named_echo.sh
+COM qstat_named_echo qstat
+QCOM explicit_oe_echo.sh
+sleep 2
+COM explicit_oe_echo_o cat my_out
+COM explicit_oe_echo_e cat my_err
+sleep 40
+QCOM workdir_echo.sh
+sleep 2
+COMOUT
+QCOM get_walltime_echo.sh
+sleep 40
+COMOUT
+QCOM set_walltime_echo.sh
+sleep 2
+COM qstat_set_walltime_echo qstat -f | grep '[.]walltime'
+sleep 38
+QCOM depend_echo.sh
+sleep 2
+COM qstat_depend_echo qstat
+sleep 50
+COM depend_echo_out cat C_out
+QCOM while_echo.sh
+sleep 10
+QCOM array_echo.sh
+sleep 2
+COM qstat_array_echo qstat
+sleep 50
+QCOM array_depend_echo.sh
+sleep 2
+COM qstat_array_depend_echo qstat
+sleep 60
+COM array_depend_echo_out cat sum_out
+QCOM qstat_particular_echo.sh
+sleep 40
+QCOM qstat_full_echo.sh
+QCOM tracejob_echo.sh
+COM qstat_queue_info qstat -q
+COM qstat_server_info qstat -B
+QCOM qalter_depend_echo.sh
+QCOM qdel_echo.sh
diff --git a/posts/Abax/pbs_queues/depend_echo.sh b/posts/Abax/pbs_queues/depend_echo.sh
new file mode 100755
index 0000000..53504e9
--- /dev/null
+++ b/posts/Abax/pbs_queues/depend_echo.sh
@@ -0,0 +1,6 @@
+AID=`echo "cd \\\$PBS_O_WORKDIR && sleep 30 && echo \"we're in\" > A_out" | qsub`
+BID=`echo "cd \\\$PBS_O_WORKDIR && sleep 30 && pwd > B_out" | qsub`
+COND="depend=afterany:$AID:$BID -o C_out -W depend=afterany:$AID:$BID"
+CID=`echo "cd \\\$PBS_O_WORKDIR && cat A_out B_out" | \
+        qsub -W depend=afterany:$AID:$BID -o C_out`
+echo -e "A: $AID\nB: $BID\nC: $CID"
diff --git a/posts/Abax/pbs_queues/depend_echo.sh_out b/posts/Abax/pbs_queues/depend_echo.sh_out
new file mode 100644
index 0000000..4d270a6
--- /dev/null
+++ b/posts/Abax/pbs_queues/depend_echo.sh_out
@@ -0,0 +1,9 @@
+$ AID=`echo "cd \\\$PBS_O_WORKDIR && sleep 30 && echo \"we're in\" > A_out" | qsub`
+$ BID=`echo "cd \\\$PBS_O_WORKDIR && sleep 30 && pwd > B_out" | qsub`
+$ COND="depend=afterany:$AID:$BID -o C_out -W depend=afterany:$AID:$BID"
+$ CID=`echo "cd \\\$PBS_O_WORKDIR && cat A_out B_out" | \
+>         qsub -W depend=afterany:$AID:$BID -o C_out`
+$ echo -e "A: $AID\nB: $BID\nC: $CID"
+A: 2712.n0.physics.drexel.edu
+B: 2713.n0.physics.drexel.edu
+C: 2714.n0.physics.drexel.edu
diff --git a/posts/Abax/pbs_queues/depend_echo_out_err b/posts/Abax/pbs_queues/depend_echo_out_err
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/depend_echo_out_out b/posts/Abax/pbs_queues/depend_echo_out_out
new file mode 100644
index 0000000..3912622
--- /dev/null
+++ b/posts/Abax/pbs_queues/depend_echo_out_out
@@ -0,0 +1,3 @@
+$ cat C_out
+we're in
+/home/sysadmin/howto/cluster/pbs_queues
diff --git a/posts/Abax/pbs_queues/dummy.sh_out b/posts/Abax/pbs_queues/dummy.sh_out
new file mode 100644
index 0000000..c22f616
--- /dev/null
+++ b/posts/Abax/pbs_queues/dummy.sh_out
@@ -0,0 +1 @@
+dummy dependency for makefile
diff --git a/posts/Abax/pbs_queues/echo_script.sh b/posts/Abax/pbs_queues/echo_script.sh
new file mode 100755
index 0000000..63e0cfa
--- /dev/null
+++ b/posts/Abax/pbs_queues/echo_script.sh
@@ -0,0 +1,7 @@
+#!/bin/bash
+# file: echo_script.sh
+sleep 30
+echo "a really,"
+echo "really,"
+echo "complicated"
+echo "script"
diff --git a/posts/Abax/pbs_queues/echo_script.sh.e2684 b/posts/Abax/pbs_queues/echo_script.sh.e2684
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/echo_script.sh.e2706 b/posts/Abax/pbs_queues/echo_script.sh.e2706
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/echo_script.sh.o2684 b/posts/Abax/pbs_queues/echo_script.sh.o2684
new file mode 100644
index 0000000..d607e2e
--- /dev/null
+++ b/posts/Abax/pbs_queues/echo_script.sh.o2684
@@ -0,0 +1,4 @@
+a really,
+really,
+complicated
+script
diff --git a/posts/Abax/pbs_queues/echo_script.sh.o2706 b/posts/Abax/pbs_queues/echo_script.sh.o2706
new file mode 100644
index 0000000..d607e2e
--- /dev/null
+++ b/posts/Abax/pbs_queues/echo_script.sh.o2706
@@ -0,0 +1,4 @@
+a really,
+really,
+complicated
+script
diff --git a/posts/Abax/pbs_queues/example1.tex b/posts/Abax/pbs_queues/example1.tex
new file mode 100644
index 0000000..94cbeb8
--- /dev/null
+++ b/posts/Abax/pbs_queues/example1.tex
@@ -0,0 +1,30 @@
+\section{Example 1: jobs depending on jobs}
+\label{sec.example1}
+
+As an example of a medium-complexity compound job, consider my single-proccessor Monte Carlo with two layers of post processing batch submission script.
+The simulation program is the compiled binary |sawsim| which models the unfolding of a multi-domain protein.
+It takes command line arguments specifying model parameters and experimental conditions and prints the forces at which the various domains unfold to |stdout|.
+I want to run with a certain set of parameters, but sweep one of the parameters (pulling velocity) through a number of different values.
+For each basic-parameter--pulling-velocity combination, I want to run the simulation 400 times, to get statistically significant output.
+The first stage of the post-processing gathers the output of all the |sawsim| jobs into a single list of unfolding forces at each velocity.
+The second stage of the post-processing generates histograms of unfolding forces for each velocity and a file with pulling speed vs.\ unfolding force data for plotting.
+\inputfile{run_vel_dist.sh}{run\_vel\_dist.sh}
+We start off by generating and |cd|ing into a new directory, since we'll be generating \emph{lots} of files.
+Then we loop through our velocities, creating a new job array (|sawsim|) executing |sawsim| with the appropriate parameters.
+
+Job arrays are still young in Torque (our batch queue manager), and we can't depend on the job array as a whole.
+Because the number of jobs we can depend on simultaneously is limited (command line size?), we bundle several |sawsim| executions into a single array job instance.
+This allows us to keep the number of executions up without generating huge dependency lists for the analysis jobs.
+We add job array with a user hold (|-h|) to ensure that the jobs don't begin executing immediately and exit before we start our analysis jobs.
+We also sleep for a few seconds after adding the job, because sometimes the array takes a bit to `deploy' into the queue (symptom: out-of-order job IDs in the queue).
+
+The first postprocessing stage (|sawComp|) gathers the output from the array job for it's particular velocity, strips comment lines, and dumps the output into a central file (|data\_\$V|).
+We set this job to depend on each of the array job instances at that velocity (since we can't depend on the entire array job).
+We also start \emph{this} job on hold, so that it doesn't exit before we add the second layer analysis job.
+After adding the job, we release the user holds on the array job instances to let them get going (another place where access by array job ID would be nice\ldots).
+
+The final analysis stage involves enough steps that it has it's own script, but nothing in that script involves the batch queue system, so it is not reproduced here.
+We depend on all the first stage analysis jobs, since we need the consolidated |data\_\$V| files for the script.
+There is no need to hold this job, since nothing depends on it.
+After adding the job, we release the \emph{user} holds on the first-stage analysis jobs.
+The \emph{system} holds due to their dependencies remain until they are satisfied as the |sawsim| jobs complete.
diff --git a/posts/Abax/pbs_queues/explicit_oe_echo.sh b/posts/Abax/pbs_queues/explicit_oe_echo.sh
new file mode 100755
index 0000000..b5e7311
--- /dev/null
+++ b/posts/Abax/pbs_queues/explicit_oe_echo.sh
@@ -0,0 +1 @@
+echo "echo 'ABC' && echo 'DEF' > /dev/stderr" | qsub -o my_out -e my_err
diff --git a/posts/Abax/pbs_queues/explicit_oe_echo.sh_out b/posts/Abax/pbs_queues/explicit_oe_echo.sh_out
new file mode 100644
index 0000000..fa2fe43
--- /dev/null
+++ b/posts/Abax/pbs_queues/explicit_oe_echo.sh_out
@@ -0,0 +1,2 @@
+$ echo "echo 'ABC' && echo 'DEF' > /dev/stderr" | qsub -o my_out -e my_err
+2708.n0.physics.drexel.edu
diff --git a/posts/Abax/pbs_queues/explicit_oe_echo_e_err b/posts/Abax/pbs_queues/explicit_oe_echo_e_err
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/explicit_oe_echo_e_out b/posts/Abax/pbs_queues/explicit_oe_echo_e_out
new file mode 100644
index 0000000..4fde57f
--- /dev/null
+++ b/posts/Abax/pbs_queues/explicit_oe_echo_e_out
@@ -0,0 +1,2 @@
+$ cat my_err
+DEF
diff --git a/posts/Abax/pbs_queues/explicit_oe_echo_o_err b/posts/Abax/pbs_queues/explicit_oe_echo_o_err
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/explicit_oe_echo_o_out b/posts/Abax/pbs_queues/explicit_oe_echo_o_out
new file mode 100644
index 0000000..15ad8fe
--- /dev/null
+++ b/posts/Abax/pbs_queues/explicit_oe_echo_o_out
@@ -0,0 +1,2 @@
+$ cat my_out
+ABC
diff --git a/posts/Abax/pbs_queues/file/_procmailrc b/posts/Abax/pbs_queues/file/_procmailrc
new file mode 100644
index 0000000..5e58a23
--- /dev/null
+++ b/posts/Abax/pbs_queues/file/_procmailrc
@@ -0,0 +1,27 @@
+# .procmailrc
+#
+# log recipt of mail from PBS queue administrator from your system mailbox
+#
+# see man procmail, procmailrc, and procmailex
+# there is also a good procmail tutorial at
+# http://userpages.umbc.edu/~ian/procmail.html
+#
+# procmail uses sed-like regexps
+
+DEFAULT=$MAIL
+MAILDIR=$HOME/.mailspool
+LOGFILE=$MAILDIR/mailfilter/procmailrc.log
+SHELL=/bin/bash
+COMPLETED=$MAILDIR/completed
+
+# log job id's and exit status for anything satifying ALL of 
+# From adm@abax.physics.drexel.edu,
+# Envelope-to: sysadmin@n0.abax.physics.drexel.edu
+# To: sysadmin@n0.abax.physics.drexel.edu
+# Subject: PBS JOB [0-9]*.n0.abax.physics.drexel.edu
+# lock the mail file while the processing is being carried out
+:0 b:    # b only use the body, (c pass copy to other rules), : use a lockfile
+* ^From adm@abax[.]physics[.]drexel[.]edu
+* ^To: sysadmin@n0[.]abax[.]physics[.]drexel[.]edu
+* ^Subject: PBS JOB [0-9]*[.]n0[.]abax[.]physics[.]drexel[.]edu
+| cat >> $COMPLETED
diff --git a/posts/Abax/pbs_queues/file/dup_env b/posts/Abax/pbs_queues/file/dup_env
new file mode 100755
index 0000000..9932428
--- /dev/null
+++ b/posts/Abax/pbs_queues/file/dup_env
@@ -0,0 +1,22 @@
+#!/bin/bash
+#
+# override Torque qsub's strange environment shelving behavior by copying all
+# your precious environmental variables back to where they should be.
+#
+# usage: source dup_env
+
+while read VARSTR
+  do
+  VAR=${VARSTR%=*} # strip everything after & including the =
+  VAL=${VARSTR#$VAR=} # strip everying up & including to the =
+  #echo "parsed '$VARSTR' to '$VAR' & '$VAL'"
+  if [ "${VAR:0:6}" == "PBS_O_" ]
+      then
+      ORIGVAR=${VAR#PBS_O_} # strip PBS_O_ from $VAR
+      if [ "${!ORIGVAR}" != "$VAL" ] && [ "$ORIGVAR" != "HOST" ]
+	  then
+	  #echo "changing $ORIGVAR from '${!ORIGVAR}' to '$VAL'"
+	  export $ORIGVAR=$VAL
+      fi
+  fi
+done < <(printenv)
diff --git a/posts/Abax/pbs_queues/file/qcleanmail b/posts/Abax/pbs_queues/file/qcleanmail
new file mode 100755
index 0000000..a0f686b
--- /dev/null
+++ b/posts/Abax/pbs_queues/file/qcleanmail
@@ -0,0 +1,47 @@
+#!/bin/bash
+#
+# Running my qcmd implementation on your system mailbox can quickly fill it up
+# with PBS junk.  This scans through the box and deletes any messages from the
+# PBS admin.
+#
+# from man procmail:
+# Procmail can also be invoked to postprocess an already filled system mailbox.
+# ...
+#
+# This script is almost verbatim from 'man procmail'
+
+ORIGMAILS="$MAIL $HOME/.mailspool/completed" 
+FILTER=$HOME/.mailspool/mailfilter/qcleanmail.proc
+
+clean ()
+{
+    ORIGMAIL=$1
+    if cd $HOME && 
+	test -s $ORIGMAIL &&
+	lockfile -r0 -l1024 .newmail.lock 2>/dev/null
+	then
+	umask 077
+	trap "rm -f .newmail.lock" 1 2 3 13 15
+	echo "cleaning $ORIGMAIL" 
+        # lock the system maildir, and extract all the messages
+	lockfile -l1024 -ml
+	cat $ORIGMAIL >>.newmail &&
+	cat /dev/null >$ORIGMAIL
+	lockfile -mu
+        # lock released, now new messages can arrive as they normally do
+	
+        # process the mail we copied out with our filter
+        # mail that should be in $ORIGMAIL will be appended as procmail notices...
+	formail -s procmail $FILTER <.newmail &&
+	  rm -f .newmail
+	  rm -f .newmail.lock
+    fi
+    return 0
+}
+
+for M in $ORIGMAILS
+  do
+  clean $M
+done
+
+exit 0
diff --git a/posts/Abax/pbs_queues/file/qcleanmail.proc b/posts/Abax/pbs_queues/file/qcleanmail.proc
new file mode 100644
index 0000000..645f2e9
--- /dev/null
+++ b/posts/Abax/pbs_queues/file/qcleanmail.proc
@@ -0,0 +1,27 @@
+# qcleanmail.proc
+#
+# remove mail from PBS queue administrator from your system mailbox
+#
+# see man procmail, procmailrc, and procmailex
+# there is also a good procmail tutorial at
+# http://userpages.umbc.edu/~ian/procmail.html
+#
+# procmail uses sed-like regexps
+
+MAILDIR=$HOME/.mailspool   #$MAIL
+LOGFILE=$MAILDIR/mailfilter/qcleanmail.log
+SHELL=/bin/bash
+HOST=`hostname -f`
+PBS_ADM=adm@abax[.]physics[.]drexel[.]edu
+
+# trash anything from satifying ALL of 
+# From adm@abax.physics.drexel.edu,
+# Envelope-to: $USER@$HOST
+# To: $USER@$HOST
+# Subject: PBS JOB [0-9]*.$HOST
+# lock the mail file while the processing is being carried out
+:0:                                     # the last colon means use a lockfile
+* ^From $PBS_ADM
+* ^To: $USER@$HOST
+* ^Subject: PBS JOB [0-9]*$HOST
+/dev/null                               # write matching messages to /dev/null
diff --git a/posts/Abax/pbs_queues/file/qcmd b/posts/Abax/pbs_queues/file/qcmd
new file mode 100755
index 0000000..b5438fe
--- /dev/null
+++ b/posts/Abax/pbs_queues/file/qcmd
@@ -0,0 +1,221 @@
+#!/bin/bash
+#
+# Run commands through a PBS queue without having to remember anything. :p
+#
+# This script keeps stdin and stdout open until the PBS job returns, monitoring
+# your email for the appropriate job-completion message from the PBS server.
+# This lets you bump a long-executing task onto a compute node, while keeping
+# the lightweight part of your processing on the master node, for easier
+# connection and dependency checking.
+#
+# Downsides to the current implementation:
+# Blocks on email with a subshell (the piped input while loop) and tail
+# running, so counting the script itself, that's 3 processes on the head node
+# for each PBS job you're waiting for.  While that's not a really big deal for
+# a few dozen jobs, it eats up memory fairly quickly for thousands of jobs.
+# 
+# Another drawback is that you can only append to your $MAIL file while this is
+# running.  I'm using a .procmailrc file to deflect PBS-related mail to a
+# seperate mailbox, so I can still edit my system mailbox while this script is
+# running, but that's one more thing you'd have to set up...
+#
+# Yet another drawback is that you can't run qsub from a screen session, but
+# that is just qsub in general.  Sigh.
+#
+# The solution to the memory problem is to have a single script handle all
+# the spawning and waiting for multiple commands.  Take a look at 'qcmds'
+#
+# usage: qcmd [-w[[H:]M:]S] command [arguments]
+# where -w optionally sets the wall time
+# for example:
+#   $ ls .
+#   file
+#   $ qcmd cp file file2 && ls .
+#   file
+#   file2
+#   $ diff file file2
+#   (none)
+# or
+#   $ qcmd pwd '&&' echo \$PBS_O_WORKDIR
+#   /home/sysadmin/script
+#   /home/sysadmin/script
+# and to prove we're running through qsub (2nd hostname is on calling system)
+#   $ qcmd hostname && hostname
+#   n8
+#   n0
+#
+# Warning:
+# This script uses process substitution which is a non-POSIX bash feature.
+#
+# command line arguments
+#          |
+#          v
+#      addjob() ---(submit job with qsub)---> Job-Queue
+#          |                                      |
+#          +----+                        (job-complete email)
+#               |                                 |
+#       while:  v                                 |
+#         checkfornew()                           v
+#         getnextmail() <---(tail -f $MAIL)--- Mailbox
+
+DEBUG=0
+MAIL=$HOME/.mailspool/completed
+# I have a ~/.procmailrc filter forwarding my PBS mail bodies to this $MAIL
+# If you don't, comment the line out so you monitor your system $MAIL.
+
+JOB_OUTSTANDING=  # Store the job id of our uncompleted job
+MAIL_BODY=""      # Store the return of getnextmail()
+
+if [ $DEBUG -eq 1 ]
+    then
+    DEBUGFILE=qcmd.$$
+    > $DEBUGFILE # clear $DEBUGFILE if it existed before (Warning: clobber)
+else
+    DEBUGFILE=/dev/null
+fi
+
+# functions for job spawning
+
+addjob ()
+{
+    CMMD=$*
+    echo running: $CMMD >> $DEBUGFILE
+    SCRIPT="cd \$PBS_O_WORKDIR && source dup_env && $CMMD"
+    # dup_env is in ~/bin even though it's a script
+    # since qsub creates it's own evironment, and moves X -> PBS_O_X
+    # for example, PATH -> PBS_O_PATH.
+    # who knows why it does this...
+    JOBID=`echo $SCRIPT | qsub -mae $WALLTIME_OPTION || exit 1`
+    # -mae : Send mail on abort or execute
+    #JOBNAME=STDIN # the qsub default for scripts piped into qsub
+    echo spawner: started $JOBID >> $DEBUGFILE
+    JOB_OUTSTANDING="$JOBID"
+    echo "add new depend:  $JOBID" >> $DEBUGFILE
+    return 0
+}
+
+# functions for the job checking loop
+
+# look for completion message bodies along the lines of:
+# PBS Job Id: 206.n0.abax.physics.drexel.edu
+# Job Name:   STDIN
+# ...
+#                                 <-- blank line
+#
+# could also poll on ls.
+# neither 'ls' or 'tail -f' busy loops seem to take measurable processor time.
+getnextmail() # blocking, fd 3 is tail -f $MAIL output
+{
+    BODY=""
+    DONE=0
+    INJOB=0
+    echo "block on mail" >> $DEBUGFILE
+    while [ $DONE -eq 0 ] && read LINE <&3
+      do
+      if [ "${LINE:0:11}" == "PBS Job Id:" ]
+	  then
+          # we're reading an email about a job.
+	  #echo "in a job" >> $DEBUGFILE
+	  INJOB=1
+      fi
+      if [ $INJOB -eq 1 ]
+	  then
+	  #echo "getting mail: $LINE" >> $DEBUGFILE
+	  if [ "${#LINE}" -eq 0 ]
+	      then
+              #echo "matched blank line" # we're leaving the email about our job.
+	      #echo "got mail" >> $DEBUGFILE
+	      DONE=1
+	      break
+	  fi
+	  BODY=`echo -e "$BODY\n$LINE"`
+      fi
+    done
+    echo "returning mail" >> $DEBUGFILE
+    MAIL_BODY="$BODY"
+    return 0
+}
+
+check4done()
+{
+    JOBID=$1
+    i=0
+    echo -n "were we waiting on $JOBID? " >> $DEBUGFILE
+    if [ "$JOBID" == "$JOB_OUTSTANDING" ]
+	then
+        # Sometimes the email comes in BEFORE STDOUT and STDERR were written
+	# stupid Torque...
+	JOBNUM=`echo "$JOBID" | sed 's/[.].*//'`
+	JOBNAME=STDIN # the qsub default for scripts piped into qsub
+	STDOUT=$JOBNAME.o$JOBNUM
+	STDERR=$JOBNAME.e$JOBNUM
+	while [ ! -e "$STDOUT" ] || [ ! -e "$STDERR" ]
+	  do
+	  sleep 0
+	done
+	# end stupid Torque hack
+
+	echo "yes" >> $DEBUGFILE
+	return 0 # job complete :)
+    fi
+    echo "no" >> $DEBUGFILE
+    return 1 # not one of our completing jobs
+}
+
+printjoboutput()
+{
+    JOBID=$1
+    shift
+    MAIL_BODY=$*
+
+    JOBNUM=`echo "$JOBID" | sed 's/[.].*//'`
+    JOBNAME=STDIN # the qsub default for scripts piped into qsub
+    STDOUT=$JOBNAME.o$JOBNUM
+    STDERR=$JOBNAME.e$JOBNUM
+
+    cat $STDOUT
+    cat $STDERR >&2
+    rm -f $STDOUT $STDERR
+
+    return 0
+}
+
+
+echo "spawning job" >> $DEBUGFILE
+
+if [ ${1:0:2} == "-w" ]
+then
+    WALLTIME=${1:2}
+    WALLTIME_OPTION="-l walltime=$WALLTIME"
+    shift
+fi
+
+addjob $*
+
+# use process substitution to keep tail running between reads, see 'man bash'
+exec 3< <(tail -f $MAIL --pid $$) # open the MAIL file for reading on fd 3
+# $$ expands to this script's PID
+# --pid $$ sets up tail to die when this script exits, see 'man tail'
+
+echo "loop on outstanding job" >> $DEBUGFILE
+
+# email checking loop, in the forground
+while [ -n "$JOB_OUTSTANDING" ]
+  do
+  getnextmail
+  JOBID=`echo "$MAIL_BODY" | sed -n 's/PBS Job Id: *//p'`
+  if check4done "$JOBID"
+      then
+      printjoboutput $JOBID "$MAIL_BODY"
+      EXIT=`echo "$BODY" | sed -n 's/Exit_status=//p'`
+      unset JOB_OUTSTANDING
+  fi
+done
+
+# the tail dies automatically because of the --pid argument
+# we'll leave the tail file descriptor open just in case
+# tail tries to print something else before it dies.
+
+echo "qcmd complete" >> $DEBUGFILE
+
+exit $EXIT
diff --git a/posts/Abax/pbs_queues/file/qcmds b/posts/Abax/pbs_queues/file/qcmds
new file mode 100755
index 0000000..765d9e1
--- /dev/null
+++ b/posts/Abax/pbs_queues/file/qcmds
@@ -0,0 +1,261 @@
+#!/bin/bash
+#
+# Run commands through a PBS queue without having to remember anything. :p
+#
+# This is an attempt to scale qcmd functionality to multiple commands.
+# The goal is to be able to spawn a bunch of PBS jobs from the command line or
+# a script, and then wait until they all finish before carrying on.  Basically
+# the bash 'wait' functionality for background commands, except for PBS jobs
+# spawned by your particular process.
+#
+# This script doesn't bother with keeping stdin and stdout open for each job,
+# since that could be a lot of file descriptors.  It also doesn keep a process
+# going for each job.  Instead, we fire up qcmds in the backround attached to
+# an input and output fifo.  New job commands get piped into the input fifo,
+# and job completion information gets piped out the output fifo.  When all the
+# jobs that qcmds has started have completed, qcmds closes it's output fifo.
+#
+# qcmds uses endlines ('\n') as it's job-delimiters when reading it's input,
+# so you can only use single-line jobs.  They can be long lines though ;).
+# 
+# Simple usage would be:
+#   echo 'sleep 5 && echo hi there' | qcmds
+# Common usage would be:
+#   #!/bin/bash
+#   i=0
+#   $QCMDS=$((while [ $i -lt 1000 ]
+#             do
+#               echo "sleep 10 && echo $i"
+#               let "i += 1"
+#             done
+#            ) | qcmds)
+#   wait # wait for qcmds tail to finish
+#   exit 0
+#
+# Downsides to the current implementation:
+# You can only append to your $MAIL file while this is running.  I'm using a
+# .procmailrc file to deflect PBS-related mail to a seperate mailbox, so I can
+# still edit my system mailbox while this script is running, but that's one 
+# more thing you'd have to set up...
+#
+# Another drawback is that you can't run qsub from a screen session, but that
+# is just qsub in general.  Sigh.
+#
+# For small numbers of jobs where the scripting overhead of a seperate process 
+# and fifos seems excessive, take a look at the more memory intensive 'qcmd'.
+#
+# For a nice introduction to bash arrays, see
+# http://tldp.org/LDP/abs/html/arrays.html
+#
+# see ~/script/.test/t_fifo_readline for a demonstration of the `threading'
+#
+# Warning:
+# This script uses process substitution which is a non-POSIX bash feature.
+#
+#        stdin
+#          |
+#    (list of jobs)
+#          |
+#          v
+#      addjob() ---(submit job with qsub)---> Job-Queue
+#          |                                      |
+#          +----+                        (job-complete email)
+#               |                                 |
+#       while:  v                                 |
+#         checkfornew()                           v
+#         getnextmail() <---(tail -f $MAIL)--- Mailbox
+
+DEBUG=0
+MAIL=$HOME/.mailspool/completed
+# I have a ~/.procmailrc filter forwarding my PBS mail bodies to this $MAIL
+# If you don't, comment the line out so you monitor your system $MAIL.
+
+JOBS_OUTSTANDING=( ) # Store job ids that haven't completed yet
+NUM_OUTSTANDING=0
+MAIL_BODY=""         # Store the return of getnextmail()
+
+if [ $DEBUG -eq 1 ]
+    then
+    DEBUGFILE=qcmds.$$
+    > $DEBUGFILE # clear $DEBUGFILE if it existed before (Warning: clobber)
+else
+    DEBUGFILE=/dev/null
+fi
+
+# functions for the job spawning subshell
+
+addjob ()
+{
+    CMMD=$*
+    echo running: $CMMD >> $DEBUGFILE
+    SCRIPT="cd \$PBS_O_WORKDIR && source dup_env && $CMMD"
+    # dup_env is in ~/bin even though it's a script
+    # since qsub creates it's own evironment, and moves X -> PBS_O_X
+    # for example, PATH -> PBS_O_PATH.
+    # who knows why it does this...
+    JOBID=`echo $SCRIPT | qsub -mae || exit 1`
+    # -mae : Send mail on abort or execute
+    #JOBNAME=STDIN # the qsub default for scripts piped into qsub
+    echo spawner: started $JOBID >> $DEBUGFILE
+    echo $JOBID # >> $SPAWN_TO_CHECK
+    return 0
+}
+
+# functions for the job checking loop
+
+check4new() # blocks for < 1 second, fd 3 is addjob() output
+{
+    read -t1 JOBID <&3 || return 1 # nothing new.
+    # add job to our outstanding list
+    JOBNUM=`echo "$JOBID" | sed 's/[.].*//'`
+    JOBS_OUTSTANDING[$JOBNUM]=1
+    let "NUM_OUTSTANDING += 1"
+    echo "add new depend:  $JOBID" >> $DEBUGFILE
+    # extra space ':  $J' to align with addjob DEBUG message
+    return 0
+}
+
+
+# look for completion message bodies along the lines of:
+# PBS Job Id: 206.n0.abax.physics.drexel.edu
+# Job Name:   STDIN
+# ...
+#                                 <-- blank line
+#
+# could also poll on ls.
+# neither 'ls' or 'tail -f' busy loops seem to take measurable processor time.
+getnextmail() # blocking, fd 4 is tail -f $MAIL output
+{
+    BODY=""
+    DONE=0
+    INJOB=0
+    echo "block on mail" >> $DEBUGFILE
+    while [ $DONE -eq 0 ] && read LINE <&4
+      do
+      if [ "${LINE:0:11}" == "PBS Job Id:" ]
+	  then
+          # we're reading an email about a job.
+	  #echo "in a job" >> $DEBUGFILE
+	  INJOB=1
+      fi
+      if [ $INJOB -eq 1 ]
+	  then
+	  #echo "getting mail: $LINE" >> $DEBUGFILE
+	  if [ "${#LINE}" -eq 0 ]
+	      then
+              #echo "matched blank line" # we're leaving the email about our job.
+	      #echo "got mail" >> $DEBUGFILE
+	      DONE=1
+	      break
+	  fi
+	  BODY=`echo -e "$BODY\n$LINE"`
+      fi
+    done
+    echo "returning mail" >> $DEBUGFILE
+    MAIL_BODY="$BODY"
+    return 0
+}
+
+check4done()
+{
+    JOBID=$1
+    JOBNUM=`echo "$JOBID" | sed 's/[.].*//'`
+    echo -n "were we waiting on $JOBID? " >> $DEBUGFILE
+    if [ -n "${JOBS_OUTSTANDING[$JOBNUM]}" ]
+	then
+	echo "yes" >> $DEBUGFILE
+        # Sometimes the email comes in BEFORE STDOUT and STDERR were written
+	# stupid Torque...
+	JOBNAME=STDIN # the qsub default for scripts piped into qsub
+	STDOUT=$JOBNAME.o$JOBNUM
+	STDERR=$JOBNAME.e$JOBNUM
+	while [ ! -e "$STDOUT" ] || [ ! -e "$STDERR" ]
+	  do
+	  sleep 0
+	done
+        # end stupid Torque hack
+        # remove the outstanding entry from the array.
+	unset JOBS_OUTSTANDING[$JOBNUM]
+	let "NUM_OUTSTANDING -= 1"
+	return 0 # job complete :)
+    fi
+    echo "no" >> $DEBUGFILE
+    return 1 # not one of our completing jobs
+}
+
+printjobinfo()
+{
+    JOBID=$1
+    shift
+    MAIL_BODY=$*
+
+    JOBNUM=`echo "$JOBID" | sed 's/[.].*//'`
+    JOBNAME=STDIN # the qsub default for scripts piped into qsub
+    STDOUT=$JOBNAME.o$JOBNUM
+    STDERR=$JOBNAME.e$JOBNUM
+
+    HOST=`echo "$BODY" | sed -n 's/Exec host: *//p'`
+    EXIT=`echo "$BODY" | sed -n 's/Exit_status=//p'`
+    WALLTIME=`echo "$BODY" | sed -n 's/resources_used.walltime=//p'`
+    SESSION=`echo "$BODY" | sed -n 's/session_id=//p'`
+
+    echo -e "$JOBID\t$JOBNUM\t$EXIT\t$STDOUT\t$STDERR\t$HOST\t$SESSION\t$WALLTIME"
+    echo -e "$JOBID\t$JOBNUM\t$EXIT\t$STDOUT\t$STDERR\t$HOST\t$SESSION\t$WALLTIME" >> $DEBUGFILE
+
+    return 0
+}
+
+nothing()
+{
+    return 0
+}
+
+echo "start the spawning subshell" >> $DEBUGFILE
+
+# job-spawning subshell, in a subshell so we can't access it's variables
+# we send this script's stdin into the subshell as it's stdin (< /dev/stdin)
+# and we open the subshell's output for reading on file descriptor 3
+# with 'exec 3< <(subshell cmmsds)' (process substitution, see 'man bash').
+exec 3< <(
+    echo "spawner: about to read" >> $DEBUGFILE
+    while read LINE
+    do
+      addjob $LINE
+    done
+    exit 0
+) < /dev/stdin
+
+# use process substitution to keep tail running between reads, see 'man bash'
+exec 4< <(tail -f $MAIL --pid $$) # open the MAIL file for reading on fd 4
+# $$ expands to this script's PID
+# --pid $$ sets up tail to die when this script exits, see 'man tail'
+
+echo "loop on outstanding jobs" >> $DEBUGFILE
+
+# email checking loop, in the forground
+check4new # make sure there is an outstanding job first...
+while [ $NUM_OUTSTANDING -gt 0 ]
+  do
+  getnextmail
+  while check4new; do nothing; done;  # clean out the pipe
+  JOBID=`echo "$MAIL_BODY" | sed -n 's/PBS Job Id: *//p'`
+  if check4done "$JOBID"
+      then
+      printjobinfo $JOBID "$MAIL_BODY"
+  fi
+  echo "still $NUM_OUTSTANDING job(s) outstanding" >> $DEBUGFILE
+done
+
+wait
+
+echo "cleanup" >>  $DEBUGFILE
+
+exec 3>&-   # close the connection from the job-spawning subshell
+
+# the tail dies automatically because of the --pid argument
+# we'll leave the tail file descriptor open just in case
+# tail tries to print something else before it dies.
+
+echo "qcmds complete" >> $DEBUGFILE
+
+exit 0
diff --git a/posts/Abax/pbs_queues/get_walltime_echo.sh b/posts/Abax/pbs_queues/get_walltime_echo.sh
new file mode 100755
index 0000000..76d4650
--- /dev/null
+++ b/posts/Abax/pbs_queues/get_walltime_echo.sh
@@ -0,0 +1 @@
+echo "time (sleep 30 && echo 'Running a job...')" | qsub -j oe
diff --git a/posts/Abax/pbs_queues/get_walltime_echo.sh_out b/posts/Abax/pbs_queues/get_walltime_echo.sh_out
new file mode 100644
index 0000000..e68ca40
--- /dev/null
+++ b/posts/Abax/pbs_queues/get_walltime_echo.sh_out
@@ -0,0 +1,9 @@
+$ echo "time (sleep 30 && echo 'Running a job...')" | qsub -j oe
+2710.n0.physics.drexel.edu
+ ... time passes ...
+$ cat STDIN.o2710
+Running a job...
+
+real	0m30.013s
+user	0m0.000s
+sys	0m0.000s
diff --git a/posts/Abax/pbs_queues/instl b/posts/Abax/pbs_queues/instl
new file mode 100755
index 0000000..af98c79
--- /dev/null
+++ b/posts/Abax/pbs_queues/instl
@@ -0,0 +1 @@
+scp main.pdf wking@einstein.physics.drexel.edu:./public_html/course/notes/ring_Efield.pdf
diff --git a/posts/Abax/pbs_queues/main.tex b/posts/Abax/pbs_queues/main.tex
new file mode 100644
index 0000000..5e4654c
--- /dev/null
+++ b/posts/Abax/pbs_queues/main.tex
@@ -0,0 +1,62 @@
+\documentclass[letterpaper, 10pt]{article}
+
+\usepackage{hyperref}
+\usepackage{fancyvrb}
+\usepackage{relsize}
+\usepackage{fullpage}
+
+% finish setting up hyperref
+\hypersetup{pdfauthor={W.\ Trevor King}}
+\hypersetup{pdfsubject={Cluster Tools}}
+\hypersetup{pdftitle={Batch Queue Job Control}}
+\hypersetup{pdfstartview=FitH} % zooming to fit the page width.
+\hypersetup{colorlinks} % don't do the ugly boxes around links
+
+\renewcommand{\maketitle}{%
+  \begin{center}
+  {\Large Cluster Tools} \\
+  Batch Queue Job Control \\
+  W.\ Trevor King \\
+  \today
+  \end{center}
+}
+
+\setlength{\parindent}{0pt}
+\setlength{\parskip}{0.5\baselineskip}
+
+\newcommand{\inputfilesimple}[1]{\VerbatimInput[fontsize=\relsize{-1}]{#1}}
+\newcommand{\inputfile}[2]{\VerbatimInput[fontsize=\relsize{-1},frame=single,label=#2]{#1}}
+
+\newcommand{\Con}{\DefineShortVerb{\|}} % code: |grep|
+\newcommand{\Coff}{\UndefineShortVerb{\|}}
+\newcommand{\mC}[1]{\emph{<#1>}} % meta-code: <regexp> 
+\newcommand{\bs}{$\backslash$}
+
+% input the output of a shell command,
+% \com{<command>}{<file_basename>}
+% \com{echo hello world}{hello_world}
+\newcommand{\com}[2]{\inputfilesimple{#2_out}}
+% run a command silently (don't input the output)
+% \scom{sleep 10}
+\newcommand{\scom}[1]{}
+% run a queued command (single word arguments only)
+% \qcom{simple_echo.sh}
+\newcommand{\qcom}[1]{\inputfilesimple{#1_out}}
+\newcommand{\comout}{}  % input the sdtout from the last queued command
+
+\newcommand{\home}{$\sim$}  % print a tilde (user home directory shortcut)
+
+%-----------------------end preamble-------------------------
+
+\begin{document}
+\Con
+
+\maketitle
+
+\input{submitting}
+\input{querying}
+\input{altering}
+\input{example1}
+\input{utilities}
+
+\end{document}
diff --git a/posts/Abax/pbs_queues/make_com_out.sh b/posts/Abax/pbs_queues/make_com_out.sh
new file mode 100755
index 0000000..c16d83f
--- /dev/null
+++ b/posts/Abax/pbs_queues/make_com_out.sh
@@ -0,0 +1,57 @@
+#!/bin/bash
+#
+# run an example command and assemble its output for printing
+
+RunLoud ()
+{
+    BASEFILE=$1
+    shift
+    echo "\$ $*" > ${BASEFILE}_out
+    $* >> ${BASEFILE}_out 2> ${BASEFILE}_err
+}
+
+PREVQFILE=""
+cat command_list | while read COM
+  do
+  echo "$COM"
+  if [ "${COM:0:6}" == "COMOUT" ] # print the stdout output of the last job
+      then # note, you need to wait long enough for it to exist!
+      # also note, it must be the most recent job stdout (\scom{sleep ...})
+      echo " ... time passes ..." >> ${PREVQFILE}_out
+      FILE=`/bin/ls -t *.o[0-9]* | head -n1`
+      echo "\$ cat $FILE" >> ${PREVQFILE}_out
+      cat $FILE >> ${PREVQFILE}_out
+  elif [ "${COM:0:5}" == "QCOM " ] # run a queued job
+      then
+      COM="${COM:5}" # "QCOM bla bla bla" --> "bla bla bla"
+      ./$COM > comO 2> comE
+      > ${COM}_out
+      # insert '# ' in front of lines without leading spaces (except 'done')
+      # insert '> ' in front of lines with leading spaces
+      sed 's/^\([^[:space:]].*\)/$ \1/;s/$ done/> done/;s/^\([[:space:]].*\)/> \1/' $COM > ${COM}_out
+      cat comO >> ${COM}_out
+      if [ ! -s comE ] # file has no size
+	  then
+	  rm -f comE
+      else
+	  echo "errors! executing a job, see comE"
+	  echo -e "\non job: $COM" >> comE
+	  #exit 0
+      fi
+      rm -f comO
+      PREVQFILE="$COM"
+      sleep 1
+  elif [ "${COM:0:4}" == "COM " ]
+      then
+      # run a non-silent shell command
+      COM=${COM:4} # "COM outfile echo 'hi'" --> "outfile echo 'hi'"
+      RunLoud $COM
+  else
+      # run a silent shell command
+      $COM
+  fi
+done
+
+echo "complete"
+
+exit 0
diff --git a/posts/Abax/pbs_queues/make_command_list.sh b/posts/Abax/pbs_queues/make_command_list.sh
new file mode 100755
index 0000000..6dd8dc7
--- /dev/null
+++ b/posts/Abax/pbs_queues/make_command_list.sh
@@ -0,0 +1,6 @@
+#!/bin/bash
+
+for FILE in $*
+do
+  sed -n 's/.*\\com{\(.*\)}{\(.*\)}.*/COM \2 \1/p;s/.*\\scom{\(.*\)}.*/\1/p;s/.*\\qcom{\(.*\)}.*/QCOM \1/p;s/.*\\comout.*/COMOUT/p' $FILE
+done
diff --git a/posts/Abax/pbs_queues/my_err b/posts/Abax/pbs_queues/my_err
new file mode 100644
index 0000000..142195f
--- /dev/null
+++ b/posts/Abax/pbs_queues/my_err
@@ -0,0 +1 @@
+DEF
diff --git a/posts/Abax/pbs_queues/my_out b/posts/Abax/pbs_queues/my_out
new file mode 100644
index 0000000..5da849b
--- /dev/null
+++ b/posts/Abax/pbs_queues/my_out
@@ -0,0 +1 @@
+ABC
diff --git a/posts/Abax/pbs_queues/myjob.e2685 b/posts/Abax/pbs_queues/myjob.e2685
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/myjob.e2707 b/posts/Abax/pbs_queues/myjob.e2707
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/myjob.o2685 b/posts/Abax/pbs_queues/myjob.o2685
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/myjob.o2685
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/myjob.o2707 b/posts/Abax/pbs_queues/myjob.o2707
new file mode 100644
index 0000000..457d533
--- /dev/null
+++ b/posts/Abax/pbs_queues/myjob.o2707
@@ -0,0 +1 @@
+Running a job...
diff --git a/posts/Abax/pbs_queues/named_echo.sh b/posts/Abax/pbs_queues/named_echo.sh
new file mode 100755
index 0000000..7090a5b
--- /dev/null
+++ b/posts/Abax/pbs_queues/named_echo.sh
@@ -0,0 +1 @@
+echo "sleep 30 && echo 'Running a job...'" | qsub -N myjob
diff --git a/posts/Abax/pbs_queues/named_echo.sh_out b/posts/Abax/pbs_queues/named_echo.sh_out
new file mode 100644
index 0000000..95b0e28
--- /dev/null
+++ b/posts/Abax/pbs_queues/named_echo.sh_out
@@ -0,0 +1,2 @@
+$ echo "sleep 30 && echo 'Running a job...'" | qsub -N myjob
+2707.n0.physics.drexel.edu
diff --git a/posts/Abax/pbs_queues/qalter_depend_echo.sh b/posts/Abax/pbs_queues/qalter_depend_echo.sh
new file mode 100755
index 0000000..7067f38
--- /dev/null
+++ b/posts/Abax/pbs_queues/qalter_depend_echo.sh
@@ -0,0 +1,2 @@
+JOBID=`echo "sleep 30 && echo 'Running a job...'" | qsub -W depend=afterok:3`
+qstat && qalter -h n $JOBID && echo "hold removed" && qstat
diff --git a/posts/Abax/pbs_queues/qalter_depend_echo.sh_out b/posts/Abax/pbs_queues/qalter_depend_echo.sh_out
new file mode 100644
index 0000000..6732cd3
--- /dev/null
+++ b/posts/Abax/pbs_queues/qalter_depend_echo.sh_out
@@ -0,0 +1,13 @@
+$ JOBID=`echo "sleep 30 && echo 'Running a job...'" | qsub -W depend=afterok:3`
+$ qstat && qalter -h n $JOBID && echo "hold removed" && qstat
+Job id                    Name             User            Time Use S Queue
+------------------------- ---------------- --------------- -------- - -----
+2725.n0                   STDIN            sysadmin               0 R batch          
+2726.n0                   STDIN            sysadmin               0 R batch          
+2727.n0                   STDIN            sysadmin               0 H batch          
+hold removed
+Job id                    Name             User            Time Use S Queue
+------------------------- ---------------- --------------- -------- - -----
+2725.n0                   STDIN            sysadmin               0 R batch          
+2726.n0                   STDIN            sysadmin               0 R batch          
+2727.n0                   STDIN            sysadmin               0 Q batch          
diff --git a/posts/Abax/pbs_queues/qdel_echo.sh b/posts/Abax/pbs_queues/qdel_echo.sh
new file mode 100755
index 0000000..cda5776
--- /dev/null
+++ b/posts/Abax/pbs_queues/qdel_echo.sh
@@ -0,0 +1,3 @@
+JOBID=`echo "sleep 30 && echo 'Running a job...'" | qsub`
+qdel $JOBID
+echo "deleted $JOBID" && qstat
diff --git a/posts/Abax/pbs_queues/qdel_echo.sh_out b/posts/Abax/pbs_queues/qdel_echo.sh_out
new file mode 100644
index 0000000..0f9b92a
--- /dev/null
+++ b/posts/Abax/pbs_queues/qdel_echo.sh_out
@@ -0,0 +1,9 @@
+$ JOBID=`echo "sleep 30 && echo 'Running a job...'" | qsub`
+$ qdel $JOBID
+$ echo "deleted $JOBID" && qstat
+deleted 2728.n0.physics.drexel.edu
+Job id                    Name             User            Time Use S Queue
+------------------------- ---------------- --------------- -------- - -----
+2725.n0                   STDIN            sysadmin               0 R batch          
+2726.n0                   STDIN            sysadmin               0 R batch          
+2727.n0                   STDIN            sysadmin               0 R batch          
diff --git a/posts/Abax/pbs_queues/qstat_array_depend_echo_err b/posts/Abax/pbs_queues/qstat_array_depend_echo_err
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/qstat_array_depend_echo_out b/posts/Abax/pbs_queues/qstat_array_depend_echo_out
new file mode 100644
index 0000000..4fbf60a
--- /dev/null
+++ b/posts/Abax/pbs_queues/qstat_array_depend_echo_out
@@ -0,0 +1,9 @@
+$ qstat
+Job id                    Name             User            Time Use S Queue
+------------------------- ---------------- --------------- -------- - -----
+2722-1.n0                 STDIN-1          sysadmin               0 R batch          
+2722-2.n0                 STDIN-2          sysadmin               0 R batch          
+2722-3.n0                 STDIN-3          sysadmin               0 R batch          
+2722-4.n0                 STDIN-4          sysadmin               0 R batch          
+2722-5.n0                 STDIN-5          sysadmin               0 R batch          
+2723.n0                   STDIN            sysadmin               0 H batch          
diff --git a/posts/Abax/pbs_queues/qstat_array_echo_err b/posts/Abax/pbs_queues/qstat_array_echo_err
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/qstat_array_echo_out b/posts/Abax/pbs_queues/qstat_array_echo_out
new file mode 100644
index 0000000..f9b4efa
--- /dev/null
+++ b/posts/Abax/pbs_queues/qstat_array_echo_out
@@ -0,0 +1,8 @@
+$ qstat
+Job id                    Name             User            Time Use S Queue
+------------------------- ---------------- --------------- -------- - -----
+2721-1.n0                 STDIN-1          sysadmin               0 R batch          
+2721-2.n0                 STDIN-2          sysadmin               0 R batch          
+2721-3.n0                 STDIN-3          sysadmin               0 R batch          
+2721-4.n0                 STDIN-4          sysadmin               0 R batch          
+2721-5.n0                 STDIN-5          sysadmin               0 R batch          
diff --git a/posts/Abax/pbs_queues/qstat_depend_echo_err b/posts/Abax/pbs_queues/qstat_depend_echo_err
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/qstat_depend_echo_out b/posts/Abax/pbs_queues/qstat_depend_echo_out
new file mode 100644
index 0000000..bfa43a0
--- /dev/null
+++ b/posts/Abax/pbs_queues/qstat_depend_echo_out
@@ -0,0 +1,6 @@
+$ qstat
+Job id                    Name             User            Time Use S Queue
+------------------------- ---------------- --------------- -------- - -----
+2712.n0                   STDIN            sysadmin               0 R batch          
+2713.n0                   STDIN            sysadmin               0 R batch          
+2714.n0                   STDIN            sysadmin               0 H batch          
diff --git a/posts/Abax/pbs_queues/qstat_full_echo.sh b/posts/Abax/pbs_queues/qstat_full_echo.sh
new file mode 100755
index 0000000..f2da027
--- /dev/null
+++ b/posts/Abax/pbs_queues/qstat_full_echo.sh
@@ -0,0 +1,2 @@
+JOBID=`echo "sleep 30 && echo 'Running a job...'" | qsub`
+sleep 2 && qstat -f
diff --git a/posts/Abax/pbs_queues/qstat_full_echo.sh_out b/posts/Abax/pbs_queues/qstat_full_echo.sh_out
new file mode 100644
index 0000000..20e46fa
--- /dev/null
+++ b/posts/Abax/pbs_queues/qstat_full_echo.sh_out
@@ -0,0 +1,40 @@
+$ JOBID=`echo "sleep 30 && echo 'Running a job...'" | qsub`
+$ sleep 2 && qstat -f
+Job Id: 2725.n0.physics.drexel.edu
+    Job_Name = STDIN
+    Job_Owner = sysadmin@n0.physics.drexel.edu
+    job_state = R
+    queue = batch
+    server = n0.physics.drexel.edu
+    Checkpoint = u
+    ctime = Thu Jun 26 13:58:54 2008
+    Error_Path = n0.physics.drexel.edu:/home/sysadmin/howto/cluster/pbs_queues
+	/STDIN.e2725
+    exec_host = n8/0
+    Hold_Types = n
+    Join_Path = n
+    Keep_Files = n
+    Mail_Points = a
+    mtime = Thu Jun 26 13:58:55 2008
+    Output_Path = n0.physics.drexel.edu:/home/sysadmin/howto/cluster/pbs_queue
+	s/STDIN.o2725
+    Priority = 0
+    qtime = Thu Jun 26 13:58:54 2008
+    Rerunable = True
+    Resource_List.nodect = 1
+    Resource_List.nodes = 1
+    Resource_List.walltime = 01:00:00
+    session_id = 18020
+    Variable_List = PBS_O_HOME=/home/sysadmin,PBS_O_LANG=en_US.UTF-8,
+	PBS_O_LOGNAME=sysadmin,
+	PBS_O_PATH=/home/sysadmin/bin:/usr/local/bin:/usr/local/sbin:/usr/bin
+	:/usr/sbin:/bin:/sbin:/usr/X11R6/bin:/usr/local/maui/bin:/home/sysadmi
+	n/script:/home/sysadmin/bin:.,PBS_O_MAIL=/var/mail/sysadmin,
+	PBS_O_SHELL=/bin/bash,PBS_SERVER=n0.physics.drexel.edu,
+	PBS_O_HOST=n0.physics.drexel.edu,
+	PBS_O_WORKDIR=/home/sysadmin/howto/cluster/pbs_queues,
+	PBS_O_QUEUE=batch
+    etime = Thu Jun 26 13:58:54 2008
+    start_time = Thu Jun 26 13:58:55 2008
+    start_count = 1
+
diff --git a/posts/Abax/pbs_queues/qstat_named_echo_err b/posts/Abax/pbs_queues/qstat_named_echo_err
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/qstat_named_echo_out b/posts/Abax/pbs_queues/qstat_named_echo_out
new file mode 100644
index 0000000..c88d728
--- /dev/null
+++ b/posts/Abax/pbs_queues/qstat_named_echo_out
@@ -0,0 +1,4 @@
+$ qstat
+Job id                    Name             User            Time Use S Queue
+------------------------- ---------------- --------------- -------- - -----
+2707.n0                   myjob            sysadmin               0 Q batch          
diff --git a/posts/Abax/pbs_queues/qstat_particular_echo.sh b/posts/Abax/pbs_queues/qstat_particular_echo.sh
new file mode 100755
index 0000000..cffdbdd
--- /dev/null
+++ b/posts/Abax/pbs_queues/qstat_particular_echo.sh
@@ -0,0 +1,2 @@
+JOBID=`echo "sleep 30 && echo 'Running a job...'" | qsub`
+sleep 2 && qstat $JOBID
diff --git a/posts/Abax/pbs_queues/qstat_particular_echo.sh_out b/posts/Abax/pbs_queues/qstat_particular_echo.sh_out
new file mode 100644
index 0000000..0deb1c5
--- /dev/null
+++ b/posts/Abax/pbs_queues/qstat_particular_echo.sh_out
@@ -0,0 +1,5 @@
+$ JOBID=`echo "sleep 30 && echo 'Running a job...'" | qsub`
+$ sleep 2 && qstat $JOBID
+Job id                    Name             User            Time Use S Queue
+------------------------- ---------------- --------------- -------- - -----
+2724.n0                   STDIN            sysadmin               0 R batch          
diff --git a/posts/Abax/pbs_queues/qstat_queue_info_err b/posts/Abax/pbs_queues/qstat_queue_info_err
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/qstat_queue_info_out b/posts/Abax/pbs_queues/qstat_queue_info_out
new file mode 100644
index 0000000..014a0a7
--- /dev/null
+++ b/posts/Abax/pbs_queues/qstat_queue_info_out
@@ -0,0 +1,9 @@
+$ qstat -q
+
+server: n0
+
+Queue            Memory CPU Time Walltime Node  Run Que Lm  State
+---------------- ------ -------- -------- ----  --- --- --  -----
+batch              --      --       --      --    2   0 --   E R
+                                               ----- -----
+                                                   2     0
diff --git a/posts/Abax/pbs_queues/qstat_server_info_err b/posts/Abax/pbs_queues/qstat_server_info_err
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/qstat_server_info_out b/posts/Abax/pbs_queues/qstat_server_info_out
new file mode 100644
index 0000000..aaf6ad0
--- /dev/null
+++ b/posts/Abax/pbs_queues/qstat_server_info_out
@@ -0,0 +1,4 @@
+$ qstat -B
+Server             Max   Tot   Que   Run   Hld   Wat   Trn   Ext Status    
+----------------   ---   ---   ---   ---   ---   ---   ---   --- ----------
+n0.physics.drexe     0     2     0     2     0     0     0     0 Active    
diff --git a/posts/Abax/pbs_queues/qstat_set_walltime_echo_err b/posts/Abax/pbs_queues/qstat_set_walltime_echo_err
new file mode 100644
index 0000000..0f76da6
--- /dev/null
+++ b/posts/Abax/pbs_queues/qstat_set_walltime_echo_err
@@ -0,0 +1,3 @@
+qstat: Unknown queue destination |
+qstat: Unknown queue destination grep
+qstat: Unknown queue destination '[.]walltime'
diff --git a/posts/Abax/pbs_queues/qstat_set_walltime_echo_out b/posts/Abax/pbs_queues/qstat_set_walltime_echo_out
new file mode 100644
index 0000000..0258ad1
--- /dev/null
+++ b/posts/Abax/pbs_queues/qstat_set_walltime_echo_out
@@ -0,0 +1 @@
+$ qstat -f | grep '[.]walltime'
diff --git a/posts/Abax/pbs_queues/qstat_simple_echo_err b/posts/Abax/pbs_queues/qstat_simple_echo_err
new file mode 100644
index 0000000..e69de29
diff --git a/posts/Abax/pbs_queues/qstat_simple_echo_out b/posts/Abax/pbs_queues/qstat_simple_echo_out
new file mode 100644
index 0000000..3beb0ae
--- /dev/null
+++ b/posts/Abax/pbs_queues/qstat_simple_echo_out
@@ -0,0 +1,4 @@
+$ qstat
+Job id                    Name             User            Time Use S Queue
+------------------------- ---------------- --------------- -------- - -----
+2705.n0                   STDIN            sysadmin               0 Q batch          
diff --git a/posts/Abax/pbs_queues/querying.tex b/posts/Abax/pbs_queues/querying.tex
new file mode 100644
index 0000000..007f3b5
--- /dev/null
+++ b/posts/Abax/pbs_queues/querying.tex
@@ -0,0 +1,24 @@
+\section{Querying}
+\label{sec.qstat}
+
+You can get information about currently running and queued jobs with |qstat|.
+In the examples in the other sections, we've been using bare |qstat|s to get information about the status of jobs in the queue.
+You get information about a particular command with
+\qcom{qstat_particular_echo.sh}
+and you can get detailed information on a every command (or a particular one, see previous example) with the |-f| (full) option.
+\scom{sleep 40}
+\qcom{qstat_full_echo.sh}
+
+The |qstat| command gives you lots of information about the current state of a job, but to get a history you should use the |tracejob| command.
+\qcom{tracejob_echo.sh}
+
+
+You can also get the status of the queue itself by passing |-q| option to |qstat|
+\com{qstat -q}{qstat_queue_info}
+or the status of the server with the |-B| option.
+\com{qstat -B}{qstat_server_info}
+
+You can get information on the status of the various nodes with |qnodes| (a symlink to |pbsnodes|).
+The output of |qnodes| is bulky and not of public interest, so we will not reproduce it here.
+For more details on flags you can pass to |qnodes|/|pbsnodes| see |man pbsnodes|,
+but I haven't had any need for fancyness yet.
diff --git a/posts/Abax/pbs_queues/run_vel_dist.sh b/posts/Abax/pbs_queues/run_vel_dist.sh
new file mode 100755
index 0000000..1a8265e
--- /dev/null
+++ b/posts/Abax/pbs_queues/run_vel_dist.sh
@@ -0,0 +1,67 @@
+#!/bin/bash
+#
+# run_vel_dist.sh
+#
+# run N simulations each at a series of pulling speeds V
+# (non-velocity parameters are copied from the get_vel_dist.sh command line)
+# for each speed, save all unfolding data data_$V, and
+#  generate a histogram of unfolding forces hist_$V
+# also generate a file v_dep with the mean unfolding force vs velocity
+
+# We can't depend on the entire array in one shot, so
+# batch executions into a single array instance
+BN=100 # sawsim executions per array job instance
+N=50   # instances in a single array
+Vs=".2e-6 .5e-6 1e-6 2e-6 5e-6"
+SAFTEY_SLEEP=2 # seconds to sleep between spawning array job and depending on it
+               # otherwise dependencies don't catch
+
+STAMP=`date +"%Y.%m.%d.%H.%M.%S"`
+mkdir v_dep-$STAMP || exit 1
+cd v_dep-$STAMP
+echo "Date "`date` > v_dep_notes
+echo "Run$ sawsim $* -v \$V" >> v_dep_notes 
+echo "for V in $*" >> v_dep_notes
+
+Is="" # build a list of allowed 'i's, since 'for' more compact than 'while'
+i=1
+while [ $i -le $N ]
+  do
+  Is="$Is $i"
+  let "i += 1"
+done
+
+vdone_condition="afterany"
+VJOBIDS=""
+for V in $Vs
+do 
+  # run N sawtooth simulations
+  idone_condition="afterany"
+  cmd="cd \$PBS_O_WORKDIR
+         i=1
+         while [ \$i -le $BN ]; do
+           ~/sawsim/sawsim $* -v $V > run_$V-\$PBS_ARRAYID-\$i
+           let \"i += 1\"
+         done"
+  JOBID=`echo "$cmd" | qsub -h -t 1-$N -N "sawsim" -o run_$V.o -e run_$V.e` || exit 1
+  #  "3643.abax.physics.drexel.edu" --> "3643"
+  ARRAYID=`echo "$JOBID" | sed "s/\\([0-9]*\\)\\([.].*\\)/\\1/g"`
+  for i in $Is; do idone_condition="$idone_condition:$ARRAYID-$i"; done
+
+  sleep $SAFTEY_SLEEP
+  # once they're done, compile a list of all unfolding forces for each speed
+  JOBID=`echo "cd \\\$PBS_O_WORKDIR && cat run_$V-* | grep -v '#' > data_$V" |\
+      qsub -h -W depend="$idone_condition" -N sawComp -o data_$V.o -e data_$V.e` \
+        || exit 1
+  vdone_condition="$vdone_condition:$JOBID"
+  VJOBIDS="$VJOBIDS $JOBID"
+  for i in $Is; do qrls "$ARRAYID-$i"; done
+done
+# once we have force lists for each velocity, make histograms and F(V) chart
+echo "Final job :"
+echo "cd \$PBS_O_WORKDIR && ../vel_dist_graph.sh $Vs" | \
+    qsub -W depend="$vdone_condition" -N sawsimGraph || exit 1
+for job in $VJOBIDS; do qrls $job; done
+cd ..
+
+exit 0
diff --git a/posts/Abax/pbs_queues/set_walltime_echo.sh b/posts/Abax/pbs_queues/set_walltime_echo.sh
new file mode 100755
index 0000000..4afe9ef
--- /dev/null
+++ b/posts/Abax/pbs_queues/set_walltime_echo.sh
@@ -0,0 +1 @@
+echo "sleep 30 && echo 'Running a job...'" | qsub -l walltime=15:00
diff --git a/posts/Abax/pbs_queues/set_walltime_echo.sh_out b/posts/Abax/pbs_queues/set_walltime_echo.sh_out
new file mode 100644
index 0000000..15751d0
--- /dev/null
+++ b/posts/Abax/pbs_queues/set_walltime_echo.sh_out
@@ -0,0 +1,2 @@
+$ echo "sleep 30 && echo 'Running a job...'" | qsub -l walltime=15:00
+2711.n0.physics.drexel.edu
diff --git a/posts/Abax/pbs_queues/simple_echo.sh b/posts/Abax/pbs_queues/simple_echo.sh
new file mode 100755
index 0000000..0bbd3a5
--- /dev/null
+++ b/posts/Abax/pbs_queues/simple_echo.sh
@@ -0,0 +1 @@
+echo "sleep 30 && echo 'Running a job...'" | qsub
diff --git a/posts/Abax/pbs_queues/simple_echo.sh_out b/posts/Abax/pbs_queues/simple_echo.sh_out
new file mode 100644
index 0000000..81eca1d
--- /dev/null
+++ b/posts/Abax/pbs_queues/simple_echo.sh_out
@@ -0,0 +1,2 @@
+$ echo "sleep 30 && echo 'Running a job...'" | qsub
+2705.n0.physics.drexel.edu
diff --git a/posts/Abax/pbs_queues/simple_script.sh b/posts/Abax/pbs_queues/simple_script.sh
new file mode 100755
index 0000000..1c9a72c
--- /dev/null
+++ b/posts/Abax/pbs_queues/simple_script.sh
@@ -0,0 +1 @@
+qsub echo_script.sh
diff --git a/posts/Abax/pbs_queues/simple_script.sh_out b/posts/Abax/pbs_queues/simple_script.sh_out
new file mode 100644
index 0000000..8df1e03
--- /dev/null
+++ b/posts/Abax/pbs_queues/simple_script.sh_out
@@ -0,0 +1,2 @@
+$ qsub echo_script.sh
+2706.n0.physics.drexel.edu
diff --git a/posts/Abax/pbs_queues/submitting.tex b/posts/Abax/pbs_queues/submitting.tex
new file mode 100644
index 0000000..e19d2ca
--- /dev/null
+++ b/posts/Abax/pbs_queues/submitting.tex
@@ -0,0 +1,164 @@
+\section{Submitting jobs}
+\label{sec.qsub}
+
+You can submit jobs to the batch queue for later proccessing with |qsub|.
+Batch queueing can get pretty fancy, so |qsub| comes with lots of options (see |man qsub|).
+For the most part, you can trust your sysadmin to have set up some good defaults, and not worry about setting any options explicitly.
+As you get used to the batch queue system, you'll want tighter control of how your jobs execute by invoking more sophisticated options yourself, but don't let that scare you off at the beginning.
+They are, after all, only \emph{options}.
+This paper will give you a good start on the options I find myself using most often.
+
+\subsection{Simple submission}
+
+The simplest example of a job submission is:
+\qcom{simple_echo.sh}
+Which submits a job executing |sleep 30 \&\& echo 'Running a job...'| to the queue.
+The job gets an identifying ID in the queue, which |qsub| prints to |stdout|.
+
+You can check the status of your job in the queue with |qstat|.
+\com{qstat}{qstat_simple_echo}
+There is more information on |qstat| in Section \ref{sec.qstat}.
+
+If your job is too complicated to fit on a single line, you can save it in a sctipt and submit the script.
+\inputfile{echo_script.sh}{echo\_script.sh}
+\qcom{simple_script.sh}
+All the arguments discussed in later sections for the command line should have comment-style analogs that you can enter in your script if you use the script-submission approach with |qsub|.
+
+Note that you \emph{cannot} run executibles (non-shell-scripts) directly with |qsub|.
+For example
+\begin{Verbatim}
+$ cat script.py
+#!/usr/bin/python
+print "hello world!"
+$ qsub python script.py
+\end{Verbatim}
+% $ (close math-mode highlighting in emacs)
+will fail because |python| is an executible.
+Either use
+\begin{Verbatim}
+$ echo python script.py | qsub
+\end{Verbatim}
+% $ (close math-mode highlighting in emacs)
+wrap your |python| script in a |bash| script
+\begin{Verbatim}
+$ cat wrapper.sh
+#!/bin/bash
+python script.py
+$ qsub wrapper.sh
+\end{Verbatim}
+or run your |python| script directly
+\begin{Verbatim}
+$ qsub script.py
+\end{Verbatim}
+
+
+\subsection{IO: Job names and working directories}
+
+You will often be interested in the |stdout| and |stderr| output from your jobs.
+The batch queue system saves this information for you (to the directory from which you called |qsub|)
+in two files \mC{jobname}|.o|\mC{jobID-number} and \mC{jobname}|.e|\mC{jobID-number}.
+Job IDs we have seen before, they're just the numeric part of |qsub| output (or the first field in the |qstat| output).
+Job IDs are assigned by the batch queue server, and are unique to each job.
+Job names are assigned by the job submitter (that's you) and need not be unique.
+They give you a method for keeping track of what job is doing what task, since you have no control over the job ID.
+The combined \mC{jobname}|.|\mC{jobID-number} pair is both unique (for the server) and recognizable (for the user), which is why it's used to label the output data from a given job.
+You control the job name by passing the |-n |\mC{jobname} argument to |qsub|.
+\scom{sleep 40}
+\qcom{named_echo.sh}
+\com{qstat}{qstat_named_echo}
+
+Perhaps you are fine with |stdin| and |stdout|, but the default naming scheme, even with the job name flexibility, is too restrictive.
+No worries, |qsub| lets you specify exactly which files you'd like to use with the unsurprisingly named |-o| and |-e| options.
+\qcom{explicit_oe_echo.sh}
+\scom{sleep 2}
+\com{cat my_out}{explicit_oe_echo_o}
+\com{cat my_err}{explicit_oe_echo_e}
+
+A time will come when you are no longer satified with |stdin| and |stdout| and you want to open your own files or worse, run a program!
+Because no sane person uses absolute paths all the time, we need to know what directory we're in so we can construct our relative paths.
+You might expect that your job will execute from the same directory that you called qsub from, but that is not the case.
+I think the reason is that that directory is not garaunteed to exist on the host that eventually runs your program.
+In any case, your job will begin executing in your home directory.
+Writing relative paths from your home directory is about as annoying as writing absolute paths, so |qsub| gives your script a nifty environment variable |PBS\_O\_WORKDIR|, which is set to the directory you called |qsub| from.
+Since \emph{you} know that this directory exists on the hosts (since the home directories are NFS mounted on all of our cluster nodes), you can move to that directory yourself, using something like
+\scom{sleep 40}
+\qcom{workdir_echo.sh}
+\scom{sleep 2}
+\comout
+Note that if we had enclosed the echo argument in double quotes (|\"|), we would have to escape the `|\$|' symbol in our |echo| argument so that it survives the shell expansion and makes it safely into |qsub|'s input.
+
+
+\subsection{Long jobs}
+
+If you have jobs that may take longer than the default wall time (currently 1 hour), you will need to tell the job manager.
+Walltimes may seem annoying, since you don't really know how long a job will run for,
+but they protect the cluster from people running broken programs that waste nodes looping around forever without accomplishing anything.
+Therefor, your walltime doesn't have to be exactly, or even close to, your actual job execution time.
+Before submitting millions of long jobs, it is a good idea to submit a timing job to see how long your jobs should run for.
+Then set the walltime a factor of 10 or so higher.
+For example
+\qcom{get_walltime_echo.sh}
+\scom{sleep 40}
+\comout
+\qcom{set_walltime_echo.sh}
+\scom{sleep 2}
+\com{qstat -f | grep '[.]walltime'}{qstat_set_walltime_echo}
+\scom{sleep 38}
+You can set walltimes in |[[H:]M:]S| format, where the number of hours, minutes, and seconds are positive integers.
+I passed the |-j oe| combines both |sdtout| and |stdin| streams on |stdin| because |time| prints to |stderr|.
+Walltimes are only accurate on the order of minutes and above, but you probably shouldn't be batch queueing jobs that take so little time anyway.
+
+\subsection{Job dependencies}
+
+You will often find yourself in a situation where the execution of one job depends on the output of another job.
+For example, |jobA| and |jobB| generate some data, and |jobC| performs some analysis on that data.
+It wouldn't do for |jobC| to go firing off as soon as there was a free node, if there was no data available yet to analyze.
+We can deal with \emph{dependencies} like these by passing a |-W depend=|\mC{dependency-list} option to |qsub|.
+The dependency list can get pretty fancy (see |man qsub|), but for the case outlined above, we'll only need |afterany| dependencies (because |jobC| should execute after jobs |A| and |B|).
+
+Looking at the |man| page, the proper format for our dependency list is |afterany:jobid[:jobid...]|, so we need to catch the job IDs output by |qsub|.
+We'll just use the |bash| ``back tick'' command substitution operators (|`|) for this.
+\qcom{depend_echo.sh}
+\scom{sleep 2}
+\com{qstat}{qstat_depend_echo}
+\scom{sleep 50}
+\com{cat C_out}{depend_echo_out}
+Note that we have to escape the |PBS\_O\_WORKDIR| expansion twice.
+The back tick expansion expands
+ |\bs\bs\bs\$PBS\ldots| to 
+ |\bs\$PBS\ldots|.
+The double quote expansion when evaluating the echo command expands that to 
+ |\$PBS\ldots|.
+When the job finally runs, we actually expand out to our working directory.
+
+
+\subsection{Job arrays}
+
+If you have \emph{lots} of jobs you'd like to submit at once, it is tempting try
+\qcom{while_echo.sh}
+This does work, but it puts quite a load on the server as the number of jobs gets large.
+In order to allow the execution of such repeated commands the batch server provides \emph{job arrays}.
+You simply pass |qsub| the |-t array\_request| option, listing the range or list of IDs for which you'd like to run your command.
+\scom{sleep 10}
+\qcom{array_echo.sh}
+\scom{sleep 2}
+\com{qstat}{qstat_array_echo}
+
+One possibly tricky issue is depending on a job array.
+If you have an analysis job that you need to run to compile the results of your whole array, try
+\scom{sleep 50}
+\qcom{array_depend_echo.sh}
+\scom{sleep 2}
+\com{qstat}{qstat_array_depend_echo}
+\scom{sleep 60}
+\com{cat sum_out}{array_depend_echo_out}
+Note that you must create any files needed by the dependent jobs \emph{during} the early jobs.
+The dependent job may start as soon as the early jobs finish, \emph{before} the |stdin| and |stdout| files for some early jobs have been written.
+Sadly, depending on either the returned job ID or just its numeric portion doesn't seem to work.
+
+It is important that the jobs on which you depend are loaded into the server \emph{before your depending job is submitted}.
+To ensure this, you may need to add a reasonable sleep time between submitting your job array and submitting your dependency.
+However, your depending job will also hang if some early jobs have \emph{already finished} by the time you get around to submitting it.
+
+Despite the annoyance, a single sleep per job array is cheaper than a sleep after each non-array job submission to avoid crashing the server in the |while|-loop approach.
+See the examples sections and |man qsub| for more details.
diff --git a/posts/Abax/pbs_queues/sum_out b/posts/Abax/pbs_queues/sum_out
new file mode 100644
index 0000000..7ed6ff8
--- /dev/null
+++ b/posts/Abax/pbs_queues/sum_out
@@ -0,0 +1 @@
+5
diff --git a/posts/Abax/pbs_queues/tracejob_echo.sh b/posts/Abax/pbs_queues/tracejob_echo.sh
new file mode 100755
index 0000000..c5e4ee8
--- /dev/null
+++ b/posts/Abax/pbs_queues/tracejob_echo.sh
@@ -0,0 +1,2 @@
+JOBID=`echo "sleep 30 && echo 'Running a job...'" | qsub`
+sleep 2 && tracejob $JOBID
diff --git a/posts/Abax/pbs_queues/tracejob_echo.sh_out b/posts/Abax/pbs_queues/tracejob_echo.sh_out
new file mode 100644
index 0000000..27c4dd4
--- /dev/null
+++ b/posts/Abax/pbs_queues/tracejob_echo.sh_out
@@ -0,0 +1,10 @@
+$ JOBID=`echo "sleep 30 && echo 'Running a job...'" | qsub`
+$ sleep 2 && tracejob $JOBID
+
+Job: 2726.n0.physics.drexel.edu
+
+06/26/2008 13:58:57  S    enqueuing into batch, state 1 hop 1
+06/26/2008 13:58:57  S    Job Queued at request of sysadmin@n0.physics.drexel.edu, owner = sysadmin@n0.physics.drexel.edu, job name = STDIN, queue = batch
+06/26/2008 13:58:58  S    Job Modified at request of root@n0.physics.drexel.edu
+06/26/2008 13:58:58  S    Job Run at request of root@n0.physics.drexel.edu
+06/26/2008 13:58:58  S    Job Modified at request of root@n0.physics.drexel.edu
diff --git a/posts/Abax/pbs_queues/utilities.tex b/posts/Abax/pbs_queues/utilities.tex
new file mode 100644
index 0000000..b18433d
--- /dev/null
+++ b/posts/Abax/pbs_queues/utilities.tex
@@ -0,0 +1,50 @@
+\section{Utilities}
+\label{sec.utilities}
+
+After playing around on the cluster for a while, I've developed some utilities to integrate job submission with the rest of my workflow.
+So far, I have written
+ |dup\_env| for setting up an environment like the one you called qsub from,
+ |qcmd| for submitting a single command as a job,
+ and
+ |qcmds| for submitting lots of one-line jobs and waiting until they complete.
+
+\subsection{dup\_env}
+
+When |qsub| starts your job on a compute node, it sets up your environmental variables for you.
+Unfortunately, it doesn't do this by cloning your old environment, for example your new |PATH| will come from |/etc/bash.bashrc| (I think?).
+Your old |PATH| get's stored in |PBS\_O\_PATH| (see |man qsub|).
+|dup\_env| just loops through your current environment and overrides |qsub|'s decisions and reinstalls your original enviroment.
+There may be, of course, some exceptions, since you don't want to set your |HOST| to the host you called |qsub| from.
+Remember to place |dup\_env| somewhere where your restricted |qsub| path will find it.
+\inputfile{file/dup_env}{\home/bin/dup\_env}
+
+\subsection{qcmd}
+
+I often have scripts with a few big, slow jobs at the beginning, generating some data that I process in the remainder of the script.
+Rather than chew up the processors on the master node, I'd rather bump them out to the compute nodes.
+To make this alteration as painless as possible, I've written |qcmd| which submits the command in your current environment, waits for the job to finish, and outputs the job's |stdout|/|stderr|/exit-status as it's own.
+You can carry on exactly as if the job executed on your local machine.
+The script works by listening at your mailbox for the job completion email from the queue manager, but I talk about all that in the script itself.
+\inputfile{file/qcmd}{\home/script/qcmd}
+
+\subsection{qcmds}
+
+With three processes open for each waiting job, |qcmd| scales pretty poorly.
+For scripts where you need a bit more muscle to handle several tens to thousands of jobs, I wrote |qcmds|.
+|qcmds| is almost identical to |qcmd|, except that where you had one |qcmd| instance per job, now you have one |qcmds| instance per \emph{set} of jobs.
+With a whole bunch of jobs and only one |stdout|, |qcmds| needed a seperate interface.
+You just pipe your endline (`|\bs n|') delimited jobs into |qcmds| |stdin|, and out the |stdout| comes a list of completing jobs, in a tab delimited list for easy processing.
+Run the examples suggested in the script comments to get a feel for how these work.
+\inputfile{file/qcmds}{\home/script/qcmds}
+
+\subsection{qcleanmail}
+
+Both |qcmd| and |qcmds| listen for job-completion emails from the queue manager, and these can fill up your inbox pretty quickly.
+I wrote this little script to run through my mailboxes and delete all PBS-related messages.
+\inputfile{file/qcleanmail}{\home/script/qcleanmail}
+
+The referenced |procmail| filter is:
+\inputfile{file/qcleanmail.proc}{\home/.mailspool/mailfilter/qcleanmail.proc}
+
+In order to avoid filling up my system mailox with PBS junk between calls to |qcleanmail|, I use the following |\$HOME/.procmailrc| filter to deflect PBS messages to |\$HOME/.mailspool/completed|:
+\inputfile{file/_procmailrc}{\home/.procmailrc}
diff --git a/posts/Abax/pbs_queues/val1_out b/posts/Abax/pbs_queues/val1_out
new file mode 100644
index 0000000..d00491f
--- /dev/null
+++ b/posts/Abax/pbs_queues/val1_out
@@ -0,0 +1 @@
+1
diff --git a/posts/Abax/pbs_queues/val2_out b/posts/Abax/pbs_queues/val2_out
new file mode 100644
index 0000000..d00491f
--- /dev/null
+++ b/posts/Abax/pbs_queues/val2_out
@@ -0,0 +1 @@
+1
diff --git a/posts/Abax/pbs_queues/val3_out b/posts/Abax/pbs_queues/val3_out
new file mode 100644
index 0000000..d00491f
--- /dev/null
+++ b/posts/Abax/pbs_queues/val3_out
@@ -0,0 +1 @@
+1
diff --git a/posts/Abax/pbs_queues/val4_out b/posts/Abax/pbs_queues/val4_out
new file mode 100644
index 0000000..d00491f
--- /dev/null
+++ b/posts/Abax/pbs_queues/val4_out
@@ -0,0 +1 @@
+1
diff --git a/posts/Abax/pbs_queues/val5_out b/posts/Abax/pbs_queues/val5_out
new file mode 100644
index 0000000..d00491f
--- /dev/null
+++ b/posts/Abax/pbs_queues/val5_out
@@ -0,0 +1 @@
+1
diff --git a/posts/Abax/pbs_queues/while_echo.sh b/posts/Abax/pbs_queues/while_echo.sh
new file mode 100755
index 0000000..9072eee
--- /dev/null
+++ b/posts/Abax/pbs_queues/while_echo.sh
@@ -0,0 +1,6 @@
+i=0
+while [ $i -le 5 ]
+  do
+  JOBID=`echo "echo 'Running a job...'" | qsub`
+  let "i += 1"
+done
diff --git a/posts/Abax/pbs_queues/while_echo.sh_out b/posts/Abax/pbs_queues/while_echo.sh_out
new file mode 100644
index 0000000..c8e307b
--- /dev/null
+++ b/posts/Abax/pbs_queues/while_echo.sh_out
@@ -0,0 +1,6 @@
+$ i=0
+$ while [ $i -le 5 ]
+>   do
+>   JOBID=`echo "echo 'Running a job...'" | qsub`
+>   let "i += 1"
+> done
diff --git a/posts/Abax/pbs_queues/workdir_echo.sh b/posts/Abax/pbs_queues/workdir_echo.sh
new file mode 100755
index 0000000..1ffa44d
--- /dev/null
+++ b/posts/Abax/pbs_queues/workdir_echo.sh
@@ -0,0 +1 @@
+echo 'pwd && cd $PBS_O_WORKDIR && pwd' | qsub
diff --git a/posts/Abax/pbs_queues/workdir_echo.sh_out b/posts/Abax/pbs_queues/workdir_echo.sh_out
new file mode 100644
index 0000000..00c0afb
--- /dev/null
+++ b/posts/Abax/pbs_queues/workdir_echo.sh_out
@@ -0,0 +1,6 @@
+$ echo 'pwd && cd $PBS_O_WORKDIR && pwd' | qsub
+2709.n0.physics.drexel.edu
+ ... time passes ...
+$ cat STDIN.o2709
+/home/sysadmin
+/home/sysadmin/howto/cluster/pbs_queues