Make dylib paths absolute

One problem I have come across in my trek through the murky swamps of open source libraries and so forth have been occasional issues with shared libraries on the mac. Shared libraries (dylib files on the mac, so files on Linux systems) are modules of code loaded at run time; the binary executables (or other object files) only store a path to the shared library file (and sometimes a version number). Sometimes this path is wrong: either the shared library has moved, or in the case of Boost's generated libraries, it compiled in a relative path rather than an absolute path. I ran into this issue before with CUDA's dynamic library (I was flabbergasted that it didn't automatically store an absolute path when I passed an absolute path to the linker), and my half-kludge of a fix was to add the environmental variable DYLD_LIBRARY_PATH=:/usr/local/cuda/lib.

Well, it happened again today; Boost was failing with dyld not loaded, image not found. I hit upon the google query that brought up an Apple developer page that explained the issue well.

I invented a general solution, since this will probably happen again to me. If you have shared libraries that don't use an absolute path (or don't use the right path), cd to the directory in which they reside and run the following command:

for f in *.dylib; do sudo install_name_tool -id $(pwd)/$f $f; done

Ta-da!

Ah, keyboard shortcuts.

I'm too used to using vim, where Ctrl-w deletes the preceding word in insert mode. (The same shortcut works in the terminal and other places, as well.) The downside is that when I'm on a Windows machine, i.e. using Firefox at the library, trying to delete a word reflexively closes the window in which I'm typing. This has happened three times already and almost happened once while typing in this post.

Abbreviating Russian names in BibTeX

Some of my research involves a Russian-developed method called Quasidiffusion, so naturally the appropriate works need to be properly cited. BibTeX, however, was mangling the correctly abbreviated Russian names.

"Дмитрий Ю. Анистратов" is transliterated as "Dmitriy Yu. Anistratov." Note that the middle initial is one character in the Cyrillic alphabet, but it becomes two Latin characters. Even when I enclosed the middle initial with a set of braces (which in most cases forces it to represent the text exactly), BibTeX truncated it to one Latin character (ANISTRATOV, D. Y.), which is incorrect.

The solution, gleaned from this document, is to use {\relax Yu}. as the middle name. Now, the author entry Anistratov, Dmitriy {\relax Yu}. is correctly abbreviated as "ANISTRATOV, D. YU."

Trilinos, MPI, Boost on my mac

Using CMake as the foundation of my code's build system has really made my life easier. When I add a compiler to /usr/local/bin, even though my PATH is set to prefer binaries in that directory, my old build directories will still compile with the original Apple-installed gcc in /usr/bin, all because CMake uses absolute paths for everything set at configuration time.

Experience has also shown me that, when installing software packages, it's best to put them in different subdirectories of /usr/local . CMake also makes using this structure much less of a hassle, since it's easy to specify the "-L" directory for each package.

Anyway, today I downloaded the pre-compiled GCC 4.4 package from the mac HPC site and installed it to its default location of /usr/local . Next, after setting CXX=/usr/local/bin/g++ and similarly with CC, FC, and F77, I downloaded and installed the latest version of Open MPI, with --prefix=/usr/local.

Finally, I installed Trilinos to a new path at /usr/local/trilinos-9.0.2-parallel/:

./configure
--cache-file=config.cache \
--prefix=$INSTALLDIR \
--enable-mpi \
--with-mpi-compilers \
CFLAGS="-O3 -ftree-vectorize" \
CXXFLAGS="-O3 -ftree-vectorize" \
FFLAGS="-O2 -ftree-vectorize" \
--with-libs="-framework vecLib" \
--with-ldflags="-Wl,-multiply_defined -Wl,suppress" \
--enable-epetra \
--enable-aztecoo \
--enable-amesos \
--enable-examples \
--enable-didasko \
--enable-teuchos \
--enable-triutils \
--enable-galeri

Now, after making sure that mpirun points to the correct version in /usr/local, I can link against the Trilinos libraries when using the auto-vectorizing GCC 4.4.

I also compiled Boost with the new GCC into its own directory (bjam file at ${HOME}/user-config.jam:

using mpi ; using gcc : 4.4 ; 
using python : 2.5 : /opt/local/bin :
 /opt/local/include/python2.5 : /opt/local/lib : ;

) using bjam --with-mpi --with-python --with-math --with-serialization --with-test --toolset=darwin install.

For some reason, though, even when I link against the libraries using an absolute path (/usr/local/bin/g++ -O3 -DNDEBUG -Wl,-search_paths_first -headerpad_max_install_names -fPIC CMakeFiles/basicTest.dir/basicTest.cpp.o -o ../bin/basicTest ../lib/libimclib.a ../lib/libmclib.a /usr/local/lib/libmpi_cxx.dylib /usr/local/lib/libmpi.dylib /usr/local/lib/libopen-rte.dylib /usr/local/lib/libopen-pal.dylib /usr/lib/libutil.dylib /usr/local/boost-gcc44/lib/libboost_mpi-xgcc44-mt-1_39.dylib /usr/local/boost-gcc44/lib/libboost_serialization-xgcc44-mt-1_39.dylib ), it doesn't store the absolute path to the shared library in the binary. That is to say, otool -L basicTest spits out:

 /usr/local/lib/libmpi_cxx.0.dylib (compatibility version 1.0.0, current version 1.0.0)
 /usr/local/lib/libmpi.0.dylib (compatibility version 1.0.0, current version 1.0.0)
 /usr/local/lib/libopen-rte.0.dylib (compatibility version 1.0.0, current version 1.0.0)
 /usr/local/lib/libopen-pal.0.dylib (compatibility version 1.0.0, current version 1.0.0)
 /usr/lib/libutil.dylib (compatibility version 1.0.0, current version 1.0.0)
 libboost_mpi-xgcc44-mt-1_39.dylib (compatibility version 0.0.0, current version 0.0.0)
 libboost_serialization-xgcc44-mt-1_39.dylib (compatibility version 0.0.0, current version 0.0.0)
 /usr/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.12.0)
 /usr/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.1.4)

which means that the program fails when trying to load, because the library search paths don't include /usr/local/boost-gcc44/lib. The solution I came up with, after reading through documentation for Apple's ld and finding nothing to help, was to just symlink Boost's libraries into /usr/local/lib. Fortunately, it doesn't conflict with my gcc40 binaries because of Boost's name mangling scheme.

So now, by specifying absolute paths to the compiler, to mpirun, and to the libraries, I can compile with either Apple's compilers or with my own. Woot.

It's crazy how much of this build process hacking I've come to understand over the last couple of years. When I was at Oak Ridge, I don't think I even understood what a shared library was, or why my linker would spit out errors when I put the wrong path to a library.

Checking vectors of doubles with Google Test framework

The Google C++ testing framework makes writing unit tests (with helpful output) much easier. One of the capabilities it lacks is to verify that two containers of doubles (usually std::vector) are equal in each element and in their size.

It will work on any iterable container of doubles: lists, vectors, Blitz::TinyVectors, etc.
It will also check that the lengths are the same and verify that one is not shorter than the other, and can do it without assertions that cause the unit test to prematurely abort.

#define EXPECT_ITERABLE_DOUBLE_EQ( TYPE, ref, target) \
{ \
const TYPE& _ref(ref); \
const TYPE& _target(target); \
TYPE::const_iterator tarIter   = _target.begin(); \
TYPE::const_iterator refIter = _ref.begin(); \
unsigned int i = 0; \
while(refIter != _ref.end()) { \
    if ( tarIter == _target.end() ) { \
        ADD_FAILURE() << #target \
            " has a smaller length than " #ref ; \
        break; \
    } \
    EXPECT_DOUBLE_EQ(* refIter, * tarIter) \
        << "Vectors " #ref  " (refIter) " \
           "and " #target " (tarIter) " \
           "differ at index " << i; \
    ++refIter; ++tarIter; ++i; \
} \
EXPECT_TRUE( tarIter == _target.end() ) \
    << #ref " has a smaller length than " \
       #target ; \
}

Call it with something like:

 EXPECT_ITERABLE_DOUBLE_EQ(std::vector, expectedTemp, radTemperature);

and if there's a problem, you'll get an error like:

tRadiationController.cpp:163: Failure
Value of: * refIter
  Actual: 1.0999999999999992
Expected: * tarIter
Which is: 666
Vectors expectedTemp (refIter) and radTemperature (tarIter) differ at index 1

EDIT 7/7/09: Modified to allow passing the results of function calls into the arguments. This would unquestionably be better written as a templated function rather than a macro, except that we would not have the correct line number/file name, and our error message would not have the variable names.

CMake, Python, and MacPorts

Over the last few days, I've been learning the CMake build system. There's a lot that I like about it—it's certainly more scalable than SCons, which I've been using until now. In about a day of fooling around from scratch, I converted one of my main projects (MCGeometry) to CMake.

However, some stuff that should be trivial is tricky (the most frustrating thing is trying to copy files to a build directory -- it installs stuff to the prefix path without a problem, but intermediate copies don't work well at all). I've gone over the fooling around needed to get boost.python to work with MacPorts, but one would expect CMake (since its configuration allows the manual selection of most compilers, libraries, etc.) to handle it without a problem. Sadly, though, when using the FindPythonLibs module and FindSWIG and UseSWIG, it defaults to linking against the Python framework found in /System/Library. I spent probably a few hours googling for a solution, but finally today I found one that works without any hacks at all.

The key is that on Mac OS X systems, rather than looking for headers and libraries, FindPython prefers to search for the Python framework. There are several locations it checks in some preferential order. All that needs to be done is to symlink the Python framework installed by MacPorts to one of the higher-preference locations:

ln -s /opt/local/Library/Frameworks/Python.framework /Library/Frameworks/

Ta-da, now running otool -L on the SWIG modules that CMake generates properly shows that they link against the MacPorts framework, and the right include and library paths are used.

Boost mpi and python with macports

On my system, I installed MacPorts (formerly DarwinPorts) so that I could get matplotlib and numpy without any extra effort. In the process, it installed its own version of Python in /opt/local instead of the default mac one at /System/Library/Frameworks/Python.framework/Versions/2.5/. The former is now the only Python version I use on my computer. It hadn't posed a problem until I tried to install boostmpi (see previous post on MPI with C++ and Python).

Well, after I told it to use the correct I got the unfortunate error:
"Fatal Python error: Interpreter not initialized (version mismatch?)" A little searching showed that the problem was that the libraries (specifically, the libboost_python dynamic library) was linking against the dynamic library found in /System/Library rather than the one set up in my environment and everywhere else. Using otool -L on the dynamic lib confirmed this, and I used the same tool while trying to find a solution (rather than using the Python failure to check).

On the Mac, installing Boost properly (with any of the libraries that need compiling) requires passing the --toolset=darwin option to bjam. Grepping the Boost source for "/System/Library" revealed that location to be hardwired as the default Python location if the darwin toolset is used. The solution is to specify in the user-config.jam file exactly what version of Python to use and where. My final install script (echo, bjam, and rm are the three commands):
echo "using mpi ; using python : 2.5 : /opt/local/bin : /opt/local/include/python2.5 : /opt/local/lib : ;" > ${HOME}/user-config.jam
bjam --with-mpi --with-python --with-math --with-serialization --toolset=darwin install --prefix=$INSTALLDIR
rm ${HOME}/user-config.jam

Using otool -L on the new libboost_python-xgcc40-mt-1_39.dylib verified that it linked against the correct Python, and boostmpi ran correctly after installing it.

Python MPI, with C++

The front end to my experimental code is in Python (with C++ running all the important stuff, and SWIG coupling them), which is how I want it to stay. Sooner or later my problems are going to outgrow my computer, and I'll need to run them on a cluster in parallel using MPI. It wasn't clear whether this was possible, or how it was to be done. But Python at its heart is C, and the shared objects it runs with are just code. I knew that, for example, Trilinos was somehow able to use both MPI and Python; so it's certainly possible.

The first thing I did was to search for implementations where people just used MPI with Python (without worrying about the C++ side). There are several such implementations, including the apparently stillborn pyMPI, the numpy-dependent Pypar, the basic mpi4py, and Python bindings to MPI through Boost::Python and Boost.MPI. Boost.MPI looked like the best implementation of them all, but unfortunately I could not for the life of me figure out where it was in the distribution or how to install it. (It turns out the version I was using, 1.36, didn't have it, but even in the latest 1.39 I can't see how to install the Python module.)

So the first task I did was to install mpi4py, which was trivial, and to write a quick program. mpi4py is a laughably thin front-end, so the code is not at all Pythonic.
from mpi4py import MPI
comm = MPI.COMM_WORLD
print "Hi, I am proc %d of %d" % (mpi.rank + 1, mpi.size)
Then, to run it, mpirun -np 2 python run.py

That worked fine. Surprising to me was how easily the MPI calls in C++ worked with this. I created a small class, Thing, with a member function Thing::sendRecv() that did some MPI_Send and MPI_Recv calls (depending on whether the processor was rank 0 or not), and return. I put a quick SWIG wrapper around it, compiled it to generate _thing.so and thing.py, and changed my Python code to:
from mpi4py import MPI
import thing
comm = MPI.COMM_WORLD
print "Hi, I am proc %d of %d" % (comm.Get_rank(), comm.Get_size())
t = thing.Thing()
t.sendRecv()


Again, mpirun -np 2 python run.py, and it worked! The subroutine did its thing and returned with no problem. It looks like using both MPI and Python with scientific computing is good to go.

After this was working, I decided to try to use the "standalone" boost.mpi, but that was opening up a whole other can of worms. My excursion into the land of otool and bjam will be detailed soon. Suffice it to say that, once I got boost.mpi compiled and linked against the right version of Python, the same code (except the MPI in Python looked much better).

I surmise that the use of any Python MPI library can be rendered unnecessary as long as your C++ code calls MPI_Init and MPI_Finalize in the right places.

MCNP criticality search Python script

One of the occasional tasks of a nuclear engineering student is to perform a criticality search using some parameter. That is, by varying (for example) the radius r of a uranium sphere, at some point the neutron production (from fission) and loss (from absorption and leakage) terms exactly balance, and the reactor is critical. The issue is complicated a little when using a Monte Carlo simulation, since the k-eigenvalue (unity when critical) is only reported with a certain probability or tolerance; running more particles will decrease the uncertainty.

Undergraduates, being unthinking animals, usually try this arduous task by hand, laboriously changing the parameter in an input file, running it, and repeating. As a graduate student, such tediousness is beneath me, as I would rather spend an equal amount of time writing a script to do the task for me. (Experience dictates that most of the time, some error or revision will require re-doing the task at least once, which means that a script will often save time by a factor of two or more. Besides, scripts are reusable and an educational experience.)

These Python scripts (gzipped archive) will do just that. It's a fairly simple binary search (no Newton–Krylov iterations) partly because of the complicating factor of the uncertainty in each answer. When criticality is reported to within the experimental standard deviation, the script increases the number of particles until such a parameter is determined that results in criticality within the user's specified tolerance. The script will run MCNP, process the output to determine k-effective, and iterate to your heart's content. If you have matplotlib installed, it will also produce a plot of how k varies with the parameter:

The MCNP search script is by no means perfect, and it assumes you have mcnp5 in your path. You have to know some Python to create a subclass of McnpSolver for your particular problem.

Included are scripts for finding the critical parameters of HEU cubes, cylinders, hemispheres, etc. There's also a script for generating a spherical shell volume tally to calculate the radial flux of the converged critical sphere.

Future updates of the MCNP criticality script will be posted on my projects page, although you should not expect there to be any. If you use this script, please cite its use if you report or present the results.

En dash

Today, having noticed the behavior in my literature review, I learned that en dashes are the appropriate punctuation to use between the names of joint authors. For example, "Newton-Krylov methods" should be rewritten as "Newton–Krylov methods."

See this google books result.