Trilinos, MPI, Boost on my mac

Using CMake as the foundation of my code's build system has really made my life easier. When I add a compiler to /usr/local/bin, even though my PATH is set to prefer binaries in that directory, my old build directories will still compile with the original Apple-installed gcc in /usr/bin, all because CMake uses absolute paths for everything set at configuration time.

Experience has also shown me that, when installing software packages, it's best to put them in different subdirectories of /usr/local . CMake also makes using this structure much less of a hassle, since it's easy to specify the "-L" directory for each package.

Anyway, today I downloaded the pre-compiled GCC 4.4 package from the mac HPC site and installed it to its default location of /usr/local . Next, after setting CXX=/usr/local/bin/g++ and similarly with CC, FC, and F77, I downloaded and installed the latest version of Open MPI, with --prefix=/usr/local.

Finally, I installed Trilinos to a new path at /usr/local/trilinos-9.0.2-parallel/:

./configure
--cache-file=config.cache \
--prefix=$INSTALLDIR \
--enable-mpi \
--with-mpi-compilers \
CFLAGS="-O3 -ftree-vectorize" \
CXXFLAGS="-O3 -ftree-vectorize" \
FFLAGS="-O2 -ftree-vectorize" \
--with-libs="-framework vecLib" \
--with-ldflags="-Wl,-multiply_defined -Wl,suppress" \
--enable-epetra \
--enable-aztecoo \
--enable-amesos \
--enable-examples \
--enable-didasko \
--enable-teuchos \
--enable-triutils \
--enable-galeri

Now, after making sure that mpirun points to the correct version in /usr/local, I can link against the Trilinos libraries when using the auto-vectorizing GCC 4.4.

I also compiled Boost with the new GCC into its own directory (bjam file at ${HOME}/user-config.jam:

using mpi ; using gcc : 4.4 ; 
using python : 2.5 : /opt/local/bin :
 /opt/local/include/python2.5 : /opt/local/lib : ;

) using bjam --with-mpi --with-python --with-math --with-serialization --with-test --toolset=darwin install.

For some reason, though, even when I link against the libraries using an absolute path (/usr/local/bin/g++ -O3 -DNDEBUG -Wl,-search_paths_first -headerpad_max_install_names -fPIC CMakeFiles/basicTest.dir/basicTest.cpp.o -o ../bin/basicTest ../lib/libimclib.a ../lib/libmclib.a /usr/local/lib/libmpi_cxx.dylib /usr/local/lib/libmpi.dylib /usr/local/lib/libopen-rte.dylib /usr/local/lib/libopen-pal.dylib /usr/lib/libutil.dylib /usr/local/boost-gcc44/lib/libboost_mpi-xgcc44-mt-1_39.dylib /usr/local/boost-gcc44/lib/libboost_serialization-xgcc44-mt-1_39.dylib ), it doesn't store the absolute path to the shared library in the binary. That is to say, otool -L basicTest spits out:

 /usr/local/lib/libmpi_cxx.0.dylib (compatibility version 1.0.0, current version 1.0.0)
 /usr/local/lib/libmpi.0.dylib (compatibility version 1.0.0, current version 1.0.0)
 /usr/local/lib/libopen-rte.0.dylib (compatibility version 1.0.0, current version 1.0.0)
 /usr/local/lib/libopen-pal.0.dylib (compatibility version 1.0.0, current version 1.0.0)
 /usr/lib/libutil.dylib (compatibility version 1.0.0, current version 1.0.0)
 libboost_mpi-xgcc44-mt-1_39.dylib (compatibility version 0.0.0, current version 0.0.0)
 libboost_serialization-xgcc44-mt-1_39.dylib (compatibility version 0.0.0, current version 0.0.0)
 /usr/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.12.0)
 /usr/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.1.4)

which means that the program fails when trying to load, because the library search paths don't include /usr/local/boost-gcc44/lib. The solution I came up with, after reading through documentation for Apple's ld and finding nothing to help, was to just symlink Boost's libraries into /usr/local/lib. Fortunately, it doesn't conflict with my gcc40 binaries because of Boost's name mangling scheme.

So now, by specifying absolute paths to the compiler, to mpirun, and to the libraries, I can compile with either Apple's compilers or with my own. Woot.

It's crazy how much of this build process hacking I've come to understand over the last couple of years. When I was at Oak Ridge, I don't think I even understood what a shared library was, or why my linker would spit out errors when I put the wrong path to a library.

Checking vectors of doubles with Google Test framework

The Google C++ testing framework makes writing unit tests (with helpful output) much easier. One of the capabilities it lacks is to verify that two containers of doubles (usually std::vector) are equal in each element and in their size.

It will work on any iterable container of doubles: lists, vectors, Blitz::TinyVectors, etc.
It will also check that the lengths are the same and verify that one is not shorter than the other, and can do it without assertions that cause the unit test to prematurely abort.

#define EXPECT_ITERABLE_DOUBLE_EQ( TYPE, ref, target) \
{ \
const TYPE& _ref(ref); \
const TYPE& _target(target); \
TYPE::const_iterator tarIter   = _target.begin(); \
TYPE::const_iterator refIter = _ref.begin(); \
unsigned int i = 0; \
while(refIter != _ref.end()) { \
    if ( tarIter == _target.end() ) { \
        ADD_FAILURE() << #target \
            " has a smaller length than " #ref ; \
        break; \
    } \
    EXPECT_DOUBLE_EQ(* refIter, * tarIter) \
        << "Vectors " #ref  " (refIter) " \
           "and " #target " (tarIter) " \
           "differ at index " << i; \
    ++refIter; ++tarIter; ++i; \
} \
EXPECT_TRUE( tarIter == _target.end() ) \
    << #ref " has a smaller length than " \
       #target ; \
}

Call it with something like:

 EXPECT_ITERABLE_DOUBLE_EQ(std::vector, expectedTemp, radTemperature);

and if there's a problem, you'll get an error like:

tRadiationController.cpp:163: Failure
Value of: * refIter
  Actual: 1.0999999999999992
Expected: * tarIter
Which is: 666
Vectors expectedTemp (refIter) and radTemperature (tarIter) differ at index 1

EDIT 7/7/09: Modified to allow passing the results of function calls into the arguments. This would unquestionably be better written as a templated function rather than a macro, except that we would not have the correct line number/file name, and our error message would not have the variable names.