Trilinos, MPI, Boost on my mac

Using CMake as the foundation of my code's build system has really made my life easier. When I add a compiler to /usr/local/bin, even though my PATH is set to prefer binaries in that directory, my old build directories will still compile with the original Apple-installed gcc in /usr/bin, all because CMake uses absolute paths for everything set at configuration time.

Experience has also shown me that, when installing software packages, it's best to put them in different subdirectories of /usr/local . CMake also makes using this structure much less of a hassle, since it's easy to specify the "-L" directory for each package.

Anyway, today I downloaded the pre-compiled GCC 4.4 package from the mac HPC site and installed it to its default location of /usr/local . Next, after setting CXX=/usr/local/bin/g++ and similarly with CC, FC, and F77, I downloaded and installed the latest version of Open MPI, with --prefix=/usr/local.

Finally, I installed Trilinos to a new path at /usr/local/trilinos-9.0.2-parallel/:

./configure
--cache-file=config.cache \
--prefix=$INSTALLDIR \
--enable-mpi \
--with-mpi-compilers \
CFLAGS="-O3 -ftree-vectorize" \
CXXFLAGS="-O3 -ftree-vectorize" \
FFLAGS="-O2 -ftree-vectorize" \
--with-libs="-framework vecLib" \
--with-ldflags="-Wl,-multiply_defined -Wl,suppress" \
--enable-epetra \
--enable-aztecoo \
--enable-amesos \
--enable-examples \
--enable-didasko \
--enable-teuchos \
--enable-triutils \
--enable-galeri

Now, after making sure that mpirun points to the correct version in /usr/local, I can link against the Trilinos libraries when using the auto-vectorizing GCC 4.4.

I also compiled Boost with the new GCC into its own directory (bjam file at ${HOME}/user-config.jam:

using mpi ; using gcc : 4.4 ; 
using python : 2.5 : /opt/local/bin :
 /opt/local/include/python2.5 : /opt/local/lib : ;

) using bjam --with-mpi --with-python --with-math --with-serialization --with-test --toolset=darwin install.

For some reason, though, even when I link against the libraries using an absolute path (/usr/local/bin/g++ -O3 -DNDEBUG -Wl,-search_paths_first -headerpad_max_install_names -fPIC CMakeFiles/basicTest.dir/basicTest.cpp.o -o ../bin/basicTest ../lib/libimclib.a ../lib/libmclib.a /usr/local/lib/libmpi_cxx.dylib /usr/local/lib/libmpi.dylib /usr/local/lib/libopen-rte.dylib /usr/local/lib/libopen-pal.dylib /usr/lib/libutil.dylib /usr/local/boost-gcc44/lib/libboost_mpi-xgcc44-mt-1_39.dylib /usr/local/boost-gcc44/lib/libboost_serialization-xgcc44-mt-1_39.dylib ), it doesn't store the absolute path to the shared library in the binary. That is to say, otool -L basicTest spits out:

 /usr/local/lib/libmpi_cxx.0.dylib (compatibility version 1.0.0, current version 1.0.0)
 /usr/local/lib/libmpi.0.dylib (compatibility version 1.0.0, current version 1.0.0)
 /usr/local/lib/libopen-rte.0.dylib (compatibility version 1.0.0, current version 1.0.0)
 /usr/local/lib/libopen-pal.0.dylib (compatibility version 1.0.0, current version 1.0.0)
 /usr/lib/libutil.dylib (compatibility version 1.0.0, current version 1.0.0)
 libboost_mpi-xgcc44-mt-1_39.dylib (compatibility version 0.0.0, current version 0.0.0)
 libboost_serialization-xgcc44-mt-1_39.dylib (compatibility version 0.0.0, current version 0.0.0)
 /usr/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.12.0)
 /usr/local/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
 /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.1.4)

which means that the program fails when trying to load, because the library search paths don't include /usr/local/boost-gcc44/lib. The solution I came up with, after reading through documentation for Apple's ld and finding nothing to help, was to just symlink Boost's libraries into /usr/local/lib. Fortunately, it doesn't conflict with my gcc40 binaries because of Boost's name mangling scheme.

So now, by specifying absolute paths to the compiler, to mpirun, and to the libraries, I can compile with either Apple's compilers or with my own. Woot.

It's crazy how much of this build process hacking I've come to understand over the last couple of years. When I was at Oak Ridge, I don't think I even understood what a shared library was, or why my linker would spit out errors when I put the wrong path to a library.