NWChem-7.2.3

Webpage

https://nwchemgit.github.io/

Version

7.2.3

Build Environment

  • GCC 13.1.1 (gcc-toolset-13)
  • Intel MKL 2025.0.0.1 (oneAPI 2025.0.1)
  • Open MPI 4.1.8

Files Required

  • nwchem-7.2.3-release.revision-d690e065-src.2024-08-27.tar.bz2
  • runtest.md.mpi

#!/bin/sh
./runtests.mpi.unix procs 48 \
 na_k/nak \
 na_k/nak_md \
 crown/crown_md \
 ethanol/ethanol_md \
 ethanol/ethanol_ti \
 had/had_em \
 had/had_md \
 prep/a3n \
 prep/aal \
 prep/fsc \
 water/water_md

Build Procedure

#!/bin/sh

VERSION=7.2.3
INSTALL_PREFIX=/apl/nwchem/7.2.3

BASEDIR=/home/users/${USER}/Software/NWChem/${VERSION}
TARBALL=${BASEDIR}/nwchem-7.2.3-release.revision-d690e065-src.2024-08-27.tar.bz2

WORKDIR=/gwork/users/${USER}/nwchem-release

#---------------------------------------------------------------------
umask 0022
export LANG=C
ulimit -s unlimited

module -s purge
module -s load gcc-toolset/13
module -s load mkl/2025.0.0.1
module -s load openmpi/4.1.8/gcc13

cd ${WORKDIR}
if [ -d nwchem-${VERSION} ]; then
 mv nwchem-${VERSION} nwchem-erase
 rm -rf nwchem-erase &
fi

tar jxf ${TARBALL}

export NWCHEM_TOP=${WORKDIR}/nwchem-${VERSION}
export NWCHEM_MODULES="all python"
export NWCHEM_TARGET=LINUX64
export ARMCI_NETWORK=MPI-PR

export USE_OPENMP=y
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y

export USE_NOFSCHECK=TRUE
export USE_NOIO=TRUE
export MRCC_METHODS=TRUE
export CCSDTQ=TRUE
export LIB_DEFINES=-DDFLT_TOT_MEM=180000000 # 1.8 GB/process

export PYTHONVERSION=3.6

export BLAS_SIZE=8
export BLASOPT="-m64 -L${MKLROOT}/lib -Wl,--no-as-needed -lmkl_gf_ilp64 -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl"
export LAPACK_SIZE=8
export LAPACK_LIB="${BLASOPT}"
export USE_SCALAPACK=y
export SCALAPACK_SIZE=8
export SCALAPACK="-m64 -L${MKLROOT}/lib -lmkl_scalapack_ilp64 -Wl,--no-as-needed -lmkl_gf_ilp64 -lmkl_gnu_thread -lmkl_core -lmkl_blacs_openmpi_ilp64 -lgomp -lpthread -lm -ldl"

cd ${NWCHEM_TOP}/src

# cheat
mkdir ../bin
ln -s /usr/libexec/platform-python3.6-config ../bin/python3.6-config
export PATH="${PATH}:${NWCHEM_TOP}/bin:${NWCHEM_TOP}/QA"

make nwchem_config
make

# involve also version info
cd ${NWCHEM_TOP}/src/util
make version
make
cd ${NWCHEM_TOP}/src
make link
cd ${NWCHEM_TOP}

# installation
mkdir -p ${INSTALL_PREFIX}
cp -fr LICENSE.TXT README.md release.notes.* examples ${INSTALL_PREFIX}

mkdir -p ${INSTALL_PREFIX}/bin

cp -f ${NWCHEM_TOP}/bin/${NWCHEM_TARGET}/nwchem ${INSTALL_PREFIX}/bin
chmod 755 ${INSTALL_PREFIX}/bin/nwchem

cp -fr ${NWCHEM_TOP}/src/data                ${INSTALL_PREFIX}
cp -fr ${NWCHEM_TOP}/src/basis/libraries     ${INSTALL_PREFIX}/data
cp -fr ${NWCHEM_TOP}/src/basis/libraries.bse ${INSTALL_PREFIX}/data
cp -fr ${NWCHEM_TOP}/src/nwpw/libraryps      ${INSTALL_PREFIX}/data

# create default.nwchemrc
cat << EOS > ${INSTALL_PREFIX}/default.nwchemrc
nwchem_basis_library ${INSTALL_PREFIX}/data/libraries/
nwchem_nwpw_library ${INSTALL_PREFIX}/data/libraryps/
ffield amber
amber_1 ${INSTALL_PREFIX}/data/amber_s/
amber_2 ${INSTALL_PREFIX}/data/amber_x/
#amber_3 ${INSTALL_PREFIX}/data/amber_q/
#amber_4 ${INSTALL_PREFIX}/data/amber_u/
spce    ${INSTALL_PREFIX}/data/solvents/spce.rst
charmm_s ${INSTALL_PREFIX}/data/charmm_s/
charmm_x ${INSTALL_PREFIX}/data/charmm_x/
EOS

# some tests need this...
cp -f ${INSTALL_PREFIX}/default.nwchemrc ~/.nwchemrc

# run test
export NWCHEM_EXECUTABLE=${INSTALL_PREFIX}/bin/nwchem
export OMP_NUM_THREADS=1

cd ${NWCHEM_TOP}/QA
./doqmtests.mpi 2 fast >& doqmtests.mpi.fast.log

mv testoutputs testoutputs-serial

mkdir -p ${INSTALL_PREFIX}/testlog
cp -fr doqmtests.mpi.fast.log \
      testoutputs-serial \
   ${INSTALL_PREFIX}/testlog

Tests

Parallel test was submitted as a job with the script below.

#!/bin/sh
#PBS -l select=1:ncpus=48:mpiprocs=48:ompthreads=1
#PBS -l walltime=24:00:00

VERSION=7.2.3
INSTALL_PREFIX=/apl/nwchem/7.2.3

BASEDIR=/home/users/${USER}/Software/NWChem/${VERSION}

WORKDIR=/gwork/users/${USER}/nwchem-release

RUNTESTMD=runtest.md.mpi
RUNTESTMD_PATH=${BASEDIR}/${RUNTESTMD}

#---------------------------------------------------------------------
umask 0022
export LANG=C
ulimit -s unlimited

module -s purge
module -s load gcc-toolset/13
module -s load mkl/2025.0.0.1
module -s load openmpi/4.1.8/gcc13

export NWCHEM_TOP=${WORKDIR}/nwchem-${VERSION}
# run test
export NWCHEM_EXECUTABLE=${INSTALL_PREFIX}/bin/nwchem
export PATH="$PATH:$NWCHEM_TOP/QA"

cd ${NWCHEM_TOP}/QA

./doqmtests.mpi 48 >& doqmtests.mpi.log
cp -f ${RUNTESTMD_PATH} .
sh ${RUNTESTMD} >& runtest.md.mpi.log

mv testoutputs testoutputs-mpi

mkdir -p ${INSTALL_PREFIX}/testlog
cp -fr doqmtests.mpi.log \
      runtest.md.mpi.log \
      testoutputs-mpi \
   ${INSTALL_PREFIX}/testlog

Test Results

Copy of the results are available at /apl/nwchem/7.2.3/testlog.

List of failed tests: serial version (doqmtests.mpi.fast.log)

  • h2o-b3lyp-disp/h2o-b3lyp-disp => insufficient memory specification in the input file
  • (oh2/oh2 => to be failed)
  • dft_siosi3/dft_siosi3 => insufficient memory specification in the input file
  • tce_cr_eom_t_ozone/tce_cr_eom_t_ozone => 0:createfile:failed ga_create size/nproc bytes:
  • tce_mrcc_bwcc/tce_mrcc_bwcc => minor numerical error (-76.0643135404 vs -76.0643135403)
  • tce_mrcc_mkcc/tce_mrcc_mkcc => minor numerical error (-76.0630229504 vs -76.0630229503 & -76.0702306002 vs -76.0702306001)
  • tce_mrcc_bwcc_subgroups/tce_mrcc_bwcc_subgroups => Caught signal 8 (Floating point exception: integer divide by zero)
  • qmmm_grad0/qmmm_grad0 => insufficient memory specification in the input file
  • lys_qmmm/lys_qmmm => same as above
  • ethane_qmmm/ethane_qmmm => same as above
  • qmmm_freq/qmmm_freq => same as above
  • h2o-b3lyp-disp/h2o-b3lyp-disp => same as above
  • o2_ccca/o2_ccca => 0:createfile: failed ga_create size/nproc bytes:

List of failed tests: parallel version (doqmtests.mpi.log)

  • h2o-b3lyp-disp/h2o-b3lyp-disp, oh2/oh2, qmmm_grad0/qmmm_grad0, lys_qmmm/lys_qmmm, ethane_qmmm/ethane_qmmm, qmmm_freq/qmmm_freq, h2o-b3lyp-disp/h2o-b3lyp-disp => same error as serial version
  • tce_cc2_c2/tce_cc2_c2 => not enough memory on node
  • k6h2o/k6h2o => k6h2o.err: No such file or directory (not enough memory on node?)
  • tddftgrad_h2o_cis_lda/tddftgrad_h2o_cis_lda => failed without message. The calculation was completed.
  • rt_tddft_mocap/rt_tddft_mocap => not enough memory on node
  • rt_tddft_water_abs_spec/rt_tddft_water_abs_spec => same as above
  • p2ta-vem/p2ta-vem => insufficient memory specification in the input file

List of failed tests: MD (runtest.md.mpi.log)

  • na_k/nak_md => ?
  • crown/crown_md => ethanol_md.err: No such file or directory (not enough memory on node?)
  • ethanol/ethanol_md => ethanol_md.err: No such file or directory (not enough memory on node?)
  • ethanol/ethanol_ti => ethanol_ti.err: No such file or directory (not enough memory on node?)
  • had/had_em => 0: Dimension mwm too small    0
  • had/had_md => ?
  • prep/a3n => Unresolved atom types in fragment HEM

Notes

  • Basically follows 7.2.2.
  • Calculation suddenly stops when Intel MPI was employed regardless of compiler type (gcc, intel classic, intel llvm).
    • Open MPI is free from the issue.
  • MKL is better than OpenBLAS in terms of performance. This is the same as previous versions.
  • Performance of Intel 2025 (llvm; icx, icpx, ifx) version is worse than gcc.
    • Minor tweak necessary upon build (sed -i -e "s/ -openmp/ -qopenmp/" -e "s/ifort/ifx/" config/makefile.h etc.)
  • Intel Compiler Classic 2023 (icc, icpc, ifort) version shows better performance than gcc in some general cases. However, this version is sometimes extremely slow or simply emits errors.
  • A lot of TSAN warning messages are displayed for gcc13+mkl 2025.0.0.1+libiomp5 or intel2025 (llvm)+mkl cases.
    • (e.g.) Warning: please export TSAN_OPTIONS='ignore_noninstrumented_modules=1' to avoid false positive reports from the OpenMP runtime!
    • TSAN => thread sanitizer?
    • No significant changes in the performance or the test results, though.
View PDF