Getting started using Enzo-E

This page will help you get Enzo-E and Cello up and running. It covers downloading the source code, porting the code to new platforms, configuring and compiling the code, and running a sample test problem.

Other pages are available for helping to get started on specific architectures, including the “Frontera” supercomputer at TACC and the “Pleiades” supercomputer at NASA.

Dependency Installation

Before compiling Enzo-E / Cello, you may need to download and install 1. CMake, 2. Charm++, 3. HDF5, 4. libpng, and (optionally) 5. Grackle:

1. Install `CMake`

Most systems nowaways have CMake already installed. If not, you can get the binary distribution from the CMake website.

2. Install `Charm++`

We generally recommend that you download Charm++, from the GitHub repository and then retrieve the latest version (7.0.0) known to build Enzo-E/Cello. This is demonstrated by the following command:

git clone https://github.com/UIUC-PPL/charm.git
cd charm
git checkout v7.0.0

Alternatively, if you don’t have git installed, you can also download the appropriate release version of the code from the main website or from the Release page on GitHub.

Charm++ can be configured to use different “backends” for handling communication, which include:

mpi: our recommended default choice for distributed machines. Under this configuration, communication occurs via calls to MPI commands (this obviously requires an MPI installation).
netlrts: our recommended choice for development/testing on a local machine (this can make debugging far simpler). Under this configuration, communication occurs via standard networking protocols (that all modern computers are equipped with). This is generally slower than other options.

Charm++ uses cmake as the default build system. To build Charm++ start from directory holding all downloaded files.

Invoke the following command to build with the mpi backend:

mkdir build-mpi # this directory name is arbitrary
cd build-mpi
cmake -DNETWORK=mpi -DSMP=OFF ..
make # you can specify make -j4 to compile with 4 threads

The following commands to build with the netlrts backend:

mkdir build-netlrts # this directory name is arbitrary
cd build-netlrts
cmake -DNETWORK=netlrts -DSMP=OFF ..
make # you can specify make -j4 to compile with 4 threads

As an aside Charm++ can also be configured to directly use a machine’s low-level communication primitives (that MPI implementations wrap). It can be complicated to compile with these backends, so additional details are provided on an architechture-specific basis.

3. Install `HDF5`

“HDF5 is a “data model, library, and file format for storing and managing data”, and is the primary library used by Enzo-E / Cello for data output.

If HDF5 is not already installed on your machine, it may be available through your operating system distribution, otherwise it can be downloaded from the HDF5 website. Enzo-E / Cello currently uses the “serial” (non-MPI) version of HDF5.

4. Install `libpng`

“libpng is the official PNG reference library”, and is the image format used by Enzo-E / Cello.

If libpng is not already installed on your machine, it may be available through your operating system distribution, otherwise it can be downloaded from the libpng website.

5. Install Grackle (Optional)

By default, Enzo-E requires the Grackle chemistry and cooling library. For user-convenience, the default behavior is for Grackle to be automatically downloaded and compiled as part of an Enzo-E build. Alternatively, if you want to make use of an existing Grackle installation (those installation instructions are provided here), you can set -DUSE_EXTERNAL_GRACKLE=ON.

Note

If you choose to link Enzo-E against a pre-built version of Grackle, that version must have been built with CMake. (Enzo-E no longer supports linking against version of Grackle built with its “Classic Build System”)

If you do not need to use Grackle, you can disable it by setting -DUSE_GRACKLE=OFF.

6. Install yt (Optional)

If you want to use the yt python package to analyse Enzo-E output data, you should install the latest version from source. This can be done with the following commands:

git clone https://github.com/yt-project/yt.git
cd yt
pip install -e .

This is NOT a requirement for building and running Enzo-E, but it is used in some tests.

Configuring/Building

Enzo-E / Cello is currently hosted on github.com (previously bitbucket.com). To obtain the latest version of the source code, you may clone it from the repository Enzo-E / Cello github repository:

git clone https://github.com/enzo-project/enzo-e.git

For a basic configuration on a linux system with the GNU compiler stack, the following command should work out of the box. If not, please report any problems. See the following subsection for more configuration options.

mkdir my-first-build # Again, the directory name can be anything
cd my-first-build
# replace <PATH/TO/charm/build-dir> with the path to the directory created
# when you built charm++ (either build-mpi or build-netlrts if you've been
# following along)
cmake -DCHARM_ROOT=<PATH/TO/charm/build-dir> \
      -DEnzo-E_CONFIG=linux_gcc -DUSE_GRACKLE=OFF ..
make -j4 # -j4 tells make to execute up to 4 commands in parallel

To build on a Mac, you should only need to replace linux_gcc with darwin_clang.

Note, if ninja is installed, the ninja build system can be used for faster build times. This is done by adding -GNinja to the cmake command (before the ..) and calling ninja afterwards instead of make.

The Enzo-E executable is built within bin/.

Configuration options

Current cmake options are listed in the following subsubsections. Skip ahead to Specifying Configuration Options for details about how to specify the configuration.

General Configuration

These options are related to Enzo-E’s general configuration.

General Configuration
Name	Description	Default
`USE_DOUBLE_PREC`	Use double precision. Turn off for single precision.	ON
`new_output`	Temporary setting for using new Output implementation	OFF
`node_size`	Maximum number of procesess per shared-memory node (can be larger than needed)	64
`smp`	Use Charm++ in SMP mode (Charm++ must have been compiled to support SMP mode).	OFF
`balance`	Enable charm++ dynamic load balancing	ON
`balancer_included`	Charm++ load balancer included	“CommonLBs”
`balancer_default`	Charm++ load balancer to use by default	“TreeLB”

Configuring Dependencies

The parameters in the following table control whether Enzo-E / Cello should make use of certain external dependencies. These options all assume that cmake is succesfully able to find the location of these dependencies. In some cases, you may need to provide additional hints about the location of the dependencies (see Specifying Configuration Options for more details)

External Dependencies
Name	Description	Default
`USE_GRACKLE`	Use Grackle Chemistry	ON
`USE_EXTERNAL_GRACKLE`	Indicates preference for using an existing grackle installation or directly embedding grackle within the Enzo-E build.	OFF
`use_jemalloc`	Use the jemalloc library for memory allocation	OFF
`use_papi`	Use the PAPI performance API	OFF

Floating-Point Optimization Options

Generally, Enzo-E / Cello will not be compiled with value-unsafe floating point optimizations unless they are explicitly enabled. This default behavior was chosen to support automated tests that check symmetry preservation in certain components of the code (mostly related to hydro/MHD). While this feature is not essential for all applications, it is very useful when debugging new features.

Note that the Intel compilers deviate from this behavior. By default, they enable more aggressive floating point optimizations by default (consequently the tests checking symmetry may fail). This is a large part of the reason that the Intel compilers have a reputation for producing faster code (in practice, this is not always the case).

Floating point optimizations
Name	Description	Default
`OPTIMIZE_FP`	Enable value-unsafe floating point optimizations (for some compilers, this enables `-ffast-math`).	OFF
`USE_SIMD`	Enables compiler support for OpenMP SIMD directives (for `gcc`, `icc`, and `clang` compilers this will NOT enable other OpenMP directives and should not link the openmp runtime library). Enabling this requires that `OPTIMIZE_FP=ON`.	OFF

Profiling Options

These options control compilation choices that can be used to facillitate profiling in Enzo-E / Cello

Profile-Related Configuration
Name	Description	Default
`memory`	Track dynamic memory statistics. Can be useful, but can cause problems on some systems that also override `new [] ()` / `delete [] ()`	OFF
`use_gprof`	Compile with -pg to use gprof for performance profiling	OFF
`use_performance`	Use Cello Performance class for collecting performance data (currently requires global reductions, and may not be fully functional) (basic time data on root processor is still output)	ON
`use_projections`	Compile the CHARM++ version for use with the Projections performance tool.	OFF

Testing Options

These options configure properties of automated tests. These options currently just affect tests in the ctest framework and don’t affect tests in the pytest framework.

Testing-Related Configuration
Name	Description	Default
`PARALLEL_LAUNCHER`	Launcher to use for parallel tests	charmrun
`PARALLEL_LAUNCHER_NPROC_ARG`	Argument to set number of processing elements for parallel launcher (for use with `charmrun`)	+p
`PARALLEL_LAUNCHER_NPROC`	Number of processors to run parallel unit tests	4
`BUILD_TESTING`	Whether to setup the CTest infrastructure and build unit test binaries (which are primarily built to be executed by the CTest infrastructure). This has no effect on the pytest infrastructure.	“ON”

Debugging Options

The following options are useful for debugging.

Debug Options
Name	Description	Default
`check`	Do extra run-time checking. Useful for debugging, but can potentially slow calculations down	OFF
`debug`	Whether to enable displaying messages with the DEBUG() series of statements. Also writes messages to out.debug.<P> where P is the (physical) process rank. Still requires the “DEBUG” group to be enabled in `Monitor` (that is `Monitor::is_active("DEBUG")` must be true for any output)	OFF
`debug_field`		OFF
`debug_field_face`		OFF
`debug_verbose`	Print periodically all field values. See `src/Field/field_FieldBlock.cpp`	OFF
`trace`	Print out detailed messages with the TRACE() series of statements	OFF
`trace_charm`	Print out messages with the TRACE_CHARM() and TRACEPUP() series of statements	OFF
`verbose`	Trace main phases	OFF

Misc Options

The following options don’t really belong in any other category

Misc Options
Name	Description	Default
`USE_PRECOMPILED_HEADERS`	Precompile headers to try to reduce compile time	ON

`ENZOE_<LANG>_FLIST` Variables

We strongly encourage users and developers to make use of the options described in the preceding subsections. They exist to provide a curated/consistent experience in a variety of scenarios. (If you think that some option is missing, please let us know!)

With that said, we also recognize that the need may arise where a user/developer may want to specify arbitrary flags. We provide the ENZOE_<LANG>_FLIST variables for this purpose (where <LANG> is C, CXX, Fortran). Flags passed to this variable will be passed to the compiler while compiling source files (that are written in <LANG>) that are directly used in the Cello/Enzo-E libraries and resulting executable. Here are 2 illustrative examples:

First we show that in order to pass multiple flags, the flags need to be specified by a semicolon delimited list. If you stored "-Wall;-Wpedantic;-funroll-loops" within the ENZOE_CXX_FLIST variable, then all C++ files used to build Cello and Enzo-E would be passed those flags.

Next we show that to properly pass “options groups” you may need to make use of shell-like quoting with the SHELL: prefix (this is required because of option de-duplication performed by cmake). Thus storing "SHELL:-option1 A;-Wall;SHELL:-option2 B" within ENZOE_Fortran_FLIST would cause all Fortran files used in Enzo-E and Cello to be passed -option1 A -Wall -option2 B.

CMake offers a similar set of standard variables named CMAKE_<LANG>_FLAGS that serve a similar purpose, but they behave slightly differently. Most importantly, flags passed CMAKE_<LANG>_FLAGS will be used to specify flags that are passed to ALL source-files (of the given language) compiled in a given build – including the source files of any external dependencies that the build is automatically compiling (in the near future, this may include Grackle). The contents of CMAKE_<LANG>_FLAGS are also passed to the compiler front-end during linking. Additionally, the contents of CMAKE_<LANG>_FLAGS are command line snippets (i.e. options are separated by whitespace rather than semi-colons and no shell-quoting is needed).

Specifying Configuration Options

All variables can be set either on the command line by -D<variable>=<value> or in a machine config, see below. For example, a configure line may look like

cmake -DCHARM_ROOT=$(pwd)/../../charm/build-gcc-mpi-proj -DEnzo-E_CONFIG=msu_hpcc_gcc -DGrackle_ROOT=${HOME}/src/grackle/build-gcc -Duse_projections=ON -Duse_jemalloc=ON -Dbalance=ON  ..

To see all available (and selected) options you can also run ccmake . in the build directory (after running cmake in first place), or use the ccmake GUI directly to interactively configure Enzo-E by calling ccmake .. in an empty build directory.

If packages (external libraries) are not found automatically or if the wrong one was picked up, you can specify the search path by -D<package_name>_ROOT=/PATH/TO/PACKAGE/INSTALL, cf., the cmake example command just above. Note:

these package locations are also picked up from the environment, i.e., an alternative option is export <package_name>_ROOT=/PATH/TO/PACKAGE/INSTALL .
to specify the path to a libpng install, use -DPNG_ROOT=/PATH/TO/LIBPNG instead of -DLIBPNG_ROOT=....

The last option is a machine specific configuration file (see below).

In addition, the general cmake option is available to set basic optimization flags via CMAKE_BUILD_TYPE with values of:

Release (typically -O3)
RelWithDebInfo (typically -O2 -g)
Debug (typically -O0 -g)

The first choice is generally fastest, while the second is a sensible choice during development (the compiler performs most optimizations and includes debugging information in the executable)

Note that setting -DOPTIMIZE_FP=ON will modify each of these build types to include value-unsafe floating point optimizations.

Machine files

Finally, for convenience we provide the option to set default value for your own machine/setup, see the *.cmake files in the config directory.

You can specify compilers and option in there that will be used a default when cmake is called with -DEnzo-E_CONFG=my_config_name where my_config_name requires a corresponding config/my_config_name.cmake to exist. The alternative directory for the machine configuration files is a .enzo-e directory in your home directory. If a file with the same name exists in your ${HOME}/.enzo-e directory and in the config directory only the first one will be used. Note, all command line parameter take precedence over the default options. In other words, if USE_DOUBLE_PREC is ON in the machine file (or even automatically through the global default), the running cmake -DEnzo-E_CONFIG=my_config_name -DUSE_DOUBLE_PREC=OFF .. will result in a single precision version of Enzo-E.

Some of these files may also initialize the ENZOE_<LANG>_FLIST_INIT variables. When defined, these variables are used to provide defaults for the ENZOE_<LANG>_FLIST variables, which are discussed up above. As discussed above, these variables provide a mechanism to quickly and easily provide extra flags to the compiler. Ideally you should only need to rely upon these for quick-and-easy-tests and the other options should meet most of your needs. If you find the existing options don’t meet your needs, please let us know

Options in the machine file can also include the paths to external libraries and can be set via a “cached string”, i.e., via

set(CHARM_ROOT "/home/user/Charm/charm/build-mpi" CACHE STRING "my charm build")

Advanced Options

It is worth mentioning that when using automatic dependency management of Grackle, the default behavior is to freshly download the Grackle repository every time you configure a new build-directory. You should NOT modify the downloaded source files since they can be freely overwritten at any time.

If you want to codevelop a feature in Grackle and Enzo-E, you can check out a local copy of Grackle and configure Enzo-E with -DFETCHCONTENT_SOURCE_DIR_GRACKLE=<path/to/grackle/repository>. By doing this, your Enzo-E build will directly use the source files from your Grackle repositories (and any changes you make to them will be reflected in the resulting Enzo-E binary).

Running

In this section we run Enzo-E on a simple “Hello World” test program and take a look at Enzo-E’s output.

1. Run Enzo-E

An included “Hello World” problem can be run using the following from the $CELLO_HOME directory:

charmrun +p4 bin/enzo-e input/HelloWorld/Hi.in

This assumes that the charmrun command is in your path. If it is not, then you will need to include the path name as well, e.g.:

~/Charm/bin/charmrun +p4 bin/enzo-e input/HelloWorld/Hi.in

This also assumes that local connections can be established passwordless. If errors like

Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password,hostbased).
Charmrun> Error 255 returned from remote shell (localhost:0)

are displayed, a node local run (i.e., no “remote” connections even to the local host) could be used instead by adding ++local to charmrun, e.g.:

~/Charm/bin/charmrun ++local +p4 bin/enzo-e input/HelloWorld/Hi.in

If you receive an error like

Charmrun> Timeout waiting for node-program to connect

try running ./bin/enzo-e without charmrun as crashes due to, e.g., libraries not being found may not be displaying.

If all goes well, Enzo-E will run the Hello World problem. Below are some of the generated images from the longer-running “HelloWorld.in” problem (note “HelloWorld.in” runs for about an hour, compared to a couple minutes for the shorter “Hi.in” input parameter file). These images show density and mesh hierarchy structure with blocks colored by level and by age.

Time = 0.00

Time = 0.05

Time = 0.10

If you look at the Hi.in parameter file contents, you will notice that there are some "include" directives that include other files. When Enzo-E / Cello runs, it will generate a "parameters.out" file, which is the input file but with the included files inlined. This "parameters.out" file is itself a valid Enzo-E / Cello parameter file (though you may wish to rename it before using it as a parameter file to avoid it being overwritten.)

If you encounter any problems in getting Enzo-E to compile or run, please contact the Enzo-E / Cello community at the users’ mailing list and someone will be happy to help resolve the problems.

2020-04-10: Updated with corrections from Joshua Smith.