Getting started using Enzo-E
This page will help you get Enzo-E and Cello up and running. It covers downloading the source code, porting the code to new platforms, configuring and compiling the code, and running a sample test problem.
Other pages are available for helping to get started on specific architectures, including the “Frontera” supercomputer at TACC and the “Pleiades” supercomputer at NASA.
Dependency Installation
Before compiling Enzo-E / Cello
, you may need to download
and install 1. CMake
, 2. Charm++
, 3. HDF5
, 4. libpng
, and (optionally) 5. Grackle
:
1. Install CMake
Most systems nowaways have CMake
already installed.
If not, you can get the binary distribution from the
CMake website.
2. Install Charm++
We generally recommend that you download Charm++, from the GitHub repository and then retrieve the latest version (7.0.0) known to build Enzo-E/Cello. This is demonstrated by the following command:
git clone https://github.com/UIUC-PPL/charm.git
cd charm
git checkout v7.0.0
Alternatively, if you don’t have git installed, you can also download the appropriate release version of the code from the main website or from the Release page on GitHub.
- Charm++ can be configured to use different “backends” for handling communication, which include:
mpi
: our recommended default choice for distributed machines. Under this configuration, communication occurs via calls to MPI commands (this obviously requires an MPI installation).netlrts
: our recommended choice for development/testing on a local machine (this can make debugging far simpler). Under this configuration, communication occurs via standard networking protocols (that all modern computers are equipped with). This is generally slower than other options.
Charm++ uses cmake as the default build system. To build Charm++ start from directory holding all downloaded files.
Invoke the following command to build with the mpi backend:
mkdir build-mpi # this directory name is arbitrary
cd build-mpi
cmake -DNETWORK=mpi -DSMP=OFF ..
make # you can specify make -j4 to compile with 4 threads
The following commands to build with the netlrts backend:
mkdir build-netlrts # this directory name is arbitrary
cd build-netlrts
cmake -DNETWORK=netlrts -DSMP=OFF ..
make # you can specify make -j4 to compile with 4 threads
As an aside Charm++ can also be configured to directly use a machine’s low-level communication primitives (that MPI
implementations wrap).
It can be complicated to compile with these backends, so additional details are provided on an architechture-specific basis.
3. Install HDF5
“HDF5 is a “data model, library, and file format for storing and managing data”, and is the primary library used by Enzo-E / Cello for data output.
If HDF5 is not already installed on your machine, it may be available through your operating system distribution, otherwise it can be downloaded from the HDF5 website. Enzo-E / Cello currently uses the “serial” (non-MPI) version of HDF5.
4. Install libpng
“libpng is the official PNG reference library”, and is the image format used by Enzo-E / Cello.
If libpng
is not already installed on your machine, it may be
available through your operating system distribution, otherwise it can
be downloaded from the libpng website.
5. Install Grackle (Optional)
By default, Enzo-E requires the Grackle chemistry and cooling library.
For user-convenience, the default behavior is for Grackle to be automatically downloaded and compiled as part of an Enzo-E build.
Alternatively, if you want to make use of an existing Grackle installation (those installation instructions are provided here), you can set -DUSE_EXTERNAL_GRACKLE=ON
.
Note
If you choose to link Enzo-E against a pre-built version of Grackle, that version must have been built with CMake. (Enzo-E no longer supports linking against version of Grackle built with its “Classic Build System”)
If you do not need to use Grackle, you can disable it by setting -DUSE_GRACKLE=OFF
.
6. Install yt (Optional)
If you want to use the yt python package to analyse Enzo-E output data, you should install the latest version from source. This can be done with the following commands:
git clone https://github.com/yt-project/yt.git
cd yt
pip install -e .
This is NOT a requirement for building and running Enzo-E, but it is used in some tests.
Configuring/Building
Enzo-E / Cello
is currently hosted on github.com (previously bitbucket.com).
To obtain the latest version of the source code, you may clone it from the repository Enzo-E / Cello github repository:
git clone https://github.com/enzo-project/enzo-e.git
For a basic configuration on a linux system with the GNU compiler stack, the following command should work out of the box. If not, please report any problems. See the following subsection for more configuration options.
mkdir my-first-build # Again, the directory name can be anything
cd my-first-build
# replace <PATH/TO/charm/build-dir> with the path to the directory created
# when you built charm++ (either build-mpi or build-netlrts if you've been
# following along)
cmake -DCHARM_ROOT=<PATH/TO/charm/build-dir> \
-DEnzo-E_CONFIG=linux_gcc -DUSE_GRACKLE=OFF ..
make -j4 # -j4 tells make to execute up to 4 commands in parallel
To build on a Mac, you should only need to replace linux_gcc
with darwin_clang
.
Note, if ninja
is installed, the ninja
build system can be used for faster build times.
This is done by adding -GNinja
to the cmake
command (before the ..
) and calling ninja
afterwards instead of make
.
The Enzo-E executable is built within bin/
.
Configuration options
Current cmake
options are listed in the following subsubsections.
Skip ahead to Specifying Configuration Options for details about how to specify the configuration.
General Configuration
These options are related to Enzo-E’s general configuration.
Name |
Description |
Default |
---|---|---|
|
Use double precision. Turn off for single precision. |
ON |
|
Temporary setting for using new Output implementation |
OFF |
|
Maximum number of procesess per shared-memory node (can be larger than needed) |
64 |
|
Use Charm++ in SMP mode (Charm++ must have been compiled to support SMP mode). |
OFF |
|
Enable charm++ dynamic load balancing |
ON |
|
Charm++ load balancer included |
“CommonLBs” |
|
Charm++ load balancer to use by default |
“TreeLB” |
Configuring Dependencies
The parameters in the following table control whether Enzo-E / Cello
should make use of certain external dependencies.
These options all assume that cmake
is succesfully able to find the location of these dependencies.
In some cases, you may need to provide additional hints about the location of the dependencies (see Specifying Configuration Options for more details)
Name |
Description |
Default |
---|---|---|
|
Use Grackle Chemistry |
ON |
|
Indicates preference for using an existing grackle installation or directly embedding grackle within the Enzo-E build. |
OFF |
|
Use the jemalloc library for memory allocation |
OFF |
|
Use the PAPI performance API |
OFF |
Floating-Point Optimization Options
Generally, Enzo-E / Cello
will not be compiled with value-unsafe floating point optimizations unless they are explicitly enabled.
This default behavior was chosen to support automated tests that check symmetry preservation in certain components of the code (mostly related to hydro/MHD).
While this feature is not essential for all applications, it is very useful when debugging new features.
Note that the Intel compilers deviate from this behavior. By default, they enable more aggressive floating point optimizations by default (consequently the tests checking symmetry may fail). This is a large part of the reason that the Intel compilers have a reputation for producing faster code (in practice, this is not always the case).
Name |
Description |
Default |
---|---|---|
|
Enable value-unsafe floating point optimizations (for some compilers, this enables |
OFF |
|
Enables compiler support for OpenMP SIMD directives (for |
OFF |
Profiling Options
These options control compilation choices that can be used to facillitate profiling in Enzo-E / Cello
Name |
Description |
Default |
---|---|---|
|
Track dynamic memory statistics. Can be useful, but can cause problems on some systems that also override |
OFF |
|
Compile with -pg to use gprof for performance profiling |
OFF |
|
Use Cello Performance class for collecting performance data (currently requires global reductions, and may not be fully functional) (basic time data on root processor is still output) |
ON |
|
Compile the CHARM++ version for use with the Projections performance tool. |
OFF |
Testing Options
These options configure properties of automated tests. These options currently just affect tests in the ctest framework and don’t affect tests in the pytest framework.
Name |
Description |
Default |
---|---|---|
|
Launcher to use for parallel tests |
charmrun |
|
Argument to set number of processing elements for parallel launcher (for use with |
+p |
|
Number of processors to run parallel unit tests |
4 |
|
Whether to setup the CTest infrastructure and build unit test binaries (which are primarily built to be executed by the CTest infrastructure). This has no effect on the pytest infrastructure. |
“ON” |
Debugging Options
The following options are useful for debugging.
Name |
Description |
Default |
---|---|---|
|
Do extra run-time checking. Useful for debugging, but can potentially slow calculations down |
OFF |
|
Whether to enable displaying messages with the DEBUG() series of statements. Also writes messages to out.debug.<P> where P is the (physical) process rank. Still requires the “DEBUG” group to be enabled in |
OFF |
|
OFF |
|
|
OFF |
|
|
Print periodically all field values. See |
OFF |
|
Print out detailed messages with the TRACE() series of statements |
OFF |
|
Print out messages with the TRACE_CHARM() and TRACEPUP() series of statements |
OFF |
|
Trace main phases |
OFF |
Misc Options
The following options don’t really belong in any other category
Name |
Description |
Default |
---|---|---|
|
Precompile headers to try to reduce compile time |
ON |
ENZOE_<LANG>_FLIST
Variables
We strongly encourage users and developers to make use of the options described in the preceding subsections. They exist to provide a curated/consistent experience in a variety of scenarios. (If you think that some option is missing, please let us know!)
With that said, we also recognize that the need may arise where a user/developer may want to specify arbitrary flags.
We provide the ENZOE_<LANG>_FLIST
variables for this purpose (where <LANG>
is C
, CXX
, Fortran
).
Flags passed to this variable will be passed to the compiler while compiling source files (that are written in <LANG>
) that are directly used in the Cello/Enzo-E libraries and resulting executable.
Here are 2 illustrative examples:
First we show that in order to pass multiple flags, the flags need to be specified by a semicolon delimited list. If you stored
"-Wall;-Wpedantic;-funroll-loops"
within theENZOE_CXX_FLIST
variable, then all C++ files used to build Cello and Enzo-E would be passed those flags.Next we show that to properly pass “options groups” you may need to make use of shell-like quoting with the
SHELL:
prefix (this is required because of option de-duplication performed by cmake). Thus storing"SHELL:-option1 A;-Wall;SHELL:-option2 B"
withinENZOE_Fortran_FLIST
would cause all Fortran files used in Enzo-E and Cello to be passed-option1 A -Wall -option2 B
.
CMake offers a similar set of standard variables named CMAKE_<LANG>_FLAGS
that serve a similar purpose, but they behave slightly differently.
Most importantly, flags passed CMAKE_<LANG>_FLAGS
will be used to specify flags that are passed to ALL source-files (of the given language) compiled in a given build – including the source files of any external dependencies that the build is automatically compiling (in the near future, this may include Grackle).
The contents of CMAKE_<LANG>_FLAGS
are also passed to the compiler front-end during linking.
Additionally, the contents of CMAKE_<LANG>_FLAGS
are command line snippets (i.e. options are separated by whitespace rather than semi-colons and no shell-quoting is needed).
Specifying Configuration Options
All variables can be set either on the command line by -D<variable>=<value>
or
in a machine config, see below.
For example, a configure line may look like
cmake -DCHARM_ROOT=$(pwd)/../../charm/build-gcc-mpi-proj -DEnzo-E_CONFIG=msu_hpcc_gcc -DGrackle_ROOT=${HOME}/src/grackle/build-gcc -Duse_projections=ON -Duse_jemalloc=ON -Dbalance=ON ..
To see all available (and selected) options you can also run ccmake .
in the
build directory (after running cmake
in first place), or use the ccmake
GUI
directly to interactively configure Enzo-E by calling ccmake ..
in an empty build
directory.
If packages (external libraries) are not found automatically or if the wrong one was
picked up, you can specify the search path by
-D<package_name>_ROOT=/PATH/TO/PACKAGE/INSTALL
,
cf., the cmake
example command just above.
Note:
these package locations are also picked up from the environment, i.e., an alternative option is
export <package_name>_ROOT=/PATH/TO/PACKAGE/INSTALL
.to specify the path to a
libpng
install, use-DPNG_ROOT=/PATH/TO/LIBPNG
instead of-DLIBPNG_ROOT=...
.
The last option is a machine specific configuration file (see below).
In addition, the general cmake option is available to set basic optimization flags via CMAKE_BUILD_TYPE
with values of:
Release
(typically-O3
)RelWithDebInfo
(typically-O2 -g
)Debug
(typically-O0 -g
)
The first choice is generally fastest, while the second is a sensible choice during development (the compiler performs most optimizations and includes debugging information in the executable)
Note that setting -DOPTIMIZE_FP=ON
will modify each of these build types to include value-unsafe floating point optimizations.
Machine files
Finally, for convenience we provide the option to set default value for your own
machine/setup, see the *.cmake
files in the config
directory.
You can specify compilers and option in there that will be used a default when
cmake
is called with -DEnzo-E_CONFG=my_config_name
where my_config_name
requires
a corresponding config/my_config_name.cmake
to exist.
The alternative directory for the machine configuration files is a .enzo-e
directory
in your home directory.
If a file with the same name exists in your ${HOME}/.enzo-e
directory and in the config
directory only the first one will be used.
Note, all command line parameter take precedence over the default options.
In other words, if USE_DOUBLE_PREC
is ON
in the machine file (or even automatically
through the global default), the running
cmake -DEnzo-E_CONFIG=my_config_name -DUSE_DOUBLE_PREC=OFF ..
will result in a single
precision version of Enzo-E.
Some of these files may also initialize the ENZOE_<LANG>_FLIST_INIT
variables.
When defined, these variables are used to provide defaults for the ENZOE_<LANG>_FLIST
variables, which are discussed up above.
As discussed above, these variables provide a mechanism to quickly and easily provide extra flags to the compiler.
Ideally you should only need to rely upon these for quick-and-easy-tests and the other options should meet most of your needs.
If you find the existing options don’t meet your needs, please let us know
Options in the machine file can also include the paths to external libraries and can be set via a “cached string”, i.e., via
set(CHARM_ROOT "/home/user/Charm/charm/build-mpi" CACHE STRING "my charm build")
Advanced Options
It is worth mentioning that when using automatic dependency management of Grackle, the default behavior is to freshly download the Grackle repository every time you configure a new build-directory. You should NOT modify the downloaded source files since they can be freely overwritten at any time.
If you want to codevelop a feature in Grackle and Enzo-E, you can check out a local copy of Grackle and configure Enzo-E with -DFETCHCONTENT_SOURCE_DIR_GRACKLE=<path/to/grackle/repository>
.
By doing this, your Enzo-E build will directly use the source files from your Grackle repositories (and any changes you make to them will be reflected in the resulting Enzo-E binary).
Running
In this section we run Enzo-E on a simple “Hello World” test program and take a look at Enzo-E’s output.
1. Run Enzo-E
An included “Hello World” problem can be run using the following
from the $CELLO_HOME
directory:
charmrun +p4 bin/enzo-e input/HelloWorld/Hi.in
This assumes that the charmrun
command is in your path. If it
is not, then you will need to include the path name as well, e.g.:
~/Charm/bin/charmrun +p4 bin/enzo-e input/HelloWorld/Hi.in
This also assumes that local connections can be established passwordless. If errors like
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password,hostbased).
Charmrun> Error 255 returned from remote shell (localhost:0)
are displayed, a node local run (i.e., no “remote” connections even to the local host) could be used instead by adding ++local
to charmrun
, e.g.:
~/Charm/bin/charmrun ++local +p4 bin/enzo-e input/HelloWorld/Hi.in
If you receive an error like
Charmrun> Timeout waiting for node-program to connect
try running ./bin/enzo-e
without charmrun
as crashes due to, e.g.,
libraries not being found may not be displaying.
If all goes well, Enzo-E will run the Hello World problem. Below are some of the generated images from the longer-running “HelloWorld.in” problem (note “HelloWorld.in” runs for about an hour, compared to a couple minutes for the shorter “Hi.in” input parameter file). These images show density and mesh hierarchy structure with blocks colored by level and by age.
Time = 0.00



Time = 0.05



Time = 0.10



If you look at the Hi.in
parameter file contents, you will notice that there are some "include"
directives that include other files. When Enzo-E / Cello runs, it will generate a "parameters.out"
file, which is the input file but with the included files inlined. This "parameters.out"
file is itself a valid Enzo-E / Cello parameter file (though you may wish to rename it before using it as a parameter file to avoid it being overwritten.)
If you encounter any problems in getting Enzo-E to compile or run, please contact the Enzo-E / Cello community at the users’ mailing list and someone will be happy to help resolve the problems.
2020-04-10: Updated with corrections from Joshua Smith.