Getting started using Enzo-E¶
This page will help you get Enzo-E and Cello up and running. It covers downloading the source code, porting the code to new platforms, configuring and compiling the code, and running a sample test problem.
Other pages are available for helping to get started on specific architectures, including the “Comet” supercomputer at San Diego Supercomputing Center (SDSC), and the NSF Petascale supercomputer “Blue Waters” at the National Center for Supercomputing Applications (NCSA).
Enzo-E / Cello is currently hosted on github.com (previously bitbucket.com)in beta-testing. To obtain the latest
version of the source code, you may clone it from the
repository Enzo-E / Cello github repository:
git clone https://github.com/enzo-project/enzo-e.git
Enzo-E / Cello, you may also need to download
and install 1.``Charm++``, 2.``HDF5``, 3.``libpng``, 3.``libboost``, and 4.``scons``:
Charm++ can be downloaded here.
To build Charm++ on a Mac, try this:
./build charm++ netlrts-darwin-x86_64 -j4 --with-production
If you’re running Linux, try building Charm++ using this:
./build charm++ netlrts-linux-x86_64 -j4 --with-production
Charm++ also provides a helpful interactive script
smart-build.pl that can
be used to configure and install
Charm++ on numerous other platforms.
“HDF5 is a “data model, library, and file format for storing and managing data”, and is the primary library used by Enzo-E / Cello for data output.
If HDF5 is not already installed on your machine, it may be available through your operating system distribution, otherwise it can be downloaded from the HDF5 website. Enzo-E / Cello currently uses the “serial” (non-MPI) version of HDF5.
“libpng is the official PNG reference library”, and is the image format used by Enzo-E / Cello.
libpng is not already installed on your machine, it may be
available through your operating system distribution, otherwise it can
be downloaded from the libpng website.
“Boost provides free peer-reviewed portable C++ source libraries.”
libboost-dev is not already installed on your machine, it may be
available through your operating system distribution, otherwise it can
be downloaded from the libboost website.
“SCons” is a “software construction tool” that is
used to build
Enzo-E / Cello using scripts written in
scons can be install via
pip or if you are using Anaconda or Miniconda
conda. It may also be pre-packed for installation for your operating system
Enzo-E / Cello requires a version of
for Python 3. See Installing SCons for
There are currently two configuration settings that must be set before
building Enzo-E and Cello:
CELLO_ARCH to specify the computer platform,
CELLO_PREC to specify the floating-point precision. Additionally,
some configurations require
CHARM_HOME to be specified.
Other optional settings are available as well. These are initialized in the top-level SConstruct file under the “USER CONFIGURATION” heading. Please consult the SConstruct file itself for documentation on these optional settings.
1. Specify machine¶
Before compiling Enzo-E / Cello, you first have to define which set of
available machine-dependent configuration settings to use via the
CELLO_ARCH environment variable. Some examples include the
compile for a generic GNU Linux system
compile for NCSA’s Blue Waters Petascale Platform
compile for SDSC’s Comet cluster using GNU compilers
Note that some machines, including Blue Waters and Comet, will additionally require certain modules to be loaded, including compilers and HDF5. See the corresponding getting started pages Getting Started: SDSC Comet or Getting Started: NCSA Blue Waters for architecture-specific details.
See the porting section below for how to add your own architecture to the list, or how to modify the settings for these machines.
2. Specify precision¶
Enzo-E must be compiled either using single-precision or
double-precision. To specify the precision to use in Enzo-E, set the
CELLO_PREC environment variable:
32-bit Enzo field data
64-bit Enzo field data
3. Specify Charm++ directory¶
The location of the Charm++ installation directory should be specified
CHARM_HOME environment variable.
Set directory of Charm++ used
4. Specify Grackle directory¶
At compile time, Enzo-E will try to automatically find your Grackle installation.
If compilation fails because
grackle.h cannot be included, it is possible
that the directory was incorrectly identified. You can specify
Grackle’s installation directory with the
GRACKLE_HOME environment variable:
Set directory of Grackle installation
If Cello does not support your desired hardware platform, you will
have to provide settings so that the build system knows what compilers
to use, plus any optimization settings, library paths, etc. This is
done by creating a new machine configuration file in the
subdirectory, and editing the top-level
SConstruct file to read
your new machine configuration file.
1. Create config file¶
Machine configuration files are found in the
To create a new one for your machine, try copying an existing one
and editing it. Configuration variables include the following:
This must be set to 1
Compiler flags, such as optimization settings or warning levels
Link flags, such as optimization settings
The C compiler to use
The Fortran compiler to use
Fortran flags for using single-precision by default
Fortran flags for using double-precision by default
Path for any Fortran libraries required when linking with C++
Fortran libraries required when linking with C++
Path to the
Path to PAPI performance library include files (OPTIONAL)
Path to PAPI performance library library files (OPTIONAL)
Path for HDF5 include files
Path for HDF5 library files
Path to the
Path to the Grackle chemistry and cooling library (OPTIONAL)
To incorporate your new machine configuration file into the
Cello build system, insert a new line to the following list in the
SConstruct file in the top-level
Enzo-E / Cello directory.
(Specific names in the
SConstruct file and those in the list
fragment below may differ due to the file being updated.)
if (arch == "gordon_gnu"): from gordon_gnu import * elif (arch == "gordon_intel"): from gordon_intel import * elif (arch == "gordon_pgi"): from gordon_pgi import * elif (arch == "comet_gnu"): from comet_gnu import * elif (arch == "linux_gnu"): from linux_gnu import * ...
After configuring Cello for your computer, the Cello framework and
Enzo-E application can be compiled using “make”, which in turn calls
./build.sh script. Other options are available for
generating useful https://orgmode.org/ files, generating
doxygen documentation, running
regression tests, and running code analysis tools.
Compile Enzo-E as
Remove object and test files
Clear any settings from an incomplete build
Generate doxygen documentation from source in
Run regression tests
Compute code quality metrics in
Compile Enzo-E / Cello using the Coverity static analysis tool
In this section we run Enzo-E on a simple “Hello World” test program and take a look at Enzo-E’s output.
1. Run Enzo-E¶
An included “Hello World” problem can be run using the following
charmrun +p4 bin/enzo-e input/HelloWorld/Hi.in
This assumes that the
charmrun command is in your path. If it
is not, then you will need to include the path name as well, e.g.:
~/Charm/bin/charmrun +p4 bin/enzo-e input/HelloWorld/Hi.in
This also assumes that local connections can be established passwordless. If errors like
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password,hostbased). Charmrun> Error 255 returned from remote shell (localhost:0)
are displayed a node local run (i.e., no “remote” connections even to the local host)
could be used instead by add
~/Charm/bin/charmrun ++local +p4 bin/enzo-e input/HelloWorld/Hi.in
If you receive an error like
Charmrun> Timeout waiting for node-program to connect
charmrun as crashes due to, e.g.,
libraries not being found may not be displaying.
If all goes well, Enzo-E will run the Hello World problem. Below are some of the generated images from the longer-running “HelloWorld.in” problem (note “HelloWorld.in” runs for about an hour, compared to a couple minutes for the shorter “Hi.in” input parameter file). These images show density and mesh hierarchy structure with blocks colored by level and by age.
Time = 0.00
Time = 0.05
Time = 0.10
If you look at the
Hi.in parameter file contents, you will notice that there are some
"include" directives that include other files. When Enzo-E / Cello runs, it will generate a
"parameters.out" file, which is the input file but with the included files inlined. This
"parameters.out" file is itself a valid Enzo-E / Cello parameter file (though you may wish to rename it before using it as a parameter file to avoid it being overwritten.)
If you encounter any problems in getting Enzo-E to compile or run, please contact the Enzo-E / Cello community at email@example.com, and someone will be happy to help resolve the problems.
2020-04-10: Updated with corrections from Joshua Smith.