Member Highlight: The HDF Group

The mission of the HDF Group is to develop and maintain high quality software for managing large complex data, to provide outstanding services for users of these technologies, and to insure effective management of data throughout the data life cycle.  HDF refers to “hierarchical data format” and the software that makes it possible to access data in the format.  HDF actually consists of two products, HDF4 (the original HDF) and HDF5 (HDF4’s successor).  HDF was first developed in 1988 to help scientists move their data among a variety of supercomputing platforms. Because it is a very general format, HDF quickly found users in almost every engineering and scientific discipline. It is now used worldwide in almost every application domain to manage high volume and varied data.

The HDF project was a part of the National Center for Supercomputing Applications at the University of Illinois from 1988 until 2006, when it became a non-profit company based in Champaign, IL.

Interesting Project: 

The largest number of applications of HDF are in the Earth Sciences. The biggest user of HDF over the past 20 years has been the Earth Observing System (EOS).  The vast majority of EOS data products are stored in either HDF4 or HDF5. Many of these are described as being in the HDF-EOS format, but it’s just HDF4 or HDF5 with an Earth Science-friendly API.

The Joint Polar Satellite System also uses HDF5, as do many other NASA satellite missions, as well as several satellites outside of the US.  Others use HDF5 in the form of netCDF-4, an API that uses the HDF5 format under the hood.

HDF5 is also widely used in the High Performance Computing community. A recent HDF5 application was a trillion-particle plasma physics simulation that produced a 350 terabyte dataset.  The dataset was stored in a handful of HDF5 files of about 40 TB each.

HDF5 is used in many other fields, such as high frequency stock trading, flight testing, oil and gas, animated film production, earthquake simulations, gene sequencing, radio astronomy, plasma physics, and the study of lunar dust.

Why we participate in ESIP? 

ESIP has the best combination of people and organizations for us to interact with in the earth science community. ESIP has a blend of organizations that we need to work with and understand, from small vendors to big data centers. The ESIP meetings are an excellent venue for us to interact with our colleagues. We participate in committees, clusters, and working groups that address some of the most important data and information challenges in Earth Science, and in science generally.

HDF and NASA have shared information and ideas in an annual workshop since 1997. Last summer the Workshop was held in conjunction with the ESIP meeting. This is a great opportunity for The HDF Group to provide information and learn from the entire ESIP community. This summer we are planning sessions on Services, Python and Data Conventions, as well as sharing ideas about new products. A lot of good things grow from seeds planted at ESIP meetings!

Link: http://www.hdfgroup.org/
Type: II
Joined: 2003
Voting Rep: Ted Habermann