The so-called cetotheres have been a thorn in the side of cetacean taxonomists and systematists for the better part of a century. Traditionally the family Cetotheriidae has included almost any extinct baleen whale that can’t be assigned to the Balaenopteridae (rorquals), Eschrichtiidae (gray whales), or Balaenidae (right whales); in other words, it was a “dustbin” taxon where you could sweep away anything that’s inconvenient.
The nature of the cetothere taxa hasn’t helped. They tend to be very generalized in their characteristics, and what specializations they have tend to be limited to a single taxon. Good specimens are also not particularly common (although fragments are); many taxa are known from a single specimen, so variation isn’t well established (one paleontologist quipped that “Any well-preserved cetothere skull is automatically a new genus.)
Dustbin taxa are not particularly useful. They keep your database nice and neat, but they don’t really reveal much about the relationships of different organisms (although they do reveal our lack of knowledge about those organisms).
In recent years, the cetotheres have started to relinquish their secrets. A subgroup has been recognized by various workers, which include a suite of specializations not seen in other primitive baleen whales. Many of these are related to the structure of the ear and the lower jaw joint. Another character is the extension of the rostral bones (the maxilla and premaxilla) onto the top of the skull, posterior to the orbits and essentially reaching the supraoccipital. This is visible in the Peruvian whale Piscobalaena, above (specimen from Museo de Historia Natural). Compare the area between the orbits to the specimen from the USNM shown below, tentatively referred to Parietobalaena:
Based on this suite of similarities, Boutel and Muizon (2006) restricted the definition of Cetotheriidae to just 6 genera: Cetotherium, Mixocetus, Nannocetus, Metopocetus, Herpetocetus, and Piscobalaena. These became Cetotheriidae sensu stricto, as opposed to the old definition for the family, Cetotheriidae sensu lato. Taxa like Diorocetus and Parietobalaena became Mysticeti incertae sedis, meaning that we still don’t know where they fit (essentially, incertae sedis is the new, but somewhat emptier, dustbin).
Other workers have followed this same general pattern. Whitmore and Barnes (2008) assigned the following genera to the Cetotheriidae: Cetotherium, Metopocetus, Amphicetus, Heterocetus, Mesocetus, Cephalotropis, Mixocetus, Plesiocetopsis, Herpetocetus, and Nannocetus. They went a step further and divided the Cetotheriidae into two subfamilies, the Herpetocetinae with Herpetocetus and Nannocetus, the Cetotheriinae with everyone else. (Whitmore and Barnes did not address Piscobalaena.)
That still leaves a lot of taxa unaccounted for: Parietobalaena, Diorocetus, Aglaocetus, Pelocetus, Halicetus, Thinocetus, and possibly Eobalaenoptera, just to name the ones known from Virginia. (I’ve argued in the past that Eobalaenoptera is close to the balaenopterids, but my views have have changed somewhat and it may not be as close as I originally believed). Steeman (2007) made an effort at sorting these taxa out, and erected three new families to accommodate some of them. It’s not yet clear how these families will work out, but you have to start somewhere.
As an aside, the redefinition of Cetotheriidae has led to some awkward problems for writing and speaking about these whales. There’s really no question that the redefinition was in order, because Cetotherium, the type genus of the family, still falls within the group. The problem is that the name Cetotheriidae has a nearly century-long history with a different usage. In writing papers, authors now typically refer to Cetotheriidae s. s. or Cetotheriidae s. l. to make clear which definition they’re using, and sometimes refer to the “non-cetothere cetotheres” (!?) by using quotes (Cetotherium is a cetotheriid, while Diorocetus is a “cetothere”). It’s even tougher when speaking, unless we’re all going to make imaginary quotes in the air with our hands.
While I doubt this will ever be adopted, I would suggest the following change. Use the two subfamilies defined by Whitmore and Barnes (2007), Cetotheriinae and Herpetocetinae, (assuming they are valid taxa), and replace Cetotheriidae with Herpetocetidae. This actually has an historical basis, as the original name from 1872 was the subfamily Cetotheriinae; it was not elevated to a family until 1923. Herpetocetidae doesn’t have the historical baggage associated with Cetotheriidae. Even more ideal would be to replace Cetotheriinae as well, and only use the term cetothere in an informal sense. This is similar to what has happened with Reptilia, which as originally defined was paraphyletic. This is a more questionable suggestion, however, as Cetotherium clearly belongs in this taxon.
Make sure to check out the archives at The Coastal Paleontologist, too. Robert Boessenecker has put up a lot of posts with field and preparation photos of Herpetocetus, which is quite common in the Pliocene of California.