I’m on the road right now, headed to Wyoming for the dinosaur excavation, so this will be a short entry.
My lecture at the Aurora Fossil Museum last Saturday was on squalodont whales. I’ve long had an interest in this group, as I studied them for my doctoral dissertation. The squalodonts (literally, “shark-toothed”) are a primitive group of toothed whales known from the Oligocene to the Middle Miocene. In eastern North America they have been described from New Jersey to North Carolina.
The reconstruction above depicts the largest known squalodont, Squalodon whitmorei, which I named in 2005 (shown attacking the small toothed cetacean Xiphiacetus). Squalodon whitmorei is best known from 2 skulls from King George County, Virginia, although fragments have also been found in Maryland and North Carolina. The holotype specimen (the one that forms the basis for the species) was collected in 1975 from King George County and is stored at the Smithsonian Institution. It is one of the most complete squalodont skeletons ever found:
Squalodonts were a small but significant component of the marine fauna off the east coast in the Early to Middle Miocene. S. whitmorei is only found in deposits that are around 18 million years old, but other species survived until around 15 million years ago before the entire family went extinct. They were the last surviving group of whales with a strongly heterodont dentition, meaning that the teeth were very different shapes from the front of the mouth to the back.
Updates from the Paleontology Lab 
Very interesting, I did not know that there were even that big squalodontids. It also seems that there is some kind of ideal size for those probably macropredatory whales, including modern orcas, killer sperm whales or Macrodelphinus. Are there any other big predatory whales known which would fit in this group?
I already asked me what purpose the strange incisors of squaldonts had. Had they any relation to their way of hunting or were they probably used for interspecific figths, similar to those seen in many modern toothed whales. They are also the only whales I know, which had paired horizintally orientated incisors. I already speculated that the ancestors of modern narwhales had one or more pairs of small horizontal incisors (perhaps even in their lower jaw, as even cases of modern narwhales are known, which had an additional tusk in the mandible, could this be an atavism?).
I’m on the road right now, so my response might be lacking in detail (it’s also slow to type on my phone). That’s a good point, Markus, that a lot of the big predatory odontocetes are similar in size (S. whitmorei, Macrokentriodon, Orcinus, Macrodelphinus, the killer sperm whales). This could be a structural limitation, but I doubt it (Physeter is much bigger). Could be that prey size might be a limiting factor; I’ll have to think about that. One point, I’d put S. calvertensis and Pseudorca in the same group, and they’re much smaller.
I’ve stopped driving for the night, so now I can talk about procumbent incisors (incisors that stick forward out of the mouth instead of pointing up or down).
I think procumbent incisors may actually be a primitive characteristic of odontocetes. Besides squalodonts, they are also found in at least some kentriodontids, Waipatia, the squalodelphinids (I think), and some physeterids (there might be others, but that’s off the top of my head). I’ve even seen them in the modern bottlenose dolphin Tursiops, but they seem to usually be worn down to nubs.
That said, the condition reaches an extreme in the Squalodontidae. All three incisors are partly procumbent, and the first incisor completely so. The first incisor is also hyper-enlarged; if you include the root, its length can be almost 10% of the entire length of the skull (the crown is only about 1/3 of the total tooth length, though). The root is so long that it extends back as far as the crown of the 3rd incisor, which is one of the characters that I think defines the family.
As Markus suggested, I think these teeth were used for intraspecies combat rather than for feeding. The use of teeth (often specialized) in mating combat or rituals is common in whales, including narwhals and beaked whales. Sperm whales and dolphins also show these behaviors, even though they don’t have specialized teeth for it. So it makes a lot of sense for Squalodon. In addition, the incisors seem to shaw different wear patterns than the rest of the teeth (even the anterior ones), which is also suggestive.
On the other hand, with this hypothesis in mind, a few years ago I tried to test this by looking for evidence of sexual dimorphism (basically, to see if males had proportionally larger teeth than females). My results were completely negative; the size of the incisors was always directly related to the size of the animal, in the same proportions. Even so, I still suspect they were used for mating combat.
Thank you very much for your long and informative response. In the case of sizes I think this is mainly prey-oriented. Among modern orcas there is still a big variation between the different populations, and the largest ones are those which mainly hunt other tetrapods. It is also interesting that there is a big increase in overall bulkiness among this wide variation of similar sized but completely unrelated toothed whales. Perhaps even Zygorhiza should be included in this group, and how knows if we will find bigger relatives of Janjucetus. It is interesting that Basilosaurus seems to be the only much larger toothed whale which was not mainly a squid-and fish-eater.
I suppose the reason why modern sperm whales are so big is just because they are spezialized in squid hunting, what makes them more similar to the big beaked whales or giant forms of ichthyosaurs, which were also most probably spezialized squid eaters with very small teeth but a very big hyoid like beaked whales or sperm whales. Even given the fact that we still don´t know how exactly Physeter hunts, it must be a extremely effient hunting strategy, if an animal which can weigh more than 50 tons can life from a diet of squids and fish which are in general not longer than a half metre. Compared to their size, sperm whales are even mainly micropredators (besides giant and sometimes colossal squids, but even they are very small compared to sperm whales and in general only caught in small numbers). Okay, many orcas eat also a lot of comparable small fish like herings or salmons, but the way in which they hunt this fish is surely completely different.
Something I noted about the teeth of narwhales is that not only double tusks occur still a comparable highly percentage of individuals, but also that females have comparably often a tusk. Could this be a hint that once both males and females had small tusks which were not used for fighting between the males but for something different?
I once wrote a short article about the possible evolution of narwhales, and made also a small model of a hypothetic proto-narwhale:
http://bestiarium.kryptozoologie.net/artikel/die-evolution-des-narwals/
My friend Darren Naish informed last year or so about a highly intereting paper about skulls from Greenland which seems to come from hybrids between belugas and narwhales, but sadly I have only the abstracts, as I could nowhere find a free version of this article.
I also noted that the first lower pair of Zygophyseter´s incisors possibly looked a bit out of its mouth. I was free to give my reconstruction drawing of Zygophyseter some double-scars on its skin from hypothetic interspecific fights.
Both Zygophyseter and Brygmophyseter seem to have procumbent incisors.
Narwhals are interesting, with the double tusks and the female tusks. I’ve wondered if it could be due to variation in testosterone levels. Female hyaenas develop secondary male characteristics, apparently due to high testosterone levels.
But in hyenas (at least Crocuta crocuta) all females have very high testosterone levels. It seems to be a very rare case (at least among tetrapods) that one gender bears in general typical characteristics of the other gender, like the antlers of female reindeers.
Interestingly, all females have two small tusks, but they normally remain in the bone (seems to be similar to the teeth of female sperm whales).
Hallo Alton! Just to revive this older post and wanted to add again something. I read some time ago that in ziphiids even the females show often battle-scars and sometimes fight with other members of their species, even if their teeth are often not as big as in the males. So perhaps it was similar in squalodonts, the first incisors were actually for interspecific fights, but not only among males but also among females. This would explain why they were found in all the specimens.