Nested grouping elements using XSLT

Following XSLT

<?xml version='1.0'?>
<xsl:stylesheet version="2.0"  xmlns="http://www.w3.org/1999/xhtml"
     xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
     xmlns:RSUITE="http://www.reallysi.com" 
     xmlns:m="http://www.w3.org/1998/Math/MathML" 
     xmlns:epub="http://www.idpf.org/2007/ops" 
     epub:prefix="index: http://www.index.com/" 
     exclude-result-prefixes="RSUITE epub m">
<xsl:output method="xml" indent="yes" omit-xml-declaration="yes"/>
  <xsl:template match="/">
    <toc>
      <div class="prelims">
        <xsl:apply-templates select="//toc.entry[@level='prelims']"/>
      </div>
      <xsl:for-each select="//toc.entry[@level='part']">
        <div class="part">
          <xsl:apply-templates select="." mode="part">
            <xsl:with-param name="positionPart" select="position()"/>
          </xsl:apply-templates>
        </div>
      </xsl:for-each>
      <div class="endmatter">
        <xsl:apply-templates select="//toc.entry[@level='endmatter']"/>
      </div>
    </toc>
  </xsl:template>
  <xsl:template match="toc.entry" mode="part">
  <xsl:param name="positionPart"/>
   <xsl:apply-templates select="."/>
   <xsl:for-each select="following-sibling::*[@level='chapter'
        and not(count(preceding-sibling::*[@level='part']) > $positionPart)]">
      <xsl:variable name="positionChapter" select="position()"/>
        <div class="chapter">
          <xsl:apply-templates select="." mode="chapter">
            <xsl:with-param name="positionPart" select="$positionPart"/>
            <xsl:with-param name="positionChapter" select="$positionChapter"/>
          </xsl:apply-templates>
        </div>
   </xsl:for-each>
  </xsl:template>
  <xsl:template match="toc.entry" mode="chapter" >
  <xsl:param name="positionPart"/>
  <xsl:param name="positionChapter"/>
  <xsl:apply-templates select="."/>
  <xsl:apply-templates select="following-sibling::*[1][@level='author']"/>
    <xsl:for-each select="following-sibling::*[@level='levela'
      and not(count(preceding-sibling::*[@level='chapter']) > $positionChapter)]">
      <div class="levela">
        <xsl:apply-templates select="."/>
        <xsl:if test="following-sibling::*[@level='levelb'
             and not(count(preceding-sibling::*[@level='chapter']) > $positionChapter)]">
          <div class="levelb">
            <xsl:for-each select="following-sibling::*[@level='levelb'
                 and not(count(preceding-sibling::*[@level='chapter']) > $positionChapter)]">
              <xsl:apply-templates select="."/>
            </xsl:for-each>
          </div>
        </xsl:if>
      </div>
    </xsl:for-each>
  </xsl:template>
  <xsl:template match="toc.entry">
    <toc.entry>
      <xsl:attribute name="level" select="@level"/>
      <xsl:value-of select="."/>
    </toc.entry>
  </xsl:template>
</xsl:stylesheet>

when applied to your input XML produces the output

<toc xmlns="http://www.w3.org/1999/xhtml">
<div class="prelims">
  <toc.entry level="prelims">Half Titlepage</toc.entry>
  <toc.entry level="prelims">Titlepage</toc.entry>
  <toc.entry level="prelims">Imprint</toc.entry>
</div>
<div class="part">
  <toc.entry level="part">Part 1. This is part title</toc.entry>
  <div class="chapter">
     <toc.entry level="chapter">Chapter 1. This is chapter title</toc.entry>
     <toc.entry level="author">This is author</toc.entry>
     <div class="levela">
        <toc.entry level="levela">This is level a head</toc.entry>
        <div class="levelb">
           <toc.entry level="levelb">This is level b head</toc.entry>
           <toc.entry level="levelb">This is level b head</toc.entry>
           <toc.entry level="levelb">This is level b head</toc.entry>
        </div>
     </div>
  </div>
  <div class="chapter">
     <toc.entry level="chapter">Chapter 2. This is chapter title</toc.entry>
     <toc.entry level="author">This is author</toc.entry>
     <div class="levela">
        <toc.entry level="levela">This is level a head</toc.entry>
     </div>
  </div>
  <div class="chapter">
     <toc.entry level="chapter">Chapter 3. This is chapter title</toc.entry>
     <toc.entry level="author">This is author</toc.entry>
     <div class="levela">
        <toc.entry level="levela">This is level a head</toc.entry>
     </div>
  </div>
</div>
<div class="part">
  <toc.entry level="part">Part 2. This is part title</toc.entry>
  <div class="chapter">
     <toc.entry level="chapter">Chapter 4. This is chapter title</toc.entry>
     <toc.entry level="author">This is author</toc.entry>
  </div>
  <div class="chapter">
     <toc.entry level="chapter">Chapter 5. This is chapter title</toc.entry>
     <toc.entry level="author">This is author</toc.entry>
  </div>
  <div class="chapter">
     <toc.entry level="chapter">Chapter 6. This is chapter title</toc.entry>
     <toc.entry level="author">This is author</toc.entry>
  </div>
  </div>
  <div class="endmatter">
   <toc.entry level="endmatter">References</toc.entry>
   <toc.entry level="endmatter">Index</toc.entry>
  </div>
</toc>

I've kept the namespaces from your example XSLT, though they were not needed for the online XSLT processor I used (Saxon 9.5.1.6). In case it would be necessary to remove the xmlns-namespace from the toc root element in the output, it's possible to remove it from the stylesheet declaration.

Explanation: As the elements with the prelims and the endmatter level should be displayed at the beginning / the end of the XML, they are just applied in the main template matching the root in div containers with the appropriate class.
All part levels are applied to the template with the mode="part" and the position of the current part as parameter. To generate all chapters that belong to the current part, templates are only applied to all following sibling with the level chapter that do not have more preceding siblings with the level part as the current part element:

<xsl:for-each select="following-sibling::*[@level='chapter'
    and not(count(preceding-sibling::*[@level='part']) > $positionPart)]">
    ....

with the position of the current chapter as additional parameter.
The template mode="chapter" will only generate an author in case the next sibling of the chapter has the level author:

<xsl:apply-templates select="following-sibling::*[1][@level='author']"/>  

To generate only the levela level elements that belong to the current chapter, only the levela elements are selected that don't have more preceding chapters as the current chapter:

<xsl:for-each select="following-sibling::*[@level='levela'
     and not(count(preceding-sibling::*[@level='chapter']) > $positionChapter)]">
     ...

Similar approach was taken for the levelb level elements.
As it's not clear if it's possible that chapters don't have levela elements, I've only added a check for levelb before writing the div container:

<xsl:if test="following-sibling::*[@level='levelb'
    and not(count(preceding-sibling::*[@level='chapter']) > $positionChapter)]">
    <div class="levelb">
    ...

I've left the part with possible levelc elements out - in case this kind of approach would work for you it shouldn't be too much of a problem to add it.
In addition I want to notice that the result depends on the XSLT processor you use - e.g. when switching to Saxon 6.5.5, above XSLT won't generate the values for the level attributes.
For convenience or further testings I saved above example here: http://xsltransform.net/eiZQaF7