Hello, first note that there is no setting for a maximum number of groups, but it's just a setting for a maximum group size.
Second, while still in Joomla 1.5, make sure you update to the latest version of DT Register as some important session related fixes have been made, which may have contributed to your problem.
Third, the most important thing you'll need to do is make plans to migrate to Joomla 2.5 or later. The improvements to DT Register since the 1.5 version is massive... and issues like you had are pretty much unheard of these day. Thanks!