I sent DTH a ticket last week. I've been pulling my hair out trying to figure this one out.
Here's what I found:
Paypal has been randomly rolling out their "New Paypal Checkout" (says so at the top of the Paypal login page). They're randomly rolling it out more often than before, so that it just about shows up all the time. The random roll out occurs on both the live site and the sandbox site.
I was able to get the Classic Paypal Checkout to reappear only once in a great while after repeatedly trying different browsers with caches cleared prior to loading paypal.com.
The "New Paypal Checkout" fails every time you return to the joomla/dtregister site.
The Classic (old reliable) Paypal Checkout used to have a BUTTON to return to the joomla site. The New Paypal Checkout has a HYPERLINK.
The Classic Paypal, when you hit the button to return to the joomla site, would POST a FORM to the return page. The New Paypal DOES NOT POST ANYTHING--it simply hyperlinks you to the return page. The return page is specified in the initial form posting to paypal ("Please wait while you are being redirected to PayPal to complete your payment . . .").
Paypal is aware of many tickets about this issue. It is not joomla nor dtregister specific. Paypal says the work around is to use "PDT," whatever that may be or however difficult it will be to implement on joomla/dtregister. Here's what I found on a paypal forum:
In addition, there is a known issue with 'rm=2' not posting back transaction details with the new PayPal checkout pages. This is a known issue that is currently being addressed. A workaround will be to use PDT to get the transaction data on the return page. Information on PDT can be viewed at:
https://www.x.com/community/ppx/pdt
Changing joomla's global configuration, mail settings to phpmail, sendmail or smtp does not fix this issue.
Emails ARE being sent to the registrant as well as the event administrator after paypal processes the "pay now" button press.
While the records on my site are updated, the same page that says "could not instantiate mail function," those fields are not populated with the name or address and any tags like [EVENT_NAME] are ignored. Also, the message displayed below "could not instantiate..." is the GLOBAL dtregister "thanks" message. The event-specified thanks message, if there is one, is ignored, it seems.
Hope this helps.
joomla 1.5.15, dtregister 2.6.9. I get the same error in dtreg 2.6.8e.
(couple of edits to polish up the info).