I've tried to make Address, Phone, etc only be for Billing but it still shows up for each Group member. On top of that. I told it to be required and it's not required as a field either.
You may have only set the global defaults for those fields. Did you also set the specific settings in the events? For each event you have to go to the Event's Fields tab and set how those fields will behave for that event.