Your primary team is where you can speak freely and be who you are and can learn a lot for your career. That’s why a manager’s primary team is not the one they manage but the one they report to the same manager. And that’s fine. Your direct reports already have a lot of people to hang out with.
When you manage a team, every word that comes out of your mouth counts; even thinking out loud can cause confusion and sometimes frustration. You may not speak freely, but you should let your team express their opinions freely. Words that you will forget after you said them might not be forgotten that easily by your team.
You can certainly learn from the team you manage, but only your primary team can connect with you and relate to your experiences at your level and point of view. This is where you can and should express your opinions even if they are not coherent. This is where you can get help from others for the problems you face in your teams without causing unintended consequences among your groups.
Those who are managing managers should build strong relationships among their management team. Let them learn from one another. Let them work together rather than just turning them into reporting mates. It will take time if your team is not used to working like that. Strong managers don’t let their management team be controlled by politics when they speak to one another.