Well, that is the problem with "white feminism" (white supremacy within feminism that avoids intersectionality and applies white supremacist views to feminism, while simultaneously tone-policing/speaking over black women and black people as a whole). The reality is that although "white feminism" pretends that male privilege exists for all males, it simply isn't the case.
Any privilege that black men or women experience is purely a product of "conditional privilege" and not "institutionalized privilege". Conditional meaning that if you don't speak a certain way, go to the right schools, have the right career and amount of money, you will receive zero privilege. In addition to that, even if you attain conditional privilege, it can easily be taken away if you don't tow the line. In order to be a member of a patriarchy, you would have to maintain institutionalized privilege. In addition to that, for a patriarchy to exist as a praxis, white women would essentially have to be without privilege and oppressive power, which simply isn't the case.
This is something white feminism fails to realize and white it is named as such. And it is one of many reasons why womanism exists. You can support gender equality without maintaining any label (which I am sure you do). You can also disagree with plenty of white feminist opinions due to their lack of intersectionality. It's ok man, you aren't wrong and you are more than welcome to support gender equality in your own way. I would, however, suggest you take an interest in certain forms of womanism.