Arguments Against Interest Groups

For every interest group I agree with there's an equal amount I probably disagree with, but that's just kind of politics. It'd be unfair to shut down some group but not others because no matter how you do it there will be some sort of bias. For example, if we shut down all interest groups have a large amount of money that shows a clear bias that they can't use their wealth to influence politics, and money is a form of free speech. I also can't say I'm against having countless amounts of groups because if we were to limit how many groups there could be I wouldn't be able to form my own group if I wanted, and it would further remove the power held by citizens. I'm not against interests groups with large amounts of capital, but I am against greedy politicians, if that makes sense. I understand that money can be a critical part of a lobbying campaign or other efforts carried out by an interest group and money shows support for the group. I'm pretty sure there are laws in place that prevent direct donations from groups to politicians but I'm also pretty sure there's lots of loop holes as well.
Unfortunately I don't think it's a very black or white question. There could always be a group that helps a small segment of society at the expense of the collective good or a group that helps a powerful segment of society and the collective good. Inherently I don't think special interest groups are bad things, but how they're used and what they do makes them bad or good. It's dangerous to throw a wide net that says "they're all bad" or that "they're all