Again, I'll say, what would this city be without the colleges and universities in Boston, Cambridge, and the inner metro area (Ill throw Tufts and Brandeis into the mix)? And Meddlepal, tell that to the people who attend those schools, who work at those schools and to the businesses that surround those "completely forgettable" others if they were to just up and leave. The positives of all these schools far, far outweigh any negatives.
I think you're missing the point. The point isn't that they don't have any impact because they do, but rather that Boston doesn't need them. They are not anchors like the other mentioned schools are. There impact is negligible, heck, it might even be better off in some cases since there would be more housing stock available if we dropped the Emerson, Suffolks, Fenway Colleges etc. of the world.