Golf is also that sport that only golfers like watching (usually). So... If golfers as a whole don't care about Olympic Golf what makes you think the members of The Country Club will care enough to sacrifice their course for the summer once you factor in damage done by visitors and spectators.
But that is true about almost every sport in the Olympic program, and people watch anyway! You guys keep referring to Olympic golf, but it is not about that, it is about the Olympics, period. It is the same with tennis (almost exactly). The Olympics are not the top of the season... but you still get the games being played on Wimbledon. Why? It is all about the Olympic brand (as a whole, not Olympic golf).
In the next 100 years, there will be 100 U.S. Open venues (ok, a few less due to repetition), 25 American Ryder Cup venues (same, a few less), but only 3 or 4 American Olympic golf venues. That is the reason (I hope) the Country Club members will care. It is, literally, history. Not just golf, but sport history. You see it as a negative (being not golf focused), I see it as a positive.
On a personal note, the Olympics are the only chance to care about sports I don't care in general. As a kid in Barcelona I watched live sports that I love (Handball, Basketball, Soccer, Track-and-Field, Water-polo) and some that didn't even know existed (Badminton, table tennis,...). After that, I always watch sports on TV that are not very popular on the big picture (indoor cycling, sailing, rowing,...). I think that is a very powerful think the Olympics bring. I don't particularly care for golf in the Olympics, but, once it is there, it should have the best showing possible as the ambassador of the game, and that in Boston, is the Country Club.