How wasteful they actually are is a bit uneven actually and varies with which Olympics you're talking about. I believe Atlanta actually made money on the games, for instance. The concept of the Olympics as a huge debacle is buoyed by the times that was absolutely the case, primarily in Montreal and Athens (Montreal has the distinction of having the Olympic debacle give birth to an even bigger SNAFU at Mirabel, while the 2004 games partially led to the collapse of the entire Greek economy).
Obviously, the wastefulness of the games increases with every new structure you build and every new speculative risk you take. As I said above, I don't think an Olympic Village at North Point is that speculative - the developer is confident enough that those units will sell that it's building them even without the games. Temporary stadia are a big efficiency boost, and unlike Athens or Beijing Boston has a glut of 3,000+ seat venues which in many cases are kept in good states of repair by educational institutions with deep pockets.
Boston's advantage is that the major infrastructure costs for the Olympics would actually be on infrastructure, and locating the games in the city would mean that all of that investment would actually be useful after the games (unlike London's speculative new rail lines/stations, for instance). If you list off the projects that would likely cost the most - Turnpike improvements, refurbished GL stations, DMUs and SS expansion... which of those really sound wasteful to you? It's a whole heck of a lot better sounding than the Forbidden City of arenas in Beijing...
