XMBLY's a great case study for this. Somerville would have allowed essentially infinite height within reason - 50 Prospect Street is in a neighborhood and will be taller than anything at Assembly - but we end up with this (again, there are two adjacent projects of the same use on the same street that will be significantly taller). There's also not a great definition of maximum height/density for any project above 5 or 6 stories, because all of those projects will be special permits and negotiated on an ad hoc basis.
I'm a little familiar with Cambridge's deliberations about the Volpe site, and I think that if DivcoWest had come to them looking for 500' there would have been even more comfort with it there, since the site is less visible from a neighborhood. DivcoWest simply didn't ask, because they didn't want to build it.
I'm not aware of a municipal standard that says "250 feet is the highest you can go if you're in an urbanish neighborhood". Cambridge, again, has come closest in the negotiations around Volpe, MXP, and the Broad Canal sites, but in several cases they provisioned not to have a plateau by having signature buildings go taller. The BPDA didn't tell the Kenmore Hotel, Fenway Center, and the new WS/Red Sox project to fall within 15 feet of each other (and of the Pierce), they did that because it made economic sense.
As you state, projects are negotiated on an ad hoc basis. Each of these "special permit" projects has plenty of discussion and negotiation between the developer and the relevant municipality as to what is realistic to get approved before plans ever go public. There is no point in a developer ever pitching something in public that isn't going to get approved. And then there's often further negotiation that happens out in the public once things are initially announced.
Some "market driven" forms are also the products of regulation as well. 5-over-1 is an example of this: developers like it and use it all the time because it's good bang for the buck, but that isn't to say they wouldn't prefer, say, 7-over-1 or 14 stories of steel and concrete if either of those building types would be allowed on their parcels (by building code or zoning code, respectively).
(It's also important to think of this in terms of height
and density. Plenty of times density and use restrictions are the binding rules that dictate height, so even if maximum height isn't explicitly stated it still is enforced in practice if regulations don't allow for the density or use that would make up that height. If a developer is only allowed X units or a Y FAR, and that logically fits in Z floors, then they won't build higher than Z even if Z+2 floors might technically be allowed.)
Again, regulation isn't the governing factor for every single project. But it's always a dance as to what will be approved. In the case of Assembly, FRIT and the City of Somerville have absolutely had push and pull on what amount of height and density would be allowed (there was a big debate over unit counts in one of the apartment buildings, for example). In the example of Cambridge Crossing, the governing height and density plans there were approved years before DivcoWest entered the picture. Then when DivcoWest got involved in the development they ran with the bird-in-hand approvals they had and built right up to the tippy top of the allowed building envelopes. It is absolutely true that those building heights and densities are the product of regulation, even if DivcoWest didn't directly attempt to get those regulations significantly changed once they entered the picture. If DivcoWest could have built higher and denser *without starting the approval process all over* they would have, but it didn't make sense to delay their projects, burn bridges, and start over in the hope of (maybe) getting more density allowed.
As you allude to, market conditions at Volpe could support Downtown-scaled buildings there, but Cambridge will only approve a scale smaller than that.
As for the Fenway projects, there is absolutely regulatory negotiation going on with these buildings. Note that nothing else along Boylston is as tall as the Pierce because the governing zoning envelope only allows the Pierce to be that tall, with everything else required to be shorter. Also note that two latest non-Pierce buildings on Boylston (the Harlo and the new Scape Boylston, both residential with one going up before Pierce and one after) both got chopped down by the powers-that-be after initially proposed. That's regulation at work, not the market. The Scape project also changed its proposed use in order to get approved. Huge amounts of regulatory ink was spilled over the (god awful) new Related Beal development in Kenmore, with Citgo sign sightlines playing a big part in that. Every PNF (including the Fenway Center one) notes step downs and set backs in order to gain approval, and these are batted back and forth through the approval process. The Red Sox projects themselves are a little different, because the Red Sox care about the aesthetics of height around Fenway Park more than anybody else does.
So yeah, it's not true that the scope of every project is dictated by regulation (and now with the pandemic uncertainty has certainly thrown a wrench in plans). But most projects are still highly shaped by regulation, be it stated and codified regulation (like we see with Logan or with all the "shadow laws") or more behind-the-scenes horse trading and push-and-pull with regulatory authorities.