Nicely summarized thoughts here.I feel pretty strongly that the 39 is a success story and should continue to exist in some form no matter what.
Assuming no light rail extension beyond Hyde Square: maintain as is (with as many BRT enhancements as you can fit). Assuming surface light rail extension beyond Hyde Square: maintain the northern half of the 39 along Huntington (probably rerouted to terminate at Brookline Village). Assuming (somehow) a light rail extension in a subway all the way to Forest Hills: maintain as is (with as many BRT enhancements as you can fit).
If we're going to go to all the trouble of getting the B and C out of the Central Subway, and go to the trouble of modernizing the D & E to be as "rapid transit-y" as possible, why would we then turn around and add a 1.3-2 miles long surface line to it?
If nothing else, the 39 should remain as a local bus running from residential JP into its proximate "downtown" of Longwood. Longwood <> Forest Hills sits comfortably in the 2-3 mile range where inner surface routes can outcompete rapid transit due to the overall low travel time, added convenience of having a closer stop, and the lack of transfer penalty. The Downtown Crossing Bus Connections map illustrates this vividly:
It seems worth noting that the relocated Orange Line opened just a couple of years after Arborway closed. (How have I not noticed that coincidence before?) The 39's reduced ridership (which I'd love to see a citation for -- I don't doubt you, I'd just be interested in the data) is probably in part due to that, especially since Orange provides an OSR to downtown that the 39 doesn't.
That being said, if we really do think there is pent-up demand for a streetcar, then the 39 could just be converted to a streetcar, running surface level all the way to Back Bay. Marrying that into an extended Huntington Subway is unnecessary.
I think there is still room for improving the 39’s route in JP. Over time, perhaps in phases, eliminating more parking along Centre, eg, and having a couple stretches where it could bypass the big jams in downtown JP. And yes to it being a legit success story in its own right.
Your final comment is interesting since it’s something I’ve thought about a lot—why are we stuck with only GL-size trains as the only light rail we could ever consider? Many cities have smaller trams that are less imposing, less “heavy”, and might ultimately work better on city streets than big GL train cars.