It's really not a bad idea. Even if it was just two stories of apartments it would be progress. The library is surrounded by buildings that are between 3.5 and 7 stories; it's really the odd one out in terms of height.
The City should not make such a big deal then of how innovative it is utilizing public infrastructure projects then to construct housing --
https://www.boston.gov/departments/library/housing-public-assets#the-experiment
Also, take steps to make this faster and more efficient. It does not have to be income-restricted housing units. Any additional housing units will help the broader housing market.