In the interest of defining "urbanism" before launching into comparisons based on it - I personally think of urban as meaning "accessible without a car." Not everyone will agree with me on that, but I want to put that forward to put my remarks in context. If I can get around to lots of interesting things by comfortably walking and taking transit, I'm having a great urban experience. If I can live without a car, I'm having a phenomenal urban experience.
I obviously have to agree with how you placed New York and Chicago.
I have a hard time putting Boston behind SF and Philly. You say you can't put it first among peers, but I'll make a case for that. First of all, our transit ridership outstripes either SF or Philly. Adding Heavy Rail + Light Rail systems and excluding suburban commuter rail:
Boston: 167k + 67k = 234k (this is about equal to the Chicago L ridership)
San Fran: 129k + 51k = 180k (arguably BART is an HRT/CR hybrid, so that 129k should be discounted as a measure of urbanism)
Philly: 92k + 24k = 116k
We can include bus ridership, but it doesn't change the order.
In a list of cities with the highest walking mode share, Boston and Cambridge hit really high:
1. Cambridge, Massachusetts 25.76%
2. Ann Arbor, Michigan 16.52%
3. Berkeley, California 15.99%
4. New Haven, Connecticut 14.0%
5. Columbia, South Carolina 13.78%
6. Provo, Utah 13.39%
7. Boston, Massachusetts 13.36%
8. Providence, Rhode Island 12.56%
9. Washington, D.C. 12.27%
10. Madison, Wisconsin 10.99%
11. New York City, New York 10.72%
12. Syracuse, New York 10.31%
13. Pittsburgh, Pennsylvania 10.02%
14. San Francisco, California 9.82%
15. Wichita Falls, Texas 9.29%
16. Philadelphia, Pennsylvania 9.22%
These are just a few objective metrics to support my subjective view that Boston doesn't feel like the back of this pack. I will grant that Philly should get a lot of credit for the sheer density of Center City and SF is geometrically compact and sort of feels more accessible than it objectively is.
I think this – a lifestyle where you can do everyday activities without getting in a car – captures something important about urbanism.
I'm going to push back on your specific metric, though, because it does not include bus transit. In a city like San Francisco, busses are actually useful transit (good headways, dense coverage), so your rail-only metric misses a lot of transit riders. Here are the ACS survey results, which are percentages of city population vs. raw numbers:
Here are the cities with > 20% transit share:
The following is a list of United States cities of 100,000+ inhabitants with the 50 highest rates of public transit commuting to work, according to data from the 2015 American Community Survey. The survey measured the percentage of commuters who take public transit, as opposed to walking, driving or riding in an automobile, bicycle, boat, or some other means.
1. New York City, New York – 56.5%
2. Jersey City, New Jersey – 47.6%
3. Washington, D.C. – 37.4%
4. Boston, Massachusetts – 33.7%
5. San Francisco, California – 33.1%
6. Cambridge, Massachusetts – 28.6%
7. Chicago, Illinois – 27.6%
8. Newark, New Jersey – 26.7%
9. Arlington, Virginia – 26.4%
10. Yonkers, New York – 26.4%
11. Philadelphia, Pennsylvania – 26.2%
12. Alexandria, Virginia – 21.7%
13. Berkeley, California – 21.6%
14. Oakland, California – 20.3%
15. Seattle, Washington – 20.1%
So Boston and SF are pretty much equal in terms of transit ridership (and Cambridge is right behind – it probably "loses" more transit riders who choose to walk or bike instead).
Qualitatively, I think SF has another feature of urbanism that puts it slightly ahead of Boston. There's a better term for it, but I'll call it neighborhood independence: the ability to perform essentially all functions within the neighborhood. Ignoring the central business districts (which are graded on different metrics), the residential neighborhoods of SF (Inner/Outer Sunset, Inner/Outer Richmond, the Mission, Cow Hollow, etc.) have many of their own grocery markets (small storefronts that have real produce and groceries, not Tedeschi/City Convenience marts), as well as hardware stores, ice cream shops, libraries, etc. I think Boston/Cambridge are close, and they probably used to have this, but at some point the grocers in every neighborhood were replaced by big grocery stores that serve
a few neighborhoods, making it more difficult for people to get by day-to-day without a weekly grocery trip (probably in a car).
And for restaurants, the liberal (as in unrestricted) beer/liquor rules in CA make it much easier for a small start-up restaurant in a less trendy neighborhood to survive (vs. the insane startup costs for restaurants in Boston due to conservative liquor licensing). When your mom and pop sandwich shop, burrito joint, and Thai restaurant make it easy to stay in your neighborhood for a good time, it builds a little stronger neighborhood identity.
Now clearly, Boston/Cambridge are doing plenty well from an urbanist perspective, no matter how you slice it. But these are my subjective experiences, based on living in similar moderate-to-high density neighborhoods in each city.