Living in the bay area can be particularly maddening. Even if you're working hard and making a good living, you are likely to still not be able to afford a house in the neighborhood of your choice. The reason being its so hard to build anywhere without coming up against NIMBYs and people that already have theirs. Take the BRT disaster where Berkeley rejected even doing the study for dedicated lanes in the city limits. It seems like progress is just a step away but defeat is often snatched from the jaws of victory.
I sometimes wonder why we can't just build more dense housing in employment districts or places where NIMBYs don't exist. There's a huge supply of land in these areas of San Jose with parking lots that could use serious transit infrastructure expansion. But the fact of the matter is that areas that are really desirable and dense are for the most part built out, and since they are built out their cost continues to increase dramatically because people really want to live there and there is a limited supply.
Take for example the Mission in San Francisco. For many years it was a lower income neighborhood known for its culture but over time transitioned. There are still vestiges of this in the compact and livable urban environment, but now the hipsters have come. I'm not sure that's a bad thing per say but we've seen this story before. Certain parties populate an urban neighborhood and then others follow until it becomes upper class, it gentrifies/yuppifies (a good read here on this subject). This end state of neighborhoods is seen as awful for the folks that were pushed out, but it is also seen as progress for the city as buildings get painted and the garden flowers are potted. This very end state of the process or "Starbucks Urbanism" is what becomes the mark of progress for those seeking it.
The problem however I see with this is not the end state per say, but the fact that the process has to happen at all. The biggest issue I have with the gentrification claim is that it can be rendered useless if we actually supplied housing for the actual market for housing. I know this is a claim long pushed by the planners and CNU set, but there's actually something behind the idea that we've overproduced single family housing and under produced urban types. What we've seen in urban neighborhoods with good bones over the last decade or so is a transformation based on lack of opportunity to improve without pushing out the middle.
But I do see a possible opportunity in the massive expansion plans that exist due to the transit space race to improve without pushing away. With multi-line expansion plans in places like Los Angeles, Denver, and Seattle, so many stations will be brought on line, the market won't be able to get to them all at once. One of the major benefits and worries of these new transit lines is that they will bring increased property values and push out existing communities. While this will provide better mobility to many of these areas, it's not likely to bring wholesale change to each of them. But it does start to provide opportunities for building housing that starts to change the urban vs. suburban market, without focusing it all on one close in neighborhood such as what has been happening in smaller regions that build transit over the last boom. We'll see what happens, but this is the theory I have.