I had a similar idea, except instead of showing something and taking it away, as the character moves, things would turn just slightly to face them. Imagine you're walking down a dark road surrounded by dead trees, and just as the trees pass your peripheral vision, they turn like 10 degrees your direction.
I bet these kind of mechanics would make for a pants-wettingly terrifying horror game if done in vr too!