During many parts of the movie, it is not really like a movie. It is just Woody Allen talking about his life in the movie to us, the audience. In other parts, it is similar to Wild Strawberries, where the old man would have flashbacks where he would walk through scenes from his childhood in that the characters appear in a scene, such as a party in Alvie's childhood. In these scenes, Woody and whoever else is with him are not participants; they are merely onlookers and commentators. These elements of impossible situations are also used as comic relief-- for example, the children in the classroom unexpectedly saying, as if they were grown adults, that they were plumbers and they beat their wives or whatever.
I think that the fantastical elements certainly added to the movie and helped to clarify and develop the characters of the movie, because instead of just seeing the characters actions and hearing what they were saying, we were able to see into their minds, what they were thinking and feeling.