I really liked The Prisoner of Azkaban, but The Goblet of Fire is my favorite of the series so far. I was told many times that each book was going to get darker as the series went along. This becomes evident by the fourth book.
Besides being notably darker, The Goblet of Fire is unique because it breaks from the established formula of the first three books. You know: Harry Potter's muggle relatives are horrible people. Harry loves going back to Hogwarts. Something is up at Hogwarts. Harry, Ron, and Hermione suspect they know who is responsible. They turn out to be wrong and the culprit is someone you don't expect. Instead, Rowling introduces the Quidditch World Cup, the Triwizard Tournament, and a couple of new and interesting characters. The Goblet of Fire is also the first book of the series that doesn't tie up all of its loose ends. I am guessing that books five and six will follow suit.