Books 1 and 2 set up the initial attraction from Ginny. Harry is famous, and friendly with her family, she has a massive crush on him, and then he saves her life. Book 3 has multiple examples of her continuing crush, and also has a moment where Harry and Ginny clearly share amusement about Percy (continuing from the end of book two). Midway through book 3 Harry meets Cho Chang who is his main crush for the next couple of books.
By book 4, Ginny has figured out how to not be completely obvious about her attraction. She starts interacting with Harry regularly through her family (during the summer) and her friendship with Hermione (while at school), and she mostly plays it cool. A key passage that shows she is still attracted happens in the lead-up to the Yule Ball. First, when Harry mentions that he had asked Cho to the ball, we get this one line paragraph: Later in that scene, when Ron offhandedly suggests Ginny could go with Harry, she is first embarrassed and then looks miserable as she explains that she's going with Neville.
In book 5, you see Harry and Ginny establishing a decent level of friendship, what with her hanging out with the trio over the summer, then taking the train with Harry (because Ron and Hermione are prefects), and of course being in the DA. It's also where we start seeing her display skills in two areas that Harry is good at: combat-capable magic and quidditch. She also calls him out a few times. I think this passage is particularly key: Through the rest of the book, the friendship between Harry and Ginny continues to strengthen, first when she replaces (and impresses) Harry as seeker when he gets kicked off the team, and later when she helps originate and later execute the plan to contact Sirius. She's one of very few people outside of Ron and Hermione who Harry repeatedly takes advice from. Also, it's subtle but I think clear that Ginny still cares for Harry, most obviously in the scene where she argues passionately for joining the trio on the trip to the Ministry. It's also notable that she has multiple exchanges with Harry where they share a bit of a joke with a look or a brief phrase; I don't remember anything similar with anyone else outside the trio.
And then of course there's book six, which has a ton of setup leading up to the moment when Harry "suddenly" realizes that he's attracted to Ginny. I particularly like this passage when they board the train to Hogwarts: So yeah. Harry may have been clueless about Ginny until almost halfway through book six, when he was forced to acknowledge her sexuality by walking in on her making out with someone. Notably, this is after he's been single for a while, and has had time to process Sirius's death. But, I saw the potential coming from back in book 4 when the little hints from the earlier books kept showing up more and more often.
(Also, no chance it was a love potion--he's practicing with the team and then encouraging Ron leading up to them walking in on Ginny+Dean).
