I'm pretty sure that nothing will ever compare to or replace Grave of The Fireflies in the minds of many (self included). The film is heartbreaking enough but knowing that it's based off of the author's firsthand experience makes it even more tragic.
...but if there's anything that can potentially rival it, my vote goes to When the Wind Blows, the follow up to raymond Briggs' The Snowman (awesomely already mentioned in this thread). It offers up similar themes as GotF, but manages to make them more bleak even if it's fictional. Fans of 80's music take note: Roger Waters dominates the soundtrack that also features Genesis and a title song by David Bowie.
I also haven't seen anyone mention The Plague Dogs. The novel is a bit different (and significantly more grim). I won't ruin it for those that haven't seen it, but it can be extremely tough to watch and sad for folks who are fond of four pawed creatures. Definitely one of the most haunting animated features I've seen.