I was in NYC in June 2007 (on the way to Rhode Island), but only got to within eye-sight of the World Trade Center site, on a tour bus one block away... and if I had stayed another day, I would have gone to have a closer look.
Even though it was almost 6 years after the destruction of the buildings, and the site was cleared by May 2002 (10 months later), bureaucracy and committees dragged out the beginning of construction of the new buildings for 5 years. Aside from Tower 7 which wasn't on the actual WTC site, nothing significant started on the actual site until 2006, and took another 5 years until the first above-ground structures were completed. So even if I had have been at the site on that trip in June 2007, I probably wouldn't have seen much more than what I did from a block away.
(the couple of photos that I managed to take as the tour bus was driving past showed a fair bit of debris on the site, with a number of cranes, so they had definitely started doing something... it would just be another 4 years after that until the memorial site would be ready)