Luke Bryan was new to Nashville, and very much alone, when the devastating terrorist attacks occurred on September 11. The singer, who was just 25 at the time, recalls how lonely and afraid he felt, watching the news on TV.

“As we come up on the 20th Anniversary of 9/11, certainly a life-changing event for me,” Bryan shares. “It was really, really challenging for me, because I had just moved to Nashville. I moved to Nashville September 1st, 2001 and I’m in an apartment by myself down in Franklin. I never will forget. I was in bed and my sister called and said, ‘Turn on the TV. A plane had flown into the World Trade Center.’ I remember kind of getting my wits about me, and I turned it on and like so many other people, I saw the second plane hit.”

Bryan stayed in Tennessee, but his heart was with his family in Georgia.

“At that moment a lot of innocence is forever lost and that’s certainly when the world changed,” Bryan says. “And I remember almost getting in the car and going home and spending some time with my family, but I wound up toughing it out in Nashville. But it was a challenging moment being away from your family when that happened.”

Only six years later, Bryan, who had already lost his brother, Chris, unexpectedly lost his sister, Kelly, as well. It’s a series of losses he still has a hard time grasping.

“My sister was an entirely different dynamic, because she was like a mother to me, somebody who was always checking on me, wanting to know this and that and how I was,” Bryan tells People. “Kelly had some children that she left behind, so we get to live through her, with her children. We see so many things in her children that remind us [of them] … The main thing is you truly never get over it. You never settle in your mind that it’s happened. It’s always there.”