Before the old man died, he destroyed Asgard. It fell to Earth, which likely would have been destroyed as well if not for his death.

[And he knows what the other Loki will say next, as it's the same as Ikol would say. That wasn't you. But-]

The old man and I are the same person. The same memories. The only difference is that mine stop at this age. That means if I ever grow beyond this age, I have the potential to become him.