Each year his parents went to Jerusalem for the feast of the Passover, and when he was twelve years old, they went up according to festival custom. After they had completed its days, as they were returning, the boy Jesus remained behind in Jerusalem, but his parents did not know it. Thinking that he was in the caravan, they journeyed for a day and looked for him among their relatives and acquaintances, but not finding him, they returned to Jerusalem to look for him. After three days they found him in the temple, sitting in the midst of the teachers, listening to them and asking them questions, and all who heard him were astounded at his understanding and his answers. When his parents saw him, they were astonished, and his mother said to him, "Son, why have you done this to us? Your father and I have been looking for you with great anxiety." And he said to them, "Why were you looking for me? Did you not know that I must be in my Father's house?" But they did not understand what he said to them. He went down with them and came to Nazareth, and was obedient to them; and his mother kept all these things in her heart. And Jesus advanced in wisdom and age and favor before God and men.
It's not entirely clear from the passage, but the first line implies that Mary and Joseph went up to Jerusalem alone in past years. I find that reassuring. Even the parents of Jesus needed to get away sometimes, to go and pray alone together, to take a trip and leave the little second person of the Trinity with one of the numerous relatives or acquaintances.
I like young Jesus. His answer to Mary isn't sass, nor is it only understandable on a deeper spiritual level. He's being entirely divine, and entirely a smart kid. When he finds himself alone in Jerusalem, he goes to the most obvious place (he believes) his parents would look for him: the temple. And he stays there until they come. So when Mary tells him that she and Joseph had been all over the place looking for him, he's understandably flummoxed. Why'd you do that, Mom? Couldn't you guess I'd have been here all the time, waiting for you? Why did you look anywhere else? It's just the answer a bright twelve-year-old would give, completely plausible on the literal surface level, and precisely the sort of story a mother would keep in her heart and recount years later when asked for memories of her son.