Would it be possible to change past perfect into simple past

4. She invited me to dinner. But it _____ two years since I _____ to her house. So I lost my way.

The answer is "was", "had gone" but would it be possible to change "had gone" into "went" because if he lost his way, may be he only went to her house only one time and "went" refers to a single event.

https://www.englishtestsonline.com/mixed-tenses-advanced-level-test-quiz-online-exercise-with-answers-2/