Deuteronomy 2:8 & 9

"So we went on past our relatives the descendants of Esau, who live in Seir. We turned from the Arabah road, which comes up from Elath and Ezion Geber, and traveled along the desert road of Moab. Then the Lord said to me, 'Do not harass the Moabites or provoke them to war, for I will not give you any part of their land. I have given Ar to the descendants of Lot as a possession.'" NIV translation

As Israel headed north, they came to the land of the Moabites. These, too, were relatives as they were descendants of Lot through his daughter. Once again, Israel was not to mess with them but to pass through peacefully. 

Deuteronomy 2:10 - 12

"(The Emites used to live there - a people strong and numerous, and as tall as the Anakites. Like the Anakites, they too were considered Rephaites, but the Moabites called them Emites. Horites used to live in Seir, but the descendants of Esau drove them out. They destroyed the Horites from before them and settled in their place, just as Israel did in the land the Lord gave them as their possession.)" NIV translation

Moses reminds the people of the giants that were driven out of the lands of Seir and Moab. The Rephaim were a group of men that were controlled by demons and evidently were made to be huge by them. Moses reminds Israel that those giants were driven out by men.

This is a reminder to us, as well, that no problem is too big to be overcome. God had decided to give these lands to these people and no giants could stand in the way.