Excavation contains two tasks, first you need to find the dig site with the help of the Antiquarian's Eye, and secondly you have to excavate it.
Once you have placed the Augur in all corners and the middle you will notice that one of the areas is not red. That way you can ensure the treasure must be in that area. Now you can use the Augur again to find the sweet spot in that area.
Digging
Once you uncovered the area, you can start digging. Use the Hand Brush first to remove layers that are not equal, once they are all the same height you can use the Heavy Shovel to remove a lot of dirt at once.