When you stay in Hpa-an you can do a tuktuk tour of one day to caver and temples near Hpa-an. I did this tour with a group of 7 people out of the Soe Brothers Guesthouse in Hpa-an. Some Brothers provides this tour with their own driver.
Bat cave. An absolute must-see, the bat cave is a small cave that cannot be entered but you can go there for sunset when, literally, hundreds of thousands of bats fly out. They are said to be flying all the way to Mawlalmyine to feed and come back every morning. The cave is part of a complex of a few pagodas, one on top of the rock, accessible by a steep ladder, with amazing views over the river and the bridge. There is an old couple who live there with some kids. Consider making a small donation for them to take care of the place.