I've never been to Pakistan but recently did a trip through India including border areas with Pakistan where the Government of Canada advises against non-essential travel. I had no problems whatsoever in these areas and came across several backpackers who had been through Pakistan. I'd hoped to cross over to Pakistan myself but a friend I was travelling with is an Indian citizen and wasn't comfortable going there (in most areas, it's safer for a westerner than an Indian given the tensions between the two countries). Based on conversations with these people, the majority of the country is safe to travel with the exception of the Federally Administered Tribal Areas, the Swat Valley, and areas surrounding Peshawar in Northwest Frontier Province. These are along Pakistan's northwestern border and not near your destination. In fact, the province where the Karakoram range is located is considered one of the safest for tourists.
There is certainly a risk to going there, but as long as you prepare yourself and do your homework much of it can be mitigated. I'd suggest doing research on
Wikitravel Pakistan and reading the pages of individual destinations to get a sense of what these places are like from travellers who've actually been there.
I'm sure many of the people on here telling you not to go would have no problem going to Mexico or the Dominican on vacation. Both of these places have higher murder rates than Pakistan even when the ongoing insurgency is considered. I see your trip as similar to a trip to Mexico - it's possible that bad things can happen anywhere, but the vast majority of danger can be avoided by staying away from problem areas.