Go ahead and use the term "geocaching" on your navicache site. First, he has to claim damages. Second, he has to prove that he has a claim to the term. I don't think there's any question regarding the latter, and since his site is non-profit I can't fathom the former.
If he goes after you he has to go after geocache.org, as well as Mike Teague's page, my distance-calculator page, and a handful of others. He can't sue you without suing all of us or else it'll get thrown out for "selective" harassment.
I took a look at your page and I think it's great. It doesn't claim to be "the" official cache page (another peeve of mine) and has some great information. Back when this all started there were three or more "main" pages and we all had different features that complimented each other and we shared the data freely (mostly Mike compiled the data and we mirrored it, but there were submissions to each of our sites as well). I think that's the best model so that the activity doesn't rely on a single individual, nor do any "ownership" issues arise.