You were trading using this name BEFORE them, and therefore built up reputation before them. Therefore, you cannot be meaningfully challenged over your use of the term and have a clear defence on your original domains.
The problem for people who have registered domains but not a trademark is that someone else can reg a trademark after them, and then challenge the original person for any additional domains in the term subsequently registered by that original person. This is wrong in my opinion, and is similar to reverse hijacking (though not exactly the same). However, people can leave themselves open to this type of behaviour if they don't have a trademark and aren't actively watching the trademarks database ready to oppose applications for your terms. The problem is, it's very difficult to actively watch the trademark database for your terms. I imagine there's a service that might do this for you, but it's bound to cost a lot, and the alternative of reading the trademark journal publications yourself is rather time consuming.
I'm happy to discuss this further with you if you'd like to PM.
Rgds