It's probably a good idea, like Tamar said, to have a count done yearly. Other than that, you really don't have to worry unless you start having symptoms like bruising, bleeding or petechiae. No, it won't hurt you in the long run.
I went into remission once for two or three years and during that time, never had a count. My GP sort of insisted that I have one at my physical and my counts were in the low 30's. I never knew it and it was no big deal. I was upset with him though because I'd really rather not have known about it. At that point, I had to start monitoring again until counts dropped further.
But no, no serious damage will occur if you have no symptoms.