A big hugggg and some words.
Our mind is our strongest tool....it can bring you to the highest peak, or down in the deepest void, just by letting it speculate.
Take a calm breath, and think.
Do you love and respect your father?
Do you think you are able to support him, even if he do wrong thing from time to time....after all....are you always innocent....there is difference if something is done with purpose to hurt someone one love, or if an action leads to hurting someone you love

?
Do you know who put out the rumor about him?
Are you so confident that you will support him/her?
You as person will never know the real truth behind any of the above statements,(rumors or Fathers deny) so you have to let your heart decide.
If you still feel like you feel now, then you hurt yourself, you hurt your father, you hurt your family, because you have doubt, and that thought will always do it so you trust your father less and less.
The whole question is....are you able to forgive completely...or is the "crime" he has done so big, that you are willing to destroy the time you have left with him.
I have said the same thing to people who are married, where one have got doubt about his/her partner, if You don't manage to forgive completely..your thought will grow and make small accusations,from time to time, and that will crumble the marriage...or love.
So my advice is....love your father...support him until wrongdoing is proved...then forgive him again, if he not with purpose has damaged his own family....or if this is not possible....move out and start your life without him.
A biiggg Grandpa Huugg!!