First of all, what camera is it? Then, how is it mounted?
If you have a Moultrie A5 (and what you describe sounds just like an A5), I can tell you that your problem is with the camera. I have two. They are practically plot-watcher cameras they trigger so easy. They do a good job of catching game, but get about 1000 extra pics as well. I can feed the pics into a video editor and make a stop-action animation movie of clouds going by during the day. At night, it does not trigger as often.
If mounted to a swaying tree, that can trigger the camera as well. If that is the case, pick a larger tree or use a T-post.
Cant the camera at a slight downward angle so as to not get so much in the way of tree limbs and sky.
You can try placing tape over the top half of the sensor (externally) to blind it from above motion such as tree limbs.
I have a Moultrie A5 and it is literally driving me nuts!!! I have 4,000 pictures of nothing on my camera, it takes a picture every 5 minutes until the batteries die or the card gets filled.
What is the best way to reduce the sensitivity or at least fix it so it is somewhat functional?
I do not want to buy a new camera and I really don't have the money for it... But yeah, this thing is driving me up the wall!
By the way, mounted on a t-post facing north.