I’m sorry to hear that you’re having this problem. Actually, it’s not that you’re doing anything wrong; this is just the nature of audio and video. Our cameras record video in frames, and they record and sync up the audio as they record the video. However, audio recorders record audio in Hz, and trying to match that up with frames is often difficult, especially with long clips. The longer the clip, the more time the audio has to get out of sync.
There is a solution, however. Red Giant makes an application/plugin called PluralEyes. PluralEyes can sync up your video and audio automatically, and the current version can take measures to correct the issues that cause video and audio to lose sync over time. It is compatible with Final Cut Pro X 10.2.3 and higher. Go to http://www.redgiant.com/products/pluraleyes/ and check it out. They’ll give you a free trial.
Hope this helps.