Is this 896 MB of RAM made up of mismatched memory sticks? Or is this the remainder of a 1 gig stick, or 2 matched 512s? What's in there?
Have you tried upping the page file size (Virtual Memory via the Advanced tab from My Computer - right-click,Properties) ? If it's at 1500, try 2000 and see if it makes a difference.
Something else to wonder about... is the power supply in your PC up to the job? The CPU clearly is and 1 gig of RAM is plenty on an Athlon 64 (I have such a PC here and it copes OK with video), but if the power supply is borderline then this is about the time you'll get signs that it's struggling. If you have a TV card in there, and maybe a couple of hard drives, and whatever else, when the processor is under pressure (as with processing video), an under-rated power supply will cause all sorts of problems, this type of thing being one of them. Shutdowns and restarts are also common. Ever happen?
Taking out the TV card and temporarily disconnecting one of the hard drives would be a way of testing that. If it performs OK after doing that, my best guess would be that your PSU is under-rated. Probably a 300.
Any help? If not, post again.