Memory usage is ok, even if it hits 100% PROVIDING it releases whatever cache to the current processes or swaps stuff out or both. It only becomes a problem when the system becomes unresponsive, as it seems is your issue.
All I can suggest is if you see it heading to greater than 70% to start sampling with vmstat -m and ps aux and swapctl -l and wait until you can sample no more.
A shell script perhaps:
`#!/bin/sh
tdate=$(date '+%F_%X')
vmstat -m > vmstat-${tdate}
ps aux > ps-${tdate}
swapctl -l > swapctl-${tdate}
`
(untested, should work... - )
