I have the same problem when memory it's full. Priority isn't given to the dialer, or it takes a while to offload what's running. Very frustrating. I'm on stock+.
Solution is to keep less running. The latest Facebook app just won't not run, even with all notifications off. Uninstalling that almost, on its own, fixed my problem. I try to keep stuff running to a minimum, as we have to with 512 ram on 4+. Mostly the phone responds fantastically well, thanks to the tweaks found here.
And disable features that you don't need/use. Facebook notifications are a hog. Go launcher is a well known hog. Gmail sync, maps sync. Take a look into what's running and trim it down to what you need. All 512mb phones have this issue. You just can't run it like you would a high spec phone.