Whenever I try recording a gameplay video (such as Gradius 3), the recording time is fixed at "0:00:00" and the stop/pause/delete buttons don't work. And when I press F10 to stop recording, it waits for hell to freeze over in order to process my recorded video.
I work with another product (Adobe Captivate) and I know that from working with it extensively, sometimes we see certain conditions arise where Captivate simply has an issue with operating in the same computer memory as another application. And when that happens, you are unable to record or capture anything.
Camtasia likely shares this possibility. After all, it's just software.
In these cases, in order to coax things to play nicely, what we suggest is to run the application you need to record on a different computer. Then establish a remote desktop connection from the computer with Camtasia on it and capture from the remote window.