It turns out Camtasia v3.0.6 makes it very confusing how screen dimensions are calculated depending on whether a retina/non-retina displays are used. This is from your support team:
"Currently when Camtasia is selecting the pixel size on the screen, it will look at what the exact number of pixels are (if you have a retina display device, this means that selecting a size that seems like 1280x720 is going to be double that size (2560x1440). Looking at the options you are seeing in the list there, you are choosing 2880x1800 vs 1280z720 HiDPI (this I would assume will utilize the retina settings, doubling the amount of pixels on your screen). Do you have an option to try 1280x720 non-HiDPI (otherwise, I may recommend using 1920x1080, which is another standard HD resolution)."
See https://support.techsmith.com/hc/requests/337536 for more context.