You should be editing and producing your video at 1080 x 1920. Start by setting your project dimensions to 1080 x 1920. You access those settings directly above the center of the canvas.
Scroll open the settings and choose custom.
Type in 1080 width and 1920 height. Then select apply.
When you're done editing select Share, custom production, new custom production.
Go through the production wizard and make your choices. If you look under the tab, size. Then video size. You will see that the setting 1080 x 1920 is applied by default.
The only decisions you really need to make is do you want to produce an MP4 video? Produce it with a controller? Audio quality? Video quality. Things of this nature. The controller won't work with Instagram so there's no reason produce a video with it.
If you just choose MP4. Video, and keep hitting the next button. You will end up with a video with all the standard default settings the Camtasia produces a MP4 video with when producing an MP4 video with a controller. That's a 60 percent quality video setting and 128 kbps audio. Personally, I like the bump the audio up to 256. I find 60 percent quality video is good enough in most cases.