This didn't use to happen with prior versions of Snagit. So I assume it's a setting?
What resolution is screenshot2? What resolution is screenshot3?
You can find the resolutions beneath the image, above the open capture tray, in the Editor. See:
If screenshot2 is much different than screenshot1 in size, some pixelation might happen if there is excessive resizing happening.
Robert's answer is dead on, but I thought I'd provide some additional assistance since I run into this all the time in SnagIt 13.
In this image, (where I had to do Print Screen because you can't capture a screen in SnagIt Editor using SnagIt Capture) you see that Image 1 is 80%. The callout shows you where the resolution size is.
Then in this image, you see what happens if you paste another image into Image 1 - you get pixilation.
When Image 1 is set to Actual Size (100%), pasting in Image 2 should prevent pixelating. By the way, there's a misprint in the second callout box below. It should read, Image 2 is pasted into Image 1, and is not pixelated.
Good luck!. Like I said, Robert's comments above totally jive with what I've shown here. Just thought walking through the steps might be more useful.