Hello Vasile,
I've not seen this, and trying to find out if this is a known issue or bug. One question I had was if you disabled this theme does it work properly? I think the best thing to do would be to open a case with our support team.
Here's where you can open that case and they will be back within one business day.
http://techsmith.custhelp.com/app/uti...
I have another possible work around for you in the meantime. Can you position the dialog or windows on a solid color background? If so, click and drag to select a region larger than the window. Next, in the editor, use the paint bucket tool and pick the transparency "color". Zoom in and paint the excess around the window with the transparency. This should get you a nice smooth edge.
Oh, and I find it very handy to finish by using the Trim tool located on the Image tab.
I hope between the workaround and/or contacting support this will prove helpful.
Sincerely,
Mike Curtis