Solved Trouble with removable parts

Hey Guys, I am super new to this. I have only printed ~15 projects and <100hrs total. With that said, I did do some extensive searching before posting. I imagine I'm just getting the terminology wrong and this seemed like a good place to start.

I am using Matter Control still and haven't had any crashes/issues to speak of so I haven't ventured into the alternatives yet. My issue is with components that are intended to be removed after the print finishes. I think these are sometimes called stands? Anyway, I tried a Lego block and I am working to print a settlers of katan board. I'm new so I cannot post links. It is on thingiverse. Item 1238980.

Anyway, the board pieces are designed to take 18, 3mm ball magnets. To put those in, I have to remove small inserts from the holes. This has become a major issue for me. Sometimes they are so bad I feel like I am digging out layers that fully adhered to the inner wall. Other times they basically fall out and the hole is slightly too big. I have never successfully been able to remove the inner part of the lego so that it looks like a hollow brick. It is always close to solid.

My basic question is, how to I get more consistent on this and improve the removable parts? Here is what I have tried:
Ensure good adhesion - I am using hairspray with very good results (I think)
Level the Z axis - Double and triple checked this
Monitor the 1st layer - This always seems to be very good. The inserts have a nice gap from the rest of the print.

To trouble shoot this, I stopped a lego before the top was built and the removable portion was possible to remove, though not easily. It seems to be going south as the print goes up.

Any help would be greatly appreciated. Thanks!