I have been experimenting/researching on how the custom bokeh works and why some of the shapes become cut off. For more information about DIY custom bokeh, see here.
I have found these pages written by, Paul van Walree (toothwalker.org), very useful in understanding bokeh:
The reason why vignette is important to know in bokeh is because, "the shape of an out-of-focus highlight (OOFH) mimics the shape of the clear aperture" (Vignetting).
From my experiment, the consequence of optical vignette also applies to custom bokeh.
The image that I took of HMO Miku and butterfly bokeh were cropped from this image; shot with Tamron 60mm f/2 wide open:
As you can see, the butterflies near the edge of the image loses its shape, because the OOFH mimics the clear aperture it sees.
To show you what I mean, I took pictures of the rear of a large aperture lens (50mm f/1.8 D) from different angles. *The lens is wide open with the custom bokeh attached in front of the lens.
(Left) The lens is tilted away from the camera. The OOFH in the corners/edges would see and mimic this shape.
(Right) The lens is looking straight at the camera. The OOFH in the center would see and mimic this shape.
So in order to have perfect custom shapes of OOFH in your final image would be to crop them or have them appear near the center only
Tamron 60mm f2 wide open. The OOFH are all in center, so most of them maintain their shape.
Another difficulty, playing around with custom bokeh, is actually having a subject in the image.
My Tamron 90mm f/2.8 is actually better at maintaining shapes of the OOFH, but it requires a lot of working space to balance the size of the bokeh and the subject.
In order have this size bokeh and the subject in focus, with my limited space in my room, I had to focus very close on Black ★ Rock Shooter.
(The lights were placed approximately 7ft away from the camera.)
With shorter focal length, I was able to capture larger area of the subject with more breathing room for my self.
(The lights were placed approximately 6ft away from the camera.)
Ultimately, optical vignette plays an important role in understanding custom bokeh.
Here are some more pictures I took with custom bokeh with Tamron 60mm f/2; wide open:
Six point star shape, custom bokeh off centered, from the lens.
Thanks for reading.