I used a drill press and various tools at the Milwaukee Makerspace to rough out a bit of the inside diameter of the part. I tried different tools (see picture) starting with a little Dremel sander (not pictured), then a 1/4" drill bit, a bigger gear-looking bit, and finally, a wire brush. The sander and wire brush both worked but are incredibly slow. The drill bit did not work at all. The gear thing was too coarse. It kept gouging out parts of the plastic.
Eventually, I did shave off enough plastic to squeeze the holder onto the PVC. Unfortunately, I shaved off enough plastic to get to the inlay and crack the piece. You can see the crack in the picture of the piece.
I will see what Tim thinks but we will probably push on and try to finish assembly with what we have as a beta. Then, I can re-measure everything and try to re-size and re-print the auger and pitcher holder to fit the PVC dimensions.
I think that will be after I get the HESA project done.