Yes, you run a slip stone or very fine diamond file on the inside lip of the cutting face, basically you need a zero rake, a twist drill as standard will dig in and the brass plate as the drill breaks through to the other side, this can cause the part to catch on the drill and spin around.
I use metric drills in 1/10th mm steps for bushing since I don't use a bushing machine or the cutters, you can also buy drills in 0.05mm steps, so to drill a hole for a 3mm bush I'll use a 2.95mm drill, and all of my drills for brass have zero rake,
Run the slip stone parallel to the axis of the drill and take a few strokes off the cutting face, the drills will cut brass and cast iron without digging in.