That's not the way it's done.

You need charge a leather buff with the Tripoli and make a few even strokes, against the direction of rotation of the pivot, use moderate pressure. Just one light rub with the tripoli ON THE BUFF will do maybe 10 pivots. A leather buff stick can be made from any straight smooth piece of close grain wood. About 3/16" x 1/2" x 6" will be good. Cut the leather strip with a metal straight edge and a razor knife. Then glue on the leather strip and clamp it lightly in a vise overnight. The leather needs to be thin and hard, and it should overhang the working edge slightly, maybe 1/2mm.
I have used a 4" muslin buff for many years. A 4" buff on a standard jewelers dust collector will have a surface speed about 100 times that of a push buff stick used with a high speed jewelers lathe. This can be dangerous for both the clock part and the operator, so be extra careful until you get some experience. This only takes a few seconds on each pivot and you charge the buff by occasionally touching the rotating buff with the Tripoli stick. A Dremell rotary tool can be used in a similar manner but a hard felt buff, at a moderate speed, will work better with this much smaller wheel. This is also much less dangerous than the high speed 4" buff and almost as good but only if you can solidly mount the Dremell tool. Willie X