Here's the way I heard it ... from an elderly friend of my Dad's. His Dad was a "drummer" (travelling salesman) in the Heartland in the late 1800s.
Yard sticks were in short supply.
Not considered a neccesity when travelling west in a covered wagon.
Drummers who sold fabric would carry a wooden yard stick. The drummer would put the yard stick on a wooden counter and the ends were marked, then brass tacks were nailed on the two end marks. Brass upholstery tacks were used as they have shiny heads, easy to see and didn't generally rust.
When a woman came in to buy fabric, if the store keeper tried to measure the cloth by eye she would tell him to "Get down to brass tacks." In other words, to measure it correctly, not guess, or cheat by "shorting" her.
The phrase appears to have come into the language with that meaning.

Not being chauvanist, women were generally the fabric purchasers in those days.