Dana is describing a particularly well kept ship:
"There was no foolish gilding and gingerbread work, to take the eye of landsmen and passengers, but everything was “ship-shape and Bristol fashion.” There was no rust, no dirt, no rigging hanging slack, no fag ends of ropes and “Irish pendants” aloft, and the yards were squared “to a t” by lifts and braces.

I could find no definition for "fag" in this context. Perhaps "frayed" ? My dictionary gives as sixth meaning for whip "to wind cord or thread around the end of a rope to keep it from fraying.(It has to be sewed through the rope at start and finish to keep it in place.)

try fagot..

We used what we called fag ends of clothes line to light firecrackers when i was a kid.

the frayed end of the line would smolder for hours, and could be whipped around your head to oxigenate the smoldering rope and get it hot. it was hard to get the smoldering started, but it was in long run, easier and safer than matches.

Faggotting, is a lace-like stitching that is used to join fabric. it is at once, strong, light(lacy) and flexible.
Baseball covers are sewn on with the same faggoting stitch, (but pulled tight, so there is no lacy effect) and the same stitch is used in open heart surgery, especially in valve replacement.

its hard to tell if what is meant is oversewn ends (sewn with faggoting) or frayed ends.