Mississippi Blues Song, c.1920...pertinent excerpt:

Well they cut so many trucks and trifles,
White folks you oughta work more mules than men.


(?) (?)

Other southern usages I've heard ...

a) Don't trifle with my affections.
b) Milo's a low down, no count, trifling man.