I'm confusing this word with another. Isn't there an art-related word that is something like fractal? Must go check...

Yes. I thought it rang a bell. Here's the definition from geometry, and, by extension, you can imagine the kind of art created:

"A fractal is a set which is self-similar; fractals are repetitive in shape, but not in size. In other words, no matter how much you magnify a fractal, it will always look the same (or at least similar)." (Quick definition on onelook.com)