Given that the expression is ungrammatical in the first place, if pushed for casual purposes, I'd say, "Me neither," for "Neither do I" for the same reasons expressed in posts above.