Answer by Martin Ender for Tokenize a Stack-Based language

Retina, 6864 63 bytes

M!s`"(\\.|[^"])*"?|'.|\d+|\Sms`^'(.)|^"(([^\\"]|\\.)*$)"$1$2"

or

s`\s*((")(\\.|[^"])*(?<-2>")?|'.|\d+|.)\s*$1$2¶\ms`^'(.)"$1"

I think this covers all the funky edge cases, even those not covered by the test cases in the challenge.

Try it online!