If you are splitting the input string on either lowercase w's or uppercase W's,
you don't need the i flag. Those are the only two cases w has.
I'll get rid if the i flag. It was left over from a previous attempt when I was using something like /(\s+)|(\d+)/i and /ig test expressions...
But I don't see where all those ws are coming from.
When I used only the \w or only the \W, I got only numbers and variables with letters. It left out the other characters in the string.
And why did you replace sequences of the letter s with a space?
This was to remove any extra spaces between the "tokens" so they would not show up as blank tokens. In an earlier attempt I had used \s as a split value and was getting incorrect results with extra spaces in the entries.
And what is poor tokens_index doing?
You're right, it is not needed. Again, it was left over from a previous attempt before I tried getting help from this forum.
Thanks for your response ... any other ideas? I'm definitely not an expert on regular expression (regex) formations. It just looked like a good idea at the time.