[Discuss] 'C' string tokenizer for those who hate strtok

[pw]

Brian Quinlan wrote:
> I'm not actually sure that I agree with Peter's point that his code is 
> somehow "safer" but I think that this discussion is raising some 
> interesting issues.
> 
> One is the trade-off between correctness/reliability and performance. By 
> choosing C as your implementation language, I think that you've already 
> taking a large step in deciding that you favor performance. Usually, you 
> would be better to write the bulk of your language in a higher-level 
> language and then implement the performance critical sections in C. 
> Hopefully those sections would be small enough that you could carefully 
> test and scrutinize them.
> 
> You are also losing a lot in terms of programmer productivity, long term 
> maintenance costs, etc. by using C.
> 
> <ding> <ding> Round 3...
> 
> Cheers,
> Brian

Actually the initial point was that this linked list approach was an 
improvement over strtok, which it is. The code provides better 
functionality, handles token storage dynamically
(but that could also be improved),
and allows repeated iteration which strtok does not.

I'm not sure about productivity. There are some pretty nonproductive
high level programs that require a great deal of specialized support.
If the program is done properly in the first place there shouldn't be
any long term maintenance costs.(unless it's programmed to require a 
vampyric service contract)

The point of writing better code in any language is to build libraries 
of portable, useful and easily understandable (documented) routines.
The reason python or any other language exists is because someone(s) sat
down and coded that language with safety and robustness in mind.

Peter