[Discuss] 'C' string tokenizer for those who hate strtok

[Brian Quinlan]

David Bronaugh wrote:>
> However, I'm curious about your testing:
> - First, pw's Makefile specifies conflicting optimizations -- the last 
> one is -O0 so I believe that's the one that stands. I changed that first.

I actually used the same compiler flags as the Python build and didn't 
bother with the Makefile (it's a one file program).

> - Second, it seems that more than half the time spent in the C code is 
> spent calling printf. I removed the printf call and the program runs 
> more than twice as fast.

I wrote my own main function which just grabbed the time, called the 
tokenizing function 10,000 times in a loop (freeing the memory after 
each run) and then printed out the difference between the time at the 
start and the time at the end.

> My test input is up at http://bronaugh.linuxboxen.org/test.txt -- can 
> you please make these changes and compare? For that matter, rip the 
> print out of the Python, for fairness.

There was no printing in the Python version either. I'm leaving on 
holiday for a week beginning tomorrow morning so I might not get to this 
until I get back.

> Keep in mind that I just wanted to clean up the current code, not 
> rearchitect it. My own opinions is -- just use strtok_r.

And my opinion is -- just use Python (or Ruby [or Perl]).

Cheers,
Brian