2

I need to read a text file which may contain long lines of text. I am thinking of the best way to do this. Considering efficiency, even though I am doing this in C++, I would still choose C library functions to do the IO.

Because I don't know how long a line is, potentially really really long, I don't want to allocate a large array and then use fgets to read a line. On the other hand, I do need to know where each line ends. One use case of such is to count the words/chars in each line. I could allocate a small array and use fgets to read, and then determine whether there is \r, \n, or \r\n appearing in the line to tell whether a full line has been read. But this involves a lot of strstr calls (for \r\n, or there are better ways? for example from the return value of fgets?). I could also do fgetc to read each individual char one at a time. But does this function have buffering?

Please suggest compare these or other different ways of doing this task.

Improve this question
5
  • 6
    Use C++ and std::string and std::getline. Why not? Profile before you claim it's too slow. Commented Jan 29, 2012 at 23:10
  • Note that fgets() will not read \r as an end of line under normal circumstances. Look at POSIX 2008 and getline() but beware of the portability implications of using it. (OTOH, it is not dreadfully hard to provide your own implementation if need so be.) All possible line endings is trickier - even POSIX getline() only deals with a single delimiter character (as does getdelim() on the same page). Commented Jan 29, 2012 at 23:13
  • Read in the whole file with fread() or read(), then go searching for the '\n's. Something similar can be done using mmap(). Commented Jan 29, 2012 at 23:14
  • fgetc() does have buffering, as do getc() and getchar(). Most of the input is described in terms of 'as if by calling `getc()'. Commented Jan 29, 2012 at 23:15
  • "Considering efficiency, even though I am doing this in C++, I would still choose C library functions to do the IO": Where did you find the bottleneck to be when you profiled your C++ code? Commented Jan 30, 2012 at 1:12

1 Answer 1

Reset to default
2

The correct way to do I/O depends on what you're going to do with the data. If you're counting words, line-based input doesn't make much sense. A more natural approach is to use fgetc and deal with a character at a time and let stdio worry about the buffering. Only if you need the whole line in memory at the same time to process it should you actually allocate a buffer big enough to contain it all.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.