4

I'm working on creating a binary search algorithm in C that searches for a string in a .txt file. Each line is a string representing a stock ticker. Not being familiar with C, this is taking far too long. I have a few questions:

1.) Once I have opened a file using fopen, does it make more sense in terms of efficiency for the algorithm to step through the file using some function provided in the C library for scanning files, doing the compare directly from the file, or should I copy each line into an array and have the algorithm search the array?

2.) If I should compare directly from the file, what is the best way to step through it? Assume I have the number of lines in the file, is there some way to go directly to the middle line, scan the string and do the compare?

I'm sorry if this is too vague. Not too sure how to better explain. Thanks for your time

Improve this question

5 Answers 5

Reset to default
2

Unless your file is exceedingly big (> 2GB) then loading the file in memory prior searching it is the way to go. In case you cannot load the file in memory, you could hold the offset of each line in an int[] or (if the file contains too many lines...) create another binary file and write the offset of each lines as integers...

Having everything in memory is by far preferable, though.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Correct. Read into memory and use bsearch().
There's no need to store offsets of lines; you can do the binary search simply on bytes synchronizing to '\n', in O(m log n) time where n is the number of lines and m is the maximum length of any line. This works even if you can't load the file in memory and have to use fseek/fseeko.
if the file is read-only and the binary search is done repeatedly, I think having the line offset pre-calculated is even faster
2

You cannot binary search lines of a text-file without knowing the length of each line in advance, so you'll most likely want to read each line into memory at first (unless the file is very big).

But if your goal is only to search for a single given line as quickly as possible, you might as well just do linear search directly on the file. There's no point in getting O(log n) at the cost of a O(n) setup cost if the search is only done once.

Improve this answer

2 Comments

Yeah, I see the logic in that, but it's a school assignment. That's sort of what was confusing me about it. I don't see the point in implementing an algorithm for it's efficiency if it's going to setup and array by scanning a file linearly.
meburbo: OK. But it's considered good style to include a homework-tag for homework questions. I added that for you this time.
1

Reading it all in with a bulk read and walking through it with pointers (to memory) is very fast. Avoid doing multiple I/O calls if you can.

I should also mention that memory mapped files can be very suitable for something like this. See mmap() if on Unix. This is definitely your best bet for really large files.

Improve this answer

4 Comments

I had thought about mmap() as well, but the user requested a binary search as per the homework assignment. Since strings can vary in length, there's no way to have random access.
@chrisaycock: Not sure if I get your point.. What does random access have to do with mmap()?
The OP asked for binary search (I think that was just what his homework assignment required). Binary search requires random access. For a mmap()-ed array, random access is only possible if the elements are the same size, like if they are all 4-byte integers. But since his file contains strings, which can be different lengths, there's no way to jump around the list of strings. How do I know what the nth string is, for example, when I can't split by \n?
@chrisaycock: You misunderstand my answer, sorry if I didn't make it clear. mmap() was for the file I/O. This is I/O that has to be done anyway. Then the OP will have to process this data (now in memory) in order to properly search it. What I have often done is to end each line in the file with a '\0', and just storing an array of char* into the memory; each char* pointing to the beginning of a line. Now standard C string functions can be used on the array elements.
1

This is a great question!

The challenge of binary search is that the benefits of binary search come from being able to skip past half the elements at each step in O(1). This guarantees that, since you only do O(lg n) probes, that the runtime is O(lg n). This is why, for example, you can do a fast binary search on an array but not a linked list - in the linked list, finding the halfway point of the elements takes linear time, which dominates the time for the search.

When doing binary search on a file you are in a similar position. Since all the lines in the file might not have the same length, you can't easily jump to the nth line in the file given some number n. Consequently, implementing a good, fast binary search on a file will be a bit tricky. Somehow, you will need to know where each line starts and stops so that you can efficiently jump around in the file.

There are many ways you can do this. First, you could load all the strings from the file into an array, as you've suggested. This takes linear time, but once you have the array of strings in memory all future binary searches will be very fast. The catch is that if you have a very large file, this may take up a lot of memory, and could be prohibitively expansive. Consequently, another alternative might be not to store the actual stings in the array, but rather the offsets into the file at which each string occurs. This would let you do the binary search quickly - you could seek the file to the proper offset when doing a comparison - and for large stings can be much more space-efficient than the above. And, if all the strings are roughly the same length, you could just pad every line to some fixed size to allow for direct computation of the start position of each line.

If you're willing to expend some time implementing more complex solutions, you might want to consider preprocessing the file so that instead of having one string per line, instead you have at the top of the file a list of fixed-width integers containing the offsets of each string in the file. This essentially does the above work, but then stores the result back in the file to make future binary searches much faster. I have some experience with this sort of file structure, and it can be quite fast.

If you're REALLY up for a challenge, you could alternatively store the strings in the file using a B-tree, which would give you incredibly fast lookup times fir each string by minimizing the number of disk reads that you need to do.

Hope this helps!

Improve this answer

1 Comment

+1 for the effort, though I'm afraid most of what you mentioned here is going to be beyond the OP's understanding at the moment. Way beyond.
0

I don't see how you can do compare directly from the file. You will have to have a buffer to store data read from disk and use that buffer. So it doesn't make sense, it is just impossible.

You cannot jump to a particular line in the file. Not unless you know the offset in bytes of the beginning of that line relative to the beginning of the file.

I'd recommend using mmap to map this file directly into memory and work with it as with character array. Operating system will make work with file (like seeking, reading, writing) transparent to you, and you will just work with it like with a buffer in memory. Note that mmap is limited to 4 GB on 32-bit systems. But if that file is bigger, you probably need to ask the question - why on earth someone has this big file not in an indexed database.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.