You are getting it wrong, ListView re-uses your rows, which means number of created rows/layouts in memory are not equal to your items in array.
Typically ListView re-sets the new data to previous row upon scroll.
I would suggest you to study this blog post, here the author is maintaing the Checked state and then setting it accordingly in getView() of adapter.
The author have created an array of bolean like this:
private boolean[] thumbnailsselection;
and storing the state of check or uncheck, and later accessing it from getView(), what you will do is, you will store true for all index and refresh your adapter. It'll select all your rows.
Here is another post.