Why does my program not work with large numbers?

Answers provided by @DanyalImran and @Jean-FrançoisFabre are both incorrect. It is co-incidence, that 600851475143 is a product of 71, 839, 1471 and 6857 and all divisors are less than sqrt(num). What if the number in the OP would be 600851475149 (the prime number)?

Thus we need to search a full range [2,num], not the range [2,sqrt(num)].

So, here is what I got after a few attempts to make the search optimized using both Sieve of Eratosthenes algorithm to precompute a vector of prime flags and memorizing previously found primes.

Although the Sieve of Eratosthenes is really a fastest way to find all primes in some specified range (it has no division operations, only multiplications that are times faster than divisions), this approach can't help much as it does not eliminate the need to loop through the vector to find elements marked as primes and then divide the number in question with the found prime (I intentionally replaced vector<bool> with vector<char> in the @Jean-FrançoisFabre's implementation to avoid possible 'bit-packed' implementation of a vector<bool> as a bit position in a vector calculation is definitely more expensive than a char position calculation).

The time I get this way to solve the task in OP for 150212868857 prime is ~7:05 minutes on my 1.4GHz AMD:

150212868857

real    7m5.156s
user    7m5.063s
sys 0m0.008s

The attempt to memorize all previously found primes to speedup the isPrime() test is even worse, so I did not give it a chance to finish. This explained by the same necessity to iterate through a vector of primes and it is even more expensive due to amount of data to be read from memory.

The final variant is just an iteration of the candidate divisor in the range from 3 to num with step 2 and calling the isPrime only when num is even to candidate. This approach shows the same time as previous plus-minus a few seconds. Thus, the access to a vector element appears to be as expensive as a division as soon as the math used fits into native registers of a modern CPU.

However, when the number in question is not prime (as in the OP) there is still a place for optimization that allows to shorten the search time.

The code:

#include <iostream>
#include <vector>
#include <math.h>

using namespace std;

//#define SIEVE
vector<char> primeFlagger;

void initSieve(unsigned long long int limit) {
    unsigned long long int root = (unsigned long long int)(sqrt(limit));
    primeFlagger = vector<char>(root+1,true);
    primeFlagger[0] = primeFlagger[1] = false;

    for(unsigned long long int j=2; j<=root; j++)
    {
        if(primeFlagger[j])
        {
            for(unsigned long long int k=j; (k*j)<=root; k++)
            {
                primeFlagger[j*k]=false;
            }
        }
    }
}

#ifdef SIEVE
bool isPrime(unsigned long long int num)
{
    if (num <= 2)
        return true;

    if ((num % 2) == 0)
        return false;

    unsigned sqr = (unsigned)sqrt(num);
    for(unsigned i = 3; i <= sqr; i+=2) {
        if (primeFlagger[i] && num % i == 0)
            return false;
    }

    return true;
}
#else
bool isPrime(unsigned long long int num)
{
    if (num <= 2)
        return true;

    if ((num % 2) == 0)
        return false;

    unsigned sqr = (unsigned)sqrt(num);
    for(unsigned i = 3; i <= sqr; i+=2) {
        if (num % i == 0)
            return false;
    }

    return true;
}
#endif

int main() {
    unsigned long long int  m = 600851475143;//150212868857;//600851475149;
    std::vector<unsigned long long int> pfactors;

#ifdef SIEVE
    initSieve(m);
#endif

    if (m % 2 == 0) {
        do {
            m /= 2;
        } while (m % 2 == 0);
        pfactors.push_back(2);
    }

    for (long long int i = 3; i <= m; i+=2) {
        if (m % i == 0 && isPrime(i)) {
            do {
                m /= i;
            } while (m % i == 0);
            pfactors.push_back(i);
        }
    }

    for (vector<unsigned long long int>::iterator it = pfactors.begin(); it != pfactors.end(); it++) {
        cout << *it << endl;
    }

    return 0;
}

The result with number in the OP:

$ g++ -O3 prime1.cpp
$ time ./a.out 
71
839
1471
6857

real    0m0.004s
user    0m0.002s
sys 0m0.002s

600851475143 is a very large number and your algorithm is very inefficient. I suspect that it would eventually terminate, but that "eventually" is far beyond the time you'd be willing to wait.

The above is assuming that long long on your platform is even capable of representing the number - if it is only 32bits then it is not.

What you really need is a more efficient algorithm.

your program fails because you are using int data type. It has 32 bits, i.e 2^32 ~= 9 digits max

To access some values greater than 9 digits, try using long long or unsigned long long. It has 64 bits, i.e 2^64 ~= 18 digits for signed and 20 digits for unsigned.

Edit: ^This is for C++ 98 (Orwell Dev C++: Tested)

vector<int> pfactor;

This int on vector is making your program fail. And you don't need to reserve any space in a vector since its dynamic unless you are working on 2d Vectors.

Example Code:

#include<math.h>
#include<iostream>
using namespace std;

bool ifPrime ( int number ) {
    for ( long long int i=2; i<sqrt(number); ++i ) {
        if ( number%i == 0 ) return false;
    }

    return true;
}

int main ( ) {
    long long int number = 600851475143;
    static long long int largestPrime;

    bool found = false;
    for ( long long int i=sqrt(number); i>=0 && !found; i=i-1 ) {
        if ( number%i==0 && ifPrime(i) ) {
            largestPrime = i;
            found = true;
        }
    }

    cout << "Largest Prime for " << number << " is: " << largestPrime << endl;
}