getting invalid document error while querying mongodb database

The location in your query should be a string, enclose it in single or double quotes. Probably you have a function named location elsewhere in the module (or in other modules called with form module_name import *).

You can make the code simpler/more readable with some edits:

# since REST APIs have the get and set verb embedded into the HTTP method,
# just use "books" (this is a personal preference, though)
@app.route('/books', methods=['GET', 'POST'])
def requesting_for_books():
    mydb = mongo.db.mylogin
    # it's better to handle exception as soon as they arise,
    # with the added bonus that you don't need to check 
    # if email is in session in the following statements
    try:
        user_col = mydb.find_one({"email" : session['email']})
    except KeyError:
        flash("Please Sign in first !")
        return render_template('index.html', title="Home")
    # user_col is already a dictionary, no need to transform it
    verify = user_col['isVerified']
    form = RequestBooks()
    # if the code runs until here, then email must be in session, no need to check
    # `== True` can be omitted
    if verify and 'password' in session and request.method == 'GET':
        return render_template(
           'get_books.html',
           title="Get Books",
           donated_by=session['username'],
           form=form
        )
    # since you returned in the previous if,
    # there's no need for an else/elif statement, just use if again
    if request.method == 'POST':
        # I suppose this is wrong,
        # it should be point to the books collection and not mylogin collection
        requesting = mongo.db.mylogin
        # if you need only the first element, just use find_one
        request_books = requesting.find_one({
            "location": {
                "$nearSphere": {
                    "$geometry" : {
                        "type" : "Point",
                        "coordinates": [session['lat'], session['lang']]
                    }
                 }
            }
        })
        # you can pass request_books to render_template even if it is None
        # and handle the presence of data directly in the template
        return render_template(
            'get_books.html',
            title="Get Books",
            request_books=request_books,
            form=form
        )
    # no need to use "else", since you returned from the other branches
    flash('Please Login/Sign Up first to Search for books !')
    return redirect(url_for('home'))

If you would like to get the list of the requested books, you can simply use the find function and convert the cursor into a list, so you'll get a list of dictionaries:

request_books = list(requesting.find({
    "location": {
        "$nearSphere": {
            "$geometry" : {
                "type" : "Point",
                "coordinates": [session['lat'], session['lang']]
            }
        }
    }
}))