How to handle unexpected argument type in python

The other answer illustrates the proper way to check in advance for problems you can foresee. I'll provide a different approach.

The idiomatic solution in python is to "ask forgiveness, not permission". There are a lot of things that can go wrong, and where other languages might ask you to foresee all those problems and address them manually, python encourages just handling them as they happen. I would recommend doing:

def test_func(nested_lists, val):
    try:
        ...
    except TypeError:
        # do whatever error-handling behavior you need to
        # either throw a custom exception or return a specific value or whatever
        return 0

and then designing your code in such a way that, if nested_lists and values are not compatible types, then they throw a TypeError (e.g. trying to iterate through nested_lists should fail if nested_lists is not a list. You can experiment with this behavior in a python console, but in general trying to do something to a variable that doesn't work because it's not the right type will produce a TypeError).

If your current code is working correctly, there is no pressing need to change anything. However, there are some reasons you might want to code more defensively;

If the code will seem to work correctly when you pass in bogus values, it would be better if it raised an exception instead of return a bogus value. The responsibility to call it correctly lies squarely with the caller, but enforcing it can help make sure the code is correct.

if not isinstance(nested_lists,list):
    raise ValueError('Need a list, got {0!r}'.format(nested_lists))

This has the drawback that it hardcodes list as the type for the first argument; properly reusable code should work with any type, as long as it has the required methods and behaviors to remain compatible with your implementation. Perhaps instead check for a behavior:

try:
    something involving nested_lists[0][0]
except (IndexError,  AttributeError):
    raise ValueError('Expected nested list but got {0!r}'.format(nested_lists))

(The try is not strictly necessary here; but see below.)

If you get a traceback when you call the code incorrectly, but it is opaque or misleading, it is more helpful to catch and explicitly point out the error earlier. #or example, the snippet above (without the try wrapper) would produce

Traceback (most recent call last):
  module __main__ line 141
    traceback.print_exc()
  module <module> line 1
    test_func(1,1)
  module <module> line 2

AttributeError: 'int' object has no attribute '__getitem__'

which is somewhat unobvious to debug.

If the code will be used by third parties, both of the above considerations will be more important from a support point of view, too.

Notice how the code raises an exception when called incorrectly. This is generally better than silently returning some garbage value, and the caller can similarly trap the error with a try/except if this is well-defined (i.e. documented!) behavior.

Finally, since Python 3.5, you have the option to use type annotations:

def test_func(nested_lists: list, val: int) -> int:
    ...

As noted in the documentation, the core language does not (yet?) enforce these type checks, but they can help static code analysis tools point out possible errors.