1

For testing I use pytest so it would be great if you suggest something pytest specific.

I have some code which uses the requests library. What it does is basically simple POST/GET requests for logging in, parsing data, etc.

Surely I want to test that code locally without doing any actual HTTP requests.

A monkeypatch funcarg could be the solution, but I think that mocking request.get(...) calls or directly pythons's urllib isn't good, because, for example, there are functions which do more than one HTTP request inside , so I can't just mock the request.get("anyURL") with a simple lambda *args, **kwaargs: """<html>response</html>""".

There are different URLs which should return different content. Sometimes it should be based on POST/GET data. Also I have no idea how will requests.session behave in case of direct mocking. Besides that how to emulate session termination? How to emulate a connection failure?

So in the end in my opinion it's quite hard to use monkey patching here. At least I am not able to write a good mocking function which will take into account everything. Also if I choose to mock urllib directly and someday requests library starts using something different all my tests will fail.

So the best way I think is to use actual HTTP server which turns on on a test run, and if possible takes into account pytest's scopes, etc (so it's a funcarg). While googling I found only two solutions:

The first one sets up the HTTP server and serves predefined content over a specific URL. Definitely that does not work for me, because as I mentioned some functions which I intend to test do several requests so all inner HTTP requests.get() will get the same answer. Bad.

The second one as far a I see has the same problem. Or at least do not understand how to use it.

The third option could be writing a small Flask based service, but I guess I'll run into a problem that things I use in tests should be tested as well which is a bad practice.

Improve this question
5
  • I think that use unittest.mock (or mock for python pre 3.4) for patching and mocking urllib.requests() is exactly what you need: you can test if your app call it correctly and the behavior on requests() return values. Commented Jan 21, 2015 at 16:21
  • I mentioned the monkeypatch funcarg which is almost the same as unittest.mock. All the problems with it are described in a topic. Does not seems as a solution. Commented Jan 21, 2015 at 18:15
  • IMHO if you are speaking about unit test (and not integration test) is better is you trust urllib and mock it even if it do several request (you can use side_effect property of mocks objects to play with that cases). If you try to test your application by mock the server you will have a lot of issue to track your bugs because your sense points will be too far from your bugs. Sure, you need even integration tests that use real server and real urllib calls but not for all your expected behaviors. You are free to experiment to mock the server but i think you will know the awful truth :) Commented Jan 21, 2015 at 20:30
  • @Michele d'Amic Maybe you are right and my unittests are more integrational tests but anyhow you call it I should test this app . Unfortunetly pytest's monkeypatch does not have a side effect feature. And even if it did there is to much I need to take into account while writing a patching function. I hoped somebody faced this problem and wrote a library which does mocking as you suggest but it includes per-URL side effect so to speak. Commented Jan 21, 2015 at 22:42
  • A per-URL side_effect is very simple to do: please spend 2 minutes to read the documentation voidspace.org.uk/python/mock/mock.html#mock.Mock.side_effect . Write a callable that map a dictionary do the work. You can use mock as is without unittest. Commented Jan 21, 2015 at 22:49

2 Answers 2

Reset to default
1

You can rather unmock get after first call.

class Requester():
    def get(*args):
         ...

def mock_get(requester, response):
    orig_get = requester.get
    def return_text_and_unmock(*args, **kwargs):
        self.get = orig_get
        return response
    requester.get = return_text_and_unmock.__get__(requester, Requester)
    return requester
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

How does this solve the problem. For example a website has a 'delete' link, which redirects you to a confirmation page. So in a function def delete(...) you do POST on url1, then second POST on url2. Also you have to take into account a token in a form, etc ... Your solution will perform second get request to the actual webserver. Also will it keep requests.session consistent.
1

I believe using a local server for unit testing is not a good idea as this is not really a unit test. I you're using requests one good way of being able to mock the requests is to use the module responses that is developed and maintained by dropbox: response dropbox. With responses you will be able to mock each request you make by specifying that you want a certain content to be return when a request is issued to a given URL. The README gives a quick overview of the module's abilities.

Improve this answer

4 Comments

Why isn't that a unit testing? When you test an arbitrary Add function which calculates something, then adds it to the database. You will set up a mysql server and put assert on "SELECT ... FROM tbl" or something like that I guess. Thanks to 'responses' I'll take a look.
IMHO if you have to setup a local server or have a database running that's an integration test as you are testing something outside of your code, something you are responsible for. Using responses will reduce the scope of the test to something you can control.
I'm not sure what benefits Dropbox's Responses have over HTTPretty though.
From what I just read on HTTPretty I would say nothing :). My answer was responses specific because it is the python package I use and I've never heard of HTTPretty before.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.