None values after handle ValueError in a request?

Question

I am making some requests to an API which postag the following text as follows:

def pos(text):
    payload = {'key': 'thekey', 'of': 'json', 'ilang': 'ES', \
               'txt': text, \
               'tt': 'a', \
               'uw': 'y', 'lang': 'es'}

    r = requests.get('http://api.meaningcloud.com/parser-2.0', params=payload, stream = True)
    return r.json()

At the beginning, it gave me a ValueError:

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
<ipython-input-19-ac09c6405340> in <module>()
      1 
----> 2 df['tags'] = df['tweets'].apply(transform)
      3 df

/usr/local/lib/python3.5/site-packages/pandas/core/series.py in apply(self, func, convert_dtype, args, **kwds)
   2292             else:
   2293                 values = self.asobject
-> 2294                 mapped = lib.map_infer(values, f, convert=convert_dtype)
   2295 
   2296         if len(mapped) and isinstance(mapped[0], Series):

pandas/src/inference.pyx in pandas.lib.map_infer (pandas/lib.c:66124)()

<ipython-input-18-707ac7b399b4> in transform(a_lis)
     25 
     26 def transform(a_lis):
---> 27     analysis = pos(str(a_lis))
     28     a_list = parse_tree(analysis['token_list'], [])
     29     return a_list

<ipython-input-18-707ac7b399b4> in pos(text)
      8 
      9     r = requests.get('http://api.meaningcloud.com/parser-2.0', params=payload, stream = True)
---> 10     return r.json()
     11 
     12 def parse_tree(token, a_list):

/usr/local/lib/python3.5/site-packages/requests/models.py in json(self, **kwargs)
    864                     # used.
    865                     pass
--> 866         return complexjson.loads(self.text, **kwargs)
    867 
    868     @property

/usr/local/lib/python3.5/site-packages/simplejson/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, use_decimal, **kw)
    514             parse_constant is None and object_pairs_hook is None
    515             and not use_decimal and not kw):
--> 516         return _default_decoder.decode(s)
    517     if cls is None:
    518         cls = JSONDecoder

/usr/local/lib/python3.5/site-packages/simplejson/decoder.py in decode(self, s, _w, _PY3)
    368         if _PY3 and isinstance(s, binary_type):
    369             s = s.decode(self.encoding)
--> 370         obj, end = self.raw_decode(s)
    371         end = _w(s, end).end()
    372         if end != len(s):

/usr/local/lib/python3.5/site-packages/simplejson/decoder.py in raw_decode(self, s, idx, _w, _PY3)
    398             elif ord0 == 0xef and s[idx:idx + 3] == '\xef\xbb\xbf':
    399                 idx += 3
--> 400         return self.scan_once(s, idx=_w(s, idx).end())

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Therefore I handled the exception and applied it to a pandas dataframe column with:

df = pd.read_csv('../data.csv')
df['tagged_text'] = df['tweets'].apply(transform)

However, with some instances (columns) I got None:

text                                                tagged_text
Siento que estoy en un cuarto oscuro y hay sil...   [(sentar, VI-S1PSABL-N4), (que, CSSN9), (estar...
Los mejores de @UEoficial Sebastián Jaime, Sey...    None
#ColoColoJuegaEnEl13 la primera y adentro mier...    None
Juguito heladoooo de melón: me siento se...          None
@sxfiacrespo @lunasoledadhern Hola Luna...          [(@sxfiacrespo @lunasoledadhern, NPUU-N-), (ho...

Thus, my question is why at some texts (columns) I am getting None and how can I correctly tag those None instances?. Note that I made some tests and there is no problem with the text, since for those None a json with all the tagged content is returned. For example consider this function application.

Sounds like you did r.json[] instead (square brackets). No, a method object is indeed not subscriptable, but it is callable. Make sure you use () (round parentheses)! — Martijn Pieters
– Martijn Pieters, Commented Apr 4, 2017 at 17:06
Please don't edit changes from my answer into your question; that invalidates my answer and leaves future visitors perplexed as to what happened. — Martijn Pieters
– Martijn Pieters, Commented Apr 4, 2017 at 17:20

Martijn Pieters · Accepted Answer · 2017-04-04 19:02:01Z

2

This does next to nothing:

except ValueError:
    np.nan

That only references the np.nan object. If you want to return it, you need to do so explicitly:

except ValueError:
    return np.nan

otherwise the function just.. ends, which means None is returned.

Other notes:

r = requests.get('http://api.meaningcloud.com/parser-2.0', data=payload, stream = True)
json_data = json.dumps(r.json())
data = yaml.load(json_data)
return data

is a really expensive way of spelling

r = requests.post('http://api.meaningcloud.com/parser-2.0', data=payload)
return r.json()

Loading JSON into Python, then producing JSON again, then using a YAML parser to turn the JSON back to Python is somewhat excessive. I also removed the stream=True; that's only needed when you want to process the response data as a stream (which the response.json() method doesn't do).

According to the API documentation, txt is supposed to be a single string.I'd not use str(a_lis) to produce that; if you have a list of strings, just join those into one long string with ' '.join(a_lis). However, I'm sure that pandas.Series.apply() passes in individual values (e.g. strings) to your function, at which point there is no need to join anything at all (but your a_lis variable name is very confusing in that case).

The API also specifies that it uses POST requests (I'm surprised they accept GET still anyway). Using a POST request (requests.post()) will allow you to send much larger pieces of text for analysis. Use the data keyword. I've used the correct syntax in my last sample above.

That you used GET is also the reason you get a ValueError:

>>> r  = requests.get('http://api.meaningcloud.com/parser-2.0', params=payload)
>>> r.status_code
414
>>> r.reason
'Request-URI Too Long'
>>> r = requests.post('http://api.meaningcloud.com/parser-2.0', data=payload)
>>> r.status_code
200

edited Apr 4, 2017 at 19:02

answered Apr 4, 2017 at 16:58

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

Martijn Pieters Over a year ago

@J.Do: I can't help with your ValueError unless you can provide a sample data set (make it small) that reproduces that error.

Martijn Pieters Over a year ago

@J.Do: that's not really a small sample data set; I was hoping for the return value of api_call(), reduced to a minimal example for which parse_tree() returns np.nan.

Martijn Pieters Over a year ago

@J.Do: it looks like the ValueError comes from your use of requests, because I can't reproduce it with my code.

Martijn Pieters Over a year ago

@J.Do: You could just remove the try:...except: and include the full traceback this produces in your post, so I can confirm..

Martijn Pieters Over a year ago

@J.Do: yup, that's because you are using GET, not POST. The server gives you a 414 Request-URI Too Long error, which is why I strongly urge you to use POST requests (which are not limited by URL length).

|

Collectives™ on Stack Overflow

None values after handle ValueError in a request?

1 Answer 1

8 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Related