How to read a large Json file in python to fetch certain values

In order to find the list in this json that contains a specific 'iata', you can iterate through it as a text file in byte-chunks, parsing each chunk to see if it has what you need.

Unfortunately, if the 'iata' occurs near the end of the list, then you'll still have to read your way through the whole file, although it won't all be in-memory at once.

If this is a lookup that you need to do many times, it would probably be worth it to generate a dict with the iatas as keys and countries as values. Because dictionary are hash tables, performing this sort of lookup is a very efficient task, and you'd significantly decrease the file size by only using the two elements iata and country.

Nevertheless, if I haven't dissuaded you from this course, here are functions that should parse this json as a text file in chunks, and return the country code from the iata, assuming that iatas are unique.

def read_in_chunks(file_object, chunk_size):
    while True:
        data = file_object.read(chunk_size)
        if not data:
            break
        yield data

def parse_chunk(chunk, iata):
    if iata in chunk:
        pieces = [x.strip() for x in chunk.split(',')]
        if pieces[0] != iata:
            if pieces[pieces.index(f'"{iata}"')-1].startswith('"'):
                return pieces[pieces.index(f'"{iata}"')-1].replace('"', '')
            else:
                return "fragment"
        else:
            return None


def country_from_iata(iata):    
    count = 0
    
    # attempt to find the element immediately prior to the iata
    with open('example.json', 'rt') as f:
        for chunk in read_in_chunks(f, 64):
            parsed = parse_chunk(chunk, iata)
            if parsed:
                break
            count += 64
    
    # if the element was split, then shift half an iteration to the left.
    if parsed == "fragment":
        with open('example.json', 'rt') as f:
            f.seek(count-32)
            for chunk in read_in_chunks(f, 64):
                parsed = parse_chunk(chunk, iata)
                if parsed:
                    break
    
    return parsed

country_from_iata("LAE") # 'Papua New Guinea'

If the objective is to avoid loading the whole file onto memory, then it can be done in one of the following ways:

1. Use Ijson which is "an iterative JSON parser with standard Python iterator interfaces."

2. Use a document DB to dump the json file and then read from it. You could use TinyDB for that.

3. Or you could read and process it in chunks, something like this:

from functools import partial

def custom_operation(text):
  """
  TODO: Find last '],' , process text before '],' to
  find the names and return the text after it as residual
  """
  matches, residual = [], residual
  return matches, residual

def readfile(filename)
  with open(filename, 'r') as fh:
      filepart = partial(fh.read, 1024*1024)
      iterator = iter(filepart, b'')

      residual = ''
      for index, block in enumerate(iterator, start=1):
        matches, residual = custom_operation('%s%s' % (residual, block))
        yield matches

Hope that helps!

I would personally recommend the library pandas for this kind of task. It has a built-in function for reading JSON (read_json) and tends to be more efficient than the standard library JSON offerings. Moreover, you can customize it pretty heavily for your exact use case.

Here is a reference to the Pandas read_json function: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_json.html.