Calling a class setup function directly rather than creating a class from metaclass

Defining a function inside a class create a staticmethod in Python 3 so you don't need to use staticmethod decorator. What you might want to use is classmethod to define a method which gets the class as first arguments. You code could be rewritten as:

class GP(object):

    @classmethod
    def setup(klass, g_power):

        # ... check for valid input

        # use klass instead of `GF`
        klass.field_power = g_power
        klass.field_len = 1 << g_power

It can be used just like a you do currently GP.setup(42). What changes is that you have a reference to the current class the consequence is that you can inherit class GP an call MyChildClass.setup which will will initialize MyChildClass and its namespace and not re-use the namespace of the parent class GP. If you need to inherit and need to have separate class properties (e.g. GP.field_power and GP.field_len) for each subclass, you will need to use classmethod decorator and rework your code to use klass like it's done in the above snippet and type(self) in the methods to access the properties of the correct class e.g. type(self).field_power instead of GP.field_power.

Regarding metaclass, it depends of your usecases. Here it seems like there is only few sets of classes that can be built given the GF class template because g_power can only be 4, 8 or 16. In this case metaclass or class decorators be a good fit.*

Using class decorators you end up writing the following code:

@setupGF(8)
class GF8(GF):
    pass

setupGF will takes klass argument and do the same things as you current GF.setup.

The metaclass method looks a bit different. You have to declare your class configuration upfront too, but with a class property:

class GF8(GF, metaclass=MetaGF):
    g_power = 8

But before showing the metaclass code, this can be used also without metaclass. Say you declare a class like that:

class GF8(GF):
    g_power = 8

You can conditionally call type(self).setup() when the class is not yet initialized. This can only work if you properly defined setup as a classmethod.

The metaclass will avoid the conditional call to the setup method. So it's in this case metaclass can be used as an optimisation.

The simplest way to define a metaclass is to use a function that will construct the final class and execute setup(g_power) code:

def MetaGF(name, bases, namespace):
    # construct class just like you would dynamically create a class
    klass = type(name, bases, namespace)
    # call setup
    setup_gf_class(klass.g_power)
    # return the initialized class
    return klass

class GF8(BaseGF, metaclass=MetaGF):
    g_power = 8

But if you have more important number of class that you must generate/configure at runtime you must take a more radical path using dynamic class construction:

class BaseGF(object):
    """ implements Galois Field arithmetic for 2^4, 2^8 and 2^16
    overloads +, -, *, / and str() """

    def __init__(self, n):
        self.value = n & type(self).mask

    def __str__(self):
        return 'GF({}) element {:0>3d}d 0x{:0>2x}'.format(
            self.field_power,
            self.value,
            self.value
        )

    def __add__(self, other):
        if isinstance(other, type(self)):
            return(type(self)(self.value ^ other.value))
        else:
            raise TypeError('both args must be of GF type')

    def __sub__(self, other):
        return self + other

    def __mul__(self, other):
        if isinstance(other, type(self)):
            if self.value == 0:
                return type(self)(0)
            if other.value == 0:
                return type(self)(0)
            log_s = type(self).logs[self.value]
            log_o = type(self).logs[other.value]
            log_p = (log_s + log_o) % self.mask
            return type(self)(type(self).alogs[log_p])
        else:
            raise TypeError('both args must be of the same GF type')

    def __truediv__(self, other):
        if isinstance(other, type(self)):
            if other.value == 0:
                raise ValueError('cannot divide by 0')
            if self.value == 0:
                return type(self)(0)
            log_s = type(self).logs[self.value]
            log_o = type(self).logs[other.value]
            log_p = (log_s - log_o) % self.mask   # always returns positive
            return type(self)(type(self).alogs[log_p])
        else:
            raise TypeError('both args must be of the same GF type')


def MetaGF(g_power):
    name = 'GF(%s)' % g_power
    GF = type(name, (BaseGF,), dict())

    if g_power not in (4, 8, 16):
        raise ValueError('cant do field size of {} yet'.format(g_power))

    GF.field_power = g_power
    GF.field_len = 1 << g_power
    GF.mask = GF.field_len-1
    GF.prim_poly = (0, 0, 0, 0, 9, 0, 0, 0, 29, 0, 0, 0, 0, 0, 0, 0, 32790)[g_power]

    GF.alogs = []
    GF.logs = [None]
    sr = 1

    for index in range(GF.field_len-1):
        GF.alogs.append(sr)
        sr <<= 1
        if sr & GF.field_len:
            sr ^= GF.prim_poly
        sr &= (GF.mask)

    for sr in range(1, GF.field_len):
        GF.logs.append(GF.alogs.index(sr))

    return GF


GF8 = MetaGF(8)

print(GF8(12)+GF8(7))

# test cases from the Intel paper
print(GF8(2)*GF8(8))
print(GF8(18)*GF8(5))
print(GF8(13)/GF8(17))
print(GF8(2)/GF8(11))

This approach can also works in the other cases an is kind of the simpler both to use and to understand since it's only basic function calls. It's my prefered solution.

Have a look at How can I dynamically create derived classes from a base class for more information on how dynamic class creation works.