Read YAML file and create Python objects

Question

I am newbie in Python and try to read a YAML-File. Based on its content, I want to create Python objects. I am using ruamel-yaml library. In my case, maybe I have Python-Classes Message, Signal and Signalgroup etc. (See the example file).

My approach would be to read the YAML file, check each line for a given key word and create the related object and fill it with data. I think it is the “old school” approach and maybe there is a much more effective approach to process the file.

Maybe using the function register_class/ rep. creating tags “from_yaml” but as the keys are indexed it would not work.

Message1:
Message2:
Message3:

Is there any much more professional approach?

# Yaml Testfile

- ModuleName: myTestModule
- Version: 1.0
- ModuleNumbers: [96,97,98,99]


- Message1:
   Name: AO3_
   DLC: 8
   Signal1:
     Name: Temperature
     Length: 16
   Signal2:
     Name: AnalogOut3
     Length: 16
     SignalGroup1:  #Comment
        Name: app_fcex
        Type: Bitfield
        Signal1:
            Name: drive_ready
            Length: 1
        Signal2:
            Name: error_active
            Length: 1
        Signal3:
            Name: warning_active            
            Length: 1
   Signal3:
     Name: Temperatur 2
     Length: 8
     ValueTable:
        Name: TempStates
        Item0:
           Name: INIT
           Value: 1
        Item1:
           Name: RUN
           Value: 2
        Item2:
           Name: DONE
           Value: 3
        Item3:
           Name: ERROR
           Value: 4
- Message2:
   name: AO2_
   object: RX2
   DLC: 8

Can you change the YAML file, in order to include tags that will trigger automatic object creation? Why would from_yaml not work "because the keys are indexed" that doesn't make much sense. — Anthon
– Anthon, Commented Feb 28, 2018 at 12:43
Hello Anthon, thank you for your help! I could skip the indices, that would be no problem but it would be very fine if the structure would be the same because it can be read by users very well. But how should it be implemented please? Step 1: define classes Message, Signal, Signalgroup and Valuetable Step 2: implement a classmethod def from_yaml(cls, constructor, node): return cls(*node.value.split('-')) Step 3: register the four classed using yaml.register_class(Message), .. Step 4: load the yaml file Step 5: ?? how to process it please? Best regards Gerhard — Gerhard B
– Gerhard B, Commented Feb 28, 2018 at 13:27

Anthon · Accepted Answer · 2018-02-28 14:43:11Z

I recommend you use tags in your YAML file, and drop the use of keys with names like Item1, Item2 (replace with a list of tagged objects).

It is difficult to see the exact structure your data has, but an initial step could be to make the YAML document (assumed to be in a file input.yaml:

- ModuleName: myTestModule
- Version: 1.0
- ModuleNumbers: [96,97,98,99]


- !Message
  Name: AO3_
  DLC: 8
  Signal1:
    Name: Temperature
    Length: 16
  Signal2:
    Name: AnalogOut3
    Length: 16
    SignalGroup1:  #Comment
       Name: app_fcex
       Type: Bitfield
       Signal1:
           Name: drive_ready
           Length: 1
       Signal2:
           Name: error_active
           Length: 1
       Signal3:
           Name: warning_active
           Length: 1
  Signal3:
    Name: Temperatur 2
    Length: 8
    ValueTable:
       Name: TempStates
       items:
       - !Item
         Name: INIT
         Value: 1
       - !Item
         Name: RUN
         Value: 2
       - !Item
         Name: DONE
         Value: 3
       - !Item
         Name: ERROR
         Value: 4
- !Message
  name: AO2_
  object: RX2
  DLC: 8

and load that with:

import sys
import ruamel.yaml


class Item:
    def __init__(self, name=None, value=None):
        self.name = name
        self.value = value

    @classmethod
    def from_yaml(cls, constructor, node):
        for m in constructor.construct_yaml_map(node):
            pass
        return cls(m['Name'], m['Value'])

    def __repr__(self):
        return 'Item(name={.name}, value={.value})'.format(self, self)

class Message:
    def __init__(self, name=None, DLC=None, object=None, signals=None):
        self.name = name
        self.dlc = DLC
        self.object = object
        self.signals = [] if signals is None else signals

    @classmethod
    def from_yaml(cls, constructor, node):
        for m in constructor.construct_yaml_map(node):
            pass
        if 'Name' in m:
            name = m['Name']
        elif 'name' in m:
            name = m['name']
        else:
            name = None
        object = m['object'] if 'object' in m else None
        if 'DLC' in m:
            dlc = m['DLC']
        else:
            dlc = None
        if 'signals' in m:
            signals = m['signals']
        elif 'Signal1' in m:
            x = 1
            signals = []
            while True:
                name = "Signal{}".format(x)
                try:
                    signals.append(m[name])
                except KeyError:
                    break
                x += 1
        else:
            signals = None
        return cls(name, dlc, object, signals)

    def __repr__(self):
        return 'Message(name={}, DLC={}, object={}, signals{})'.format(
            self.name, self.dlc, self.object, '[...]' if self.signals else '[]',
        )

yaml = ruamel.yaml.YAML(typ='safe')
yaml.register_class(Item)
yaml.register_class(Message)
with open('input.yaml') as fp:
    data = yaml.load(fp)

The above has some, but limited checking on key availability (and e.g. normalizes Name and name for !Message.

With the above print('data') gives (wrapping done by hand):

[{'ModuleName': 'myTestModule'}, 
 {'Version': 1.0}, 
 {'ModuleNumbers': [96, 97, 98, 99]}, 
  Message(name=Signal4, DLC=8, object=None, signals[...]), 
  Message(name=AO2_, DLC=8, object=RX2, signals[])]

and print(data[3].signals[2]['ValueTable']['items'][2]) gives:

Item(name=DONE, value=3)

Further classes should of course be added as appropriate.

Hello Anthon, thank you very much for the help!! Wow, that works -magic. I extended the yaml-File with other nested structures and it works also!! Now if I want to check, if the content of the item is valid, e.g. if the value of item "Length" is a valid integer, or if the syntax of the current yaml-line is ok, should the validation be done in the method "from_yaml" as you did it in your example or is there any other technique to do validation and if it fails to get information about line number? Thank you, Gerhard
I would do the check in the __init__ method, then your values are checked when creating an object from scratch as well. Just raise an error there and catch it in from_yaml(), where you have the original node, that has an attribute start_mark (c.f. nodes.py) and that has line and column attributes (c.f. error.py). Post a new question if you can't figure it out from that. BTW If this answers solves your problem, please mark as such, so others know this is a valid solution (without having to read the comments).

Collectives™ on Stack Overflow

Read YAML file and create Python objects

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related