Loading nested objects in yaml files with custom constructors

Question

I have a yaml file describing a series of nested objects that I am working on importing. Because I want to have __init__ on each get called, I am writing custom constructors

def volume_constructor(loader, node):
    instance = Volume.__new__(Volume)
    yield instance
    state = loader.construct_mapping(node)
    instance.__init__(**state)

Then when I am reading in the yaml file I am adding the constructors before doing the load:

yaml.add_constructor(volume_yaml_tag, volume_constructor, yaml.SafeLoader)
yaml.add_constructor(host_yaml_tag, volume_constructor, yaml.SafeLoader)
configcontent = configfile.read()
cfg = yaml.safe_load(configcontent)

In this case the volume is the parent object which contains several properties as normal strings etc along with a list of host objects. The excerpt below shows the general form with values omitted for brevity

- !!python/object:libraries.volume.Volume
_Volume__consuminghostlist:
- !!python/object:libraries.host.Host
  _Host__property1: fc
  _Host__property2: localhost
_Volume__property1: false
_Volume__property2: sometextvalue
_Volume__property3: somenumericvalue

I can define and dump the objects without issue however when I attempt the load with the custom constructors as defined above I get the error stack below. Note that when I change the order of the properties so that the first one processed isn't a nested object the error still flags the first thing processed.

        cfg = yaml.safe_load(configcontent)
  File "....yaml\__init__.py", line 162, in safe_load
    return load(stream, SafeLoader)
  File "...yaml\__init__.py", line 114, in load
    return loader.get_single_data()
  File "...yaml\constructor.py", line 43, in get_single_data
    return self.construct_document(node)
  File "...yaml\constructor.py", line 52, in construct_document
    for dummy in generator:
  File "...volume.py", line 15, in volume_constructor
    instance.__init__(**state)
TypeError: __init__() got an unexpected keyword argument '_Volume__consuminghostlist'

If however I forgo adding the custom constructors and do a stock load, the load is successful though at that point __init__ for each is no longer called. Is there another way I should be going about having __init__ invoked or is there something wrong with the way the constructor is setup? I imagine there must be some way to avoid doing a manual parsing of each tag on the constructor and feeding them as individual parameters to init_. I toyed with setting the deepcopy property but it didn't make a difference. For reference the config file was created using:

yaml.dump(cfg, sys.stdout, Dumper=noalias_dumper, default_flow_style=False)

update: Several responses mentioned the invalid yaml formatting which I agree doesn't look right so I did some experimenting to see why that was happening. I created the yaml by defining the objects and then dumping them rather than writing it by hand so I would have expected the dump to use valid yaml formatting. I found as shown below that this gets mangled when I have @property decorators setup for a property

import yaml
import sys
class Volume(yaml.YAMLObject):
    yaml_loader = yaml.SafeLoader

    def __init__(self, input_property1, input_property2):
        self.property1 = input_property1
        self.property2 = input_property2

    @property
    def property1(self):
        return self.__property1

    @property1.setter
    def property1(self, input_property1):
        self.__property1 = input_property1


samplevolume = Volume("ABC", 1.0)
noalias_dumper = yaml.dumper.Dumper
noalias_dumper.ignore_aliases = lambda self, data: True
yaml.dump(samplevolume, sys.stdout, Dumper=noalias_dumper, default_flow_style=False)

when I run the sample code above I get the dump output below showing the expected treatment for the property that doesn't have @property definitions

!!python/object:__main__.Volume
_Volume__property1: ABC
property2: 1.0

updated 2 Based on the discussion with @flyx I tried aligning the names as shown below

import yaml
import sys
class Volume(yaml.YAMLObject):
    yaml_loader = yaml.SafeLoader

    def __init__(self, property1, property2):
        self.property1 = property1
        self.property2 = property2

    @property
    def property1(self):
        return self.property1

    @property1.setter
    def property1(self, property1):
        self.__property1 = property1


samplevolume = Volume("ABC", 1.0)
noalias_dumper = yaml.dumper.Dumper
noalias_dumper.ignore_aliases = lambda self, data: True
yaml.dump(samplevolume, sys.stdout, Dumper=noalias_dumper, default_flow_style=False, allow_unicode=True)

however it doesn't change the output formatting

!!python/object:__main__.Volume
_Volume__property1: ABC
property2: 1.0

given that using property decorators is a core feature of python I'm surprised that pyyaml requires custom representers and constructors to deal with it.

Please create a minimal, reproducible example. The excerpt you show is not valid YAML and therefore it's unclear what is happening, and your statement that „values are omitted“ doesn't help; neither does the missing definition of Volume. — flyx
– flyx, Commented Jan 5, 2020 at 0:39
@flyx I put an update in the original post exploring why the yaml formatting was odd. In short I created it using a dumper rather than by hand and the use of property decorators seems to drive the strange formatting. — ragingwookie
– ragingwookie, Commented Jan 6, 2020 at 14:14

flyx · Accepted Answer · 2020-01-06 21:24:27Z

1

The class Volume you show has an __init__ expecting two parameters: input_property1 and input_property2. However, the default representer represented the first property with the name _Volume__property1. In your call to __init__, this cannot be mapped because it doesn't correspond to the names of __init__'s parameters.

So your options are:

Edit __init__'s parameter names so that they match with the names the parameters get serialized with
Add custom representers that represent the objects with the field names that are expected by __init__
Use YAMLObject's default loader and put your custom code in a normal method that you call afterwards. e.g.:

def volume_constructor(loader, node):
    instance = loader.construct_yaml_object(loader, Volume)
    instance.myinit()
    return instance

This way, loading Volume is compatible with the default dumper, because you use the default loader. You then put any additional initialization code into the method myinit.

answered Jan 6, 2020 at 21:24

flyx

40.6k10 gold badges107 silver badges152 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

ragingwookie Over a year ago

aligning the names (property1 vs input_property1) didnt resolve the issue as noted in the 2nd update above. I would have expected pyyaml to handle property decorators since they are a core python concept. Now I am weighing writing custom representers for everything vs just making my peace with the invalid format since it works and cloning the objects after performing the safe load so that init gets called properly

flyx Over a year ago

The problem is not with PyYAML. PyYAML works perfectly with your class if you use the standard representer and constructor, as would be the case with my third proposed solution. If you want to align the names, the proper parameter name is _Volume__property1, not property1, as you can see in your YAML output. This is caused by private name mangling and is a standard Python feature. If you want to have a custom constructor but retain the standard representer, you have to play by the rules.

Anthon · Accepted Answer · 2020-01-05 08:47:27Z

0

You don't process the keyword _Volume__consuminghostlist in the __init__ of your Volume class, and of course you need to do that.

Your volume_constructor assumes that the tagged node is a mapping (since you call construct_mapping in that function). Although your YAML is invalid, one thing that is clear is that forementioned mapping has a key _Volume__consuminghostlist. So the python dictionary state will have that key, and since you pass that to __init__ with **, this will work as if you called:

__init__(_Volume__consuminghostlist: <value for _Volume__consuminghostlist>)

and you have to take care of that keyword.

If you do things this way, this is required in ruamel.yaml as well as in PyYAML.

answered Jan 5, 2020 at 8:47

Anthon

78.3k35 gold badges207 silver badges290 bronze badges

1 Comment

ragingwookie Over a year ago

@Athon I put an update in the original post exploring why the yaml formatting was odd. In short I created it using a dumper rather than by hand and the use of property decorators seems to drive the strange formatting.

Collectives™ on Stack Overflow

Loading nested objects in yaml files with custom constructors

2 Answers 2

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related