How to replace specific html tags using string tokenizer

Question

I have a string with html markup in it (differMarkup) and would like to run that string through a tokenizer that would identify specific tags (like ins, dels, movs) and replace them with the span tag and add data attributes to it as well.

So the input looks like this:

`<h1>No Changes Here</h1>
    <p>This has no changes</p>
    <p id="1"><del>Delete </del>the first word</p>
    <p id="2"><ins>insertion </ins>Insert a word at the start</p>`

And intended output would be this:

`<h1>No Changes Here</h1>
    <p>This has no changes</p>
    <p id="1"><span class="del" data-cid=1>Delete</span>the first word</p>
    <p id="2"><span class="ins" data-cid=2>insertion</span>Insert a word at the start</p>
`

This is what I currently have. For some reason I'm not able to append the html tags to the finalMarkup var when setting it to span.

const (
    htmlTagStart = 60 // Unicode `<`
    htmlTagEnd   = 62 // Unicode `>`
    differMarkup = `<h1>No Changes Here</h1>
    <p>This has no changes</p>
    <p id="1"><del>Delete </del>the first word</p>
    <p id="2"><ins>insertion </ins>Insert a word at the start</p>`  // Differ Markup Output
)

func readDifferOutput(differMarkup string) string {

    finalMarkup := ""
    tokenizer := html.NewTokenizer(strings.NewReader(differMarkup))
    token := tokenizer.Token()
loopDomTest:
    for {
        tt := tokenizer.Next()
        switch {

        case tt == html.ErrorToken:
            break loopDomTest // End of the document,  done

        case tt == html.StartTagToken, tt == html.SelfClosingTagToken:
            token = tokenizer.Token()
            tag := token.Data

            if tag == "del" {
                tokenType := tokenizer.Next()

                if tokenType == html.TextToken {
                    tag = "span"
                    finalMarkup += tag
                }

                //And add data attributes
            }

        case tt == html.TextToken:
            if token.Data == "span" {
                continue
            }
            TxtContent := strings.TrimSpace(html.UnescapeString(string(tokenizer.Text())))
            finalMarkup += TxtContent
            if len(TxtContent) > 0 {
                fmt.Printf("%s\n", TxtContent)
            }
        }
    }

    fmt.Println("tokenizer text: ", finalMarkup)

    return finalMarkup

}
```golang

icza · Accepted Answer · 2020-12-02 16:06:59Z

5

Basically you want to replace some nodes in your HTML text. For such tasks it's much easier to work with DOMs (Document Object Model) than to handle the tokens yourself.

The package you're using golang.org/x/net/html also supports modeling HTML documents using the html.Node type. To acquire the DOM of an HTML document, use the html.Parse() function.

So what you should do is traverse the DOM, and replace (modify) the nodes you want to. Once you're done with the modifications, you can get back the HTML text by rendering the DOM, for that use html.Render().

This is how it can be done:

const src = `<h1>No Changes Here</h1>
<p>This has no changes</p>
<p id="1"><del>Delete </del>the first word</p>
<p id="2"><ins>insertion </ins>Insert a word at the start</p>`

func main() {
    root, err := html.Parse(strings.NewReader(src))
    if err != nil {
        panic(err)
    }

    replace(root)

    if err = html.Render(os.Stdout, root); err != nil {
        panic(err)
    }
}

func replace(n *html.Node) {
    if n.Type == html.ElementNode {
        if n.Data == "del" || n.Data == "ins" {
            n.Attr = []html.Attribute{{Key: "class", Val: n.Data}}
            n.Data = "span"
        }
    }

    for child := n.FirstChild; child != nil; child = child.NextSibling {
        replace(child)
    }
}

This will output:

<html><head></head><body><h1>No Changes Here</h1>
<p>This has no changes</p>
<p id="1"><span class="del">Delete </span>the first word</p>
<p id="2"><span class="ins">insertion </span>Insert a word at the start</p></body></html>

This is almost what you want, the "extra" thing is that the html package added wrapper <html> and <body> elements, along with an empty <head>.

If you want to get rid of those, you may just render the content of the <body> element and not the entire DOM:

// To navigate to the <body> node:
body := root.FirstChild. // This is <html>
                FirstChild. // this is <head>
                NextSibling // this is <body>
// Render everyting in <body>
for child := body.FirstChild; child != nil; child = child.NextSibling {
    if err = html.Render(os.Stdout, child); err != nil {
        panic(err)
    }
}

This will output:

<h1>No Changes Here</h1>
<p>This has no changes</p>
<p id="1"><span class="del">Delete </span>the first word</p>
<p id="2"><span class="ins">insertion </span>Insert a word at the start</p>

And we're done. Try the examples on the Go Playground.

If you want the result as a string (instead of printed to the standard output), you may use bytes.Buffer as the output for rendering, and call its Buffer.String() method in the end:

// Render everyting in <body>
buf := &bytes.Buffer{}
for child := body.FirstChild; child != nil; child = child.NextSibling {
    if err = html.Render(buf, child); err != nil {
        panic(err)
    }
}

fmt.Println(buf.String())

This outputs the same. Try it on the Go Playground.

edited Dec 2, 2020 at 16:06

answered Nov 27, 2020 at 16:00

icza

423k78 gold badges1k silver badges905 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Dzsonah Over a year ago

Thank you very much for the insightful answer! This has worked for me! The answer has been accepted!

Dzsonah Over a year ago

Is there a way to parse the output of html.Render of the body trim function into a variable? I tried several ways and could not return it. The current solution only prints these values into the console.

icza Over a year ago

@DanielBenedek Yes, it's easy. Just use bytes.Buffer instead of os.Stdout, and call its String() method in the end. Added an example of this to the answer.

Chris Sandvik Over a year ago

This is a great answer! Super thorough, and helped me with a very similar problem I was trying to solve. I was basically able to use it as is after replacing some variable names.

Collectives™ on Stack Overflow

How to replace specific html tags using string tokenizer

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related