I have a table with the unique constraint
CREATE UNIQUE INDEX "bd_hash_index" ON "public"."bodies" USING btree ("hash");
I also have a Go program that takes "body" values on a channel, filters out the duplicates by hashing, and inserts only the non-duplicates into the database. Like this:
import (
"crypto/md5"
"database/sql"
"encoding/hex"
"log"
"strings"
"time"
)
type Process struct {
DB *sql.DB
BodiesHash map[string]bool
Channel chan BodyIterface
Logger *log.Logger
}
func (pr *Process) Run() {
bodyInsert, err := pr.DB.Prepare("INSERT INTO bodies (hash, type, source, body, created_timestamp) VALUES ($1, $2, $3, $4, $5)")
if err != nil {
pr.Logger.Println(err)
return
}
defer bodyInsert.Close()
hash := md5.New()
for p := range pr.Channel {
nowUnix := time.Now().Unix()
bodyString := strings.Join([]string{
p.GetType(),
p.GetSource(),
p.GetBodyString(),
}, ":")
hash.Write([]byte(bodyString))
bodyHash := hex.EncodeToString(hash.Sum(nil))
hash.Reset()
if _, ok := pr.BodiesHash[bodyHash]; !ok {
pr.BodiesHash[bodyHash] = true
_, err = bodyInsert.Exec(
bodyHash,
p.GetType(),
p.GetSource(),
p.GetBodyString(),
nowUnix,
)
if err != nil {
pr.Logger.Println(err, bodyString, bodyHash)
}
}
}
}
But periodically I get the error
"pq: duplicate key value violates unique constraint "bd_hash_index""
in my log file. I can't image how it can be, because I check the hash for uniqueness before I do an insert.
I am sure that when I call go processDebugBody.Run() the bodies table is empty.
The channel was created as a buffered channel with:
processDebugBody.Channel = make(chan BodyIterface, 1000)
2015-05-13 20:12:42 CEST LOG: SSL renegotiation failure. All time, when i found dublicate error, this row stand before it.boolmap (rather than the smaller map ofstruct{}) as a set is to make the existence check a simpleif pr.BodiesHash[bodyHash]without needing the longer comma ok form.