When you call your second script (I saved it as getoptit) with:

getoptit -d -h

This will print:

MYSQL_HOST=''  MYSQL_USER=''  MYSQL_PASS=''  BACKUP_DIR='-h' Additionals: 

So BACKUP_DIR is set, and you are testing with if [ ! "$BACKUP_DIR" ]; then if it is not set, so it is normal that the code inside of it is not triggered.

If you want to test if each option is set once, you have to do that before you do the assigment from the $OPTARG value. And you should probably also check for the $OPTARG to start with a '-' (for the -d -h error) before assigning:

...
            d)
                    if [ ! -z "$BACKUP_DIR" ]; then
                            echo "backup dir already set"
                            exit 2
                    fi
                    if [ z"${OPTARG:0:1}" == "z-" ]; then
                            echo "backup dir starts with option string"
                            exit 2
                    fi
                    BACKUP_DIR=$OPTARG
                    ;;
...

When you call your second script (I saved it as getoptit) with:

getoptit -d -h

This will print:

MYSQL_HOST=''  MYSQL_USER=''  MYSQL_PASS=''  BACKUP_DIR='-h' Additionals: 

So BACKUP_DIR is set, and you are testing with if [ ! "$BACKUP_DIR" ]; then if it is not set, so it is normal that the code inside of it is not triggered.

If you want to test if each option is set once, you have to do that before you do the assigment from the $OPTARG value. And you should probably also check for the $OPTARG to start with a '-' (for the -d -h error) before assigning:

...
            d)
                    if [ ! -z "$BACKUP_DIR" ]; then
                            echo "backup dir already set"
                            exit 2
                    fi
                    if [ z"${OPTARG:0:1}" == "z-" ]; then
                            echo "backup dir starts with option string"
                            exit 2
                    fi
                    BACKUP_DIR=$OPTARG
                    ;;
...