0

This script bash makes the data extraction to Oracle DB (another server), processing it and insert into MariaDB (my server), but the insertion it's misaligned and let it some columns in blank.

  • This is the code:
    #!/bin/bash
    
    ORACLE_USER="user"
    ORACLE_PASSWORD="password"
    ORACLE_DB="IP/SID"
    
    MYSQL_USER="user"
    MYSQL_PASSWORD="password"
    MYSQL_DB="DB"
    
    echo "Fetching data from Oracle..."
    ORACLE_DATA=$(sqlplus -s "$ORACLE_USER/$ORACLE_PASSWORD@$ORACLE_DB" <<EOF
    SET PAGESIZE 50000
    SET COLSEP '|'
    SET LINESIZE 1000
    SET HEADING OFF
    SET FEEDBACK OFF
    SET TERMOUT OFF
    select TRIM(REGION), TRIM(CENTRAL), TRIM(NOMBRE_BANCO), TRIM(MODELO), TRIM(BATERIA), TRIM(TECNOLOGIA_ID), TRIM(AMPERAJE_CA), TO_CHAR(MEDIDO, 'YYYY-MM-DD') AS MEDIDO, TRIM(PORCENTAJE), TRIM(VOLTAJE), TRIM(VOLTAJE_AC), TO_CHAR(CREADO, 'YYYY-MM-DD') AS CREADO
    from SIMBA_BD.VIEW_CENTRAL_MEDICIONES_PLOMO where rownum<=10;
    EXIT;
    EOF
    )
    
    echo "Data fetched from Oracle:"
    echo "$ORACLE_DATA"
    
    echo "Truncating the MariaDB table..."
    mysql -u"$MYSQL_USER" -p"$MYSQL_PASSWORD" "$MYSQL_DB" <<EOF
      TRUNCATE TABLE SIMBA;
    EOF
    
    echo "Inserting data into MariaDB..."
    while IFS='|' read -r REGION CENTRAL NOMBRE_BANCO MODELO BATERIA TECNOLOGIA_ID AMPERAJE_CA MEDIDO PORCENTAJE VOLTAJE VOLTAJE_AC CREADO; do
    
    REGION=$(echo "$REGION" | tr -d '\r\n')
    CENTRAL=$(echo "$CENTRAL" | tr -d '\r\n')
    NOMBRE_BANCO=$(echo "$NOMBRE_BANCO" | tr -d '\r\n')
      MODELO=$(echo "$MODELO" | tr -d '\r\n')
    BATERIA=$(echo "$BATERIA" | tr -d '\r\n')
    TECNOLOGIA_ID=$(echo  "$TECNOLOGIA_ID" | tr -d '\r\n')
    AMPERAJE_CA=$(echo "$AMPERAJE_CA" | tr -d '\r\n')
    MEDIDO=$(echo "$MEDIDO" | tr -d  '\r\n')
    PORCENTAJE=$(echo "$PORCENTAJE" | tr -d '\r\n')
    VOLTAJE=$(echo "$VOLTAJE" | tr -d '\r\n')
    VOLTAJE_AC=$(echo "$VOLTAJE_AC" | tr -d '\r\n')
      CREADO=$(echo "$CREADO" | tr -d '\r\n')
    
    REGION=${REGION:-""}
    CENTRAL=${CENTRAL:-""}
    NOMBRE_BANCO=${NOMBRE_BANCO:-""}
    MODELO=${MODELO:-""}
    BATERIA=${BATERIA:-""}
    TECNOLOGIA_ID=${TECNOLOGIA_ID:-""}
    AMPERAJE_CA=${AMPERAJE_CA:-0}
    MEDIDO=${MEDIDO:-"NULL"}
    PORCENTAJE=${PORCENTAJE:-0}
    VOLTAJE=${VOLTAJE:-0}
    VOLTAJE_AC=${VOLTAJE_AC:-0}
    CREADO=${CREADO:-"NULL"}
    
    echo "Region: $REGION"
        echo "Central: $CENTRAL"
        echo "Nombre_Banco: $NOMBRE_BANCO"
        echo "Modelo: $MODELO"
        echo "Bateria: $BATERIA"
        echo "Tecnologia: $TECNOLOGIA_ID"
        echo "Amperaje_CA: $AMPERAJE_CA"
        echo "Medido: $MEDIDO"
        echo "Porcentaje: $PORCENTAJE"
        echo "Voltaje: $VOLTAJE"
        echo "Voltaje_AC: $VOLTAJE_AC"
        echo "Creado: $CREADO"
    
    if ! [[ "$MEDIDO" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]]; then
            MEDIDO="NULL"
      fi
    
    if ! [[ "$CREADO" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]]; then
            CREADO="NULL"
      fi
    
    if ! [[ "$AMPERAJE_CA" =~ ^-?[0-9]+([.][0-9]+)?$ ]]; then
             AMPERAJE_CA=0
      fi
    
    if ! [[ "$PORCENTAJE" =~ ^-?[0-9]+([.][0-9]+)?$ ]]; then
            PORCENTAJE=0
    fi
    
    if ! [[ "$VOLTAJE" =~ ^-?[0-9]+([.][0-9]+)?$ ]]; then
            VOLTAJE=0
      fi
    
    if ! [[ "$VOLTAJE_AC" =~ ^-?[0-9]+([.][0-9]+)?$ ]]; then
            VOLTAJE_AC=0
      fi
    
      mysql -u"$MYSQL_USER" -p"$MYSQL_PASSWORD" "$MYSQL_DB" <<EOF
      INSERT INTO SIMBA (Region, Central, Nombre_Banco, Modelo, Bateria, Tecnologia, Amperaje_CA, Medido, Porcentaje, Voltaje, Voltaje_AC, Creado) VALUES ('$REGION', '$CENTRAL', '$NOMBRE_BANCO', '$MODELO', '$BATERIA', '$TECNOLOGIA_ID', '$AMPERAJE_CA', IF('$MEDIDO' = 'NULL', NULL, '$MEDIDO'), '$PORCENTAJE', '$VOLTAJE', '$VOLTAJE_AC', IF('$CREADO' = 'NULL', NULL, '$CREADO'));
    EOF
    
    done <<< "$ORACLE_DATA" >> SALIDA.TXT
    
    echo "Data insertion completed."
    
  • And the results are like this:
    Region:
    
    Central:
    
    Nombre_Banco:
    
    Modelo:
    
    Bateria:
    
    Tecnologia:
    
    Amperaje_CA:
    
    Medido:
    
    Porcentaje:
    
    Voltaje:
    
    Voltaje_AC:
    
    Creado:
    
    Region: REGION 7
    
    Central: LERFIC
    
    Nombre_Banco: 7LERFICCT1B6
    
    Modelo: BATERIA GNB ABSOLYTE, 100A-19 896AH
    
    Bateria:
    
    Tecnologia:
    
    Amperaje_CA:
    
    Medido:
    
    Porcentaje:
    
    Voltaje:
    
    Voltaje_AC:
    
    Creado:
    
    Region: J10B                                                                                                                        
    
    Central: TYPHOON(TY1)
    
    Nombre_Banco: 0
    
    Modelo: 2016-12-30
    
    Bateria: 54.45
    
    Tecnologia: 2.25
    
    Amperaje_CA: 0
    
    Medido: 2017-01-10
    
    Porcentaje:
    
    Voltaje:
    
    Voltaje_AC:
    
    Creado:
    
1
  • 4
    (1) Please provide example data exactly as output from your Oracle database and the corresponding output from your script. (2) Please frame a question so we know with which part of the issue you want help. (3) Please identify which of the outputs you show are correct / incorrect. You should update your question to include this information. Don't try to reply here in the comments. Commented Oct 29, 2024 at 18:05

1 Answer 1

2

I looked at this, and it's a clustershambles; we shouldn't fix that; the idea that you take unsanitized data from a database system, let the database print it as table, then parse that table in a shell script to then generate SQL inserts is just too complicated and error prone.

This could be a much smaller program in nearly any other scripting language, where you don't have to get the rows from the database query result as text and then split them back into the values yourself in the least useful syntax for that.

For example, in python you could just do the query and then literally go and insert all the results after filtering at once. Single line of code.

I'm afraid sometimes the honest answer to "how do I fix my shell script is" that you need to stop using the shell for things that the shell was really not built for.

There's other pitfalls that you would avoid by doing that and using the proper prepared statements instead of building your own insert(…) … VALUES($VARIABLE1, $VARIABLE2,…) string. Everyone loves Bobby Tables:

XKCD  327, "Exploits of a Mom"
XKCD 237: "Exploits of a Mom" by Randall Munroe; under CC-by-NC 2.5

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.