Bash script for TXT file to JSON conversion parsing issue

Question

I have to create a simple bash script that converts .txt outputs to JSON code. The pattern is as follows:

[ Example ], 1..3 tests
-----------------------------------------------------------------------------------
not ok  1  text1 (text1), 5ms
ok  2  text2 (text2, issues), 2ms
ok  3  text3 (text3, issues), 15ms
-----------------------------------------------------------------------------------
2 (of 3) tests passed, 1 tests failed, rated as 66.66%, spent 22ms

This will have to be converted to the following JSON (as an example):

{
 "testName": " Example ",
 "tests": [
   {
     "name": "",
     "status": false,
     "duration": ""
   },
   {
     "name": "",
     "status": true,
     "duration": ""
   }
 ],
 "summary": {
   "success": ,
   "failed": ,
   "rating": ,
   "duration": ""
 }
}

I managed to almost complete the requirements, the test name is taken out correctly from the txt and also the summary results are taken correctly, but the only problem is with the tests part which does not work at all, not even the variables set upfor that are recognosed.

My code is:

#!/bin/bash

###################PREREQUISITES#####################

# Check if the output file is provided as an argument (I chose to do so to be able tospecify location)

if [ $# -eq 0 ]; then
  echo "Usage: $0 <output_json_file>"
  exit 1
fi

input_file="example_file.txt"  #File that is being used as source to convert FROM
output_file=$1    #Output file location as an argument to convert TO

# Check if the input file exists

if [ ! -f "$input_file" ]; then
  echo "Input file '$input_file' not found."
  exit 1
fi

###################CODE#####################

#Setting up a pattern variable for the lines with the values that we need as JSON variables:

pattern='"^(ok|not ok)[[:space:]]{2,}([0-9])[[:space:]]{2,}(.+),[[:space:]]+([0-9]{1,3}ms)$"'

# Read the input_file line by line:

while IFS= read -r line; do
  # Extract the test_Name from the first line -> IT WORKS
  if [[ "$line" =~ \[([^\]]+)\] ]]; then
    testName="${BASH_REMATCH[1]}"
  # Extract values using BASH_REMATCH on the pattern variable -> IT DOES NOT WORK
  elif [[ "$line" =~ $pattern ]]; then
      status="${BASH_REMATCH[1]}"
      status_code="${BASH_REMATCH[2]}"
      name="${BASH_REMATCH[3]}"
      duration="${BASH_REMATCH[4]}"
    break
  elif [[ "$line" =~ ^([0-9]+)\ \(of\ [0-9]+\)\ tests\ passed,\ ([0-9]+)\ tests\ failed,\ rated\ as\ ([0-9.]+)%,\ spent\ ([0-9]+)ms$ ]]; then
      success="${BASH_REMATCH[1]}"
      failed="${BASH_REMATCH[2]}"
      rating="${BASH_REMATCH[3]}"
      duration="${BASH_REMATCH[4]}ms"
    break  
  fi
done < $input_file

# Construct the JSON output
output_json="{
 \"testName\": \"$testName\",
 \"tests\": [
   ${tests[@]}
 ],
 \"summary\": {
   \"success\": $success,
   \"failed\": $failed,
   \"rating\": $rating,
   \"duration\": \"$duration\"
 }
}"

# Write the JSON output to the output file
echo "$output_json" > "$output_file"

echo "Conversion completed. JSON output written to $output_file"

As for now the status is that either only the Name works, so the TESTS and SUMMARY section is not filled in to the JSON file, or (if modified with removing tests) the NAME and SUMMARY works but TESTS dont.

Example txt:

[ Samples ], 1..7 tests
---------
not ok  1  expecting cmd finishes successfully (bash way), 5ms 
not ok  2  expecting cmd finishes successfully (the same as above, bash way), 20ms 
ok  3  expecting cmd fails (same as above, bash way), 20ms 
ok  4  expecting cmd prints exact value (bash way), 10ms 
ok  5  expecting cmd prints exact value (the same as above, bash way), 10ms 
ok  6  expecting cmd prints some message (bash way), 15ms 
ok  7  expecting cmd prints some message (the same as above, bash way), 15ms 
--------- 
5 (of 7) tests passed, 2 tests failed, rated as 71.43%, spent 95ms

What am I doing wrong? Is there something I am doing fundamentally wrong with this?

Thanks for the advice in advance!

Remove the double-quotes from the pattern, they don't belong there. The final result refers to a test array that hasn't been assigned at all; it looks like the [[ "$line" =~ $pattern ]] section should add to that array, and not break. Also, expanding a bash array inside a string will not produce a valid JSON array. For this and many similar reasons, you shouldn't use bash to create JSON directly, you should use something like jq to JSONify the data (see this and this). — Gordon Davisson
– Gordon Davisson, Commented Sep 1, 2023 at 9:07

jqurious · Accepted Answer · 2023-09-01 12:51:28Z

jq as mentioned is the "go-to" tool for dealing with JSON.

One useful feature is the capture() function which will build an object from named capture groups.

name_re='^\[(?<testName>[^]]+)\]'

test_re='(?<status>^(?:not )?ok) +(?<status_code>[0-9]+)'
test_re+='+(?<name>.+), +(?<duration>[0-9]+)ms'

summary_re='^(?<success>[0-9]+) \(of [0-9]+\) tests passed, '
summary_re+='(?<failed>[0-9]+) tests failed, rated as (?<rating>[0-9.]+)%'
summary_re+=', spent (?<duration>[0-9]+)ms$'

jq --arg name_re    "$name_re"    \
   --arg test_re    "$test_re"    \
   --arg summary_re "$summary_re" \
   -n -R '

[inputs] as $lines |
($lines[0] | capture($name_re)) + {
  "tests": [$lines[] | capture($test_re)], 
  "summary": $lines[-1] | capture($summary_re)
}

' input.txt

{
  "testName": " Example ",
  "tests": [
    {
      "status": "not ok",
      "status_code": "1",
      "name": "  text1 (text1)",
      "duration": "5"
    },
    {
      "status": "ok",
      "status_code": "2",
      "name": "  text2 (text2, issues)",
      "duration": "2"
    },
    {
      "status": "ok",
      "status_code": "3",
      "name": "  text3 (text3, issues)",
      "duration": "15"
    }
  ],
  "summary": {
    "success": "2",
    "failed": "1",
    "rating": "66.66",
    "duration": "22"
  }
}

Kaz · Accepted Answer · 2023-09-01 20:22:22Z

Solution in TXR:

$ txr log2j.txr testlog
{
  "tests" : [
    {
      "name" : "text1",
      "status" : false,
      "duration" : "5"
    },
    {
      "name" : "text2",
      "status" : true,
      "duration" : "2"
    },
    {
      "name" : "text3",
      "status" : true,
      "duration" : "15"
    }
  ],
  "testName" : "Example",
  "summary" : {
    "success" : "2",
    "failed" : "1",
    "duration" : "22",
    "rating" : "66.66"
  }
}

If some of the numeric values should be numbers, which the question doesn't indicate, some adjustments can be made.

Code in log2j.txr:

[ @testname ], @nil tests
-----------------------------------------------------------------------------------
@(collect)
@  (cases)
not ok @num @name (@name), @{dur}ms
@    (bind status nil)
@  (or)
ok @num @name (@name, @nil), @{dur}ms
@    (bind status t)
@  (end)
@(last)
-----------------------------------------------------------------------------------
@(end)
@succ (of @total) tests passed, @fail tests failed, rated as @percent%, spent @{totaldur}ms
@(do (let ((*print-json-format* :standard)
           (tests (vec-list (mapcar (ret #J^{ "name" : ~@1,
                                              "status" : ~@2,
                                              "duration" : ~@3 })
                                    name status dur))))

       (put-jsonl
         #J^{ "testName" : ~testname,
              "tests" : ~tests,
              "summary" : { "success" : ~succ,
                            "failed" : ~fail,
                            "rating" : ~percent,
                            "duration" : ~totaldur }})))

glenn jackman · Accepted Answer · 2023-09-03 17:47:30Z

0

2 observations:

you're not actually building the tests array that you use to construct the JSON output
use jq to build JSON

#!/usr/bin/env bash

{
    IFS= read -r header
    if [[ $header =~ "["([^]]+)"]," ]]; then
        testName=${BASH_REMATCH[1]}
    fi
    read
    tests=()
    passed=0 failed=0
    while IFS= read -r line; do
        [[ $line =~ ^-+[[:space:]]*$ ]] && break
        if [[ $line =~ ("not ok"|"ok")"  "[0-9]+"  "(.*)", "(.*s)[[:space:]]*$ ]]; then
            if [[ ${BASH_REMATCH[1]} == "ok" ]]; then
                status=true
                ((++passed))
            else
                status=false
                ((++failed))
            fi
            test=$(jq -nc --arg name "${BASH_REMATCH[2]}" \
                          --argjson status "$status" \
                          --arg duration "${BASH_REMATCH[3]}" \
                      '$ARGS.named')
            tests+=("$test")
        fi
    done
    IFS= read -r footer
    if [[ $footer =~ "rated as "(.*)", spent "(.*) ]]; then
        summary=$(jq -nc --argjson success "$passed" \
                         --argjson failed "$failed" \
                         --arg rating "${BASH_REMATCH[1]}" \
                         --arg duration "${BASH_REMATCH[2]}" \
                     '$ARGS.named')
    fi

    IFS=","
    jq -n --arg testName "$testName" \
          --argjson tests "[${tests[*]}]" \
          --argjson summary "$summary" \
       '$ARGS.named'

} < test_results.txt

edited Sep 3, 2023 at 17:47

answered Sep 1, 2023 at 12:28

glenn jackman

249k42 gold badges233 silver badges362 bronze badges

5 Comments

Nysa-522 Over a year ago

I have tried this, thank you for the detailed answer. However, I am getting this error message now, and I am not quite familiar qith jq yet. Can you advise me what could be the issue here? C:\Program Files\Git\usr\bin\jq.exe: invalid JSON text passed to --argjson Use C:\Program Files\Git\usr\bin\jq.exe --help for help with command-line options, or see the jq manpage, or online docs at stedolan.github.io/jq Conversion completed. JSON output written to output.json

glenn jackman Over a year ago

Impossible to diagnose without seeing the input txt.

Nysa-522 Over a year ago

[ Samples ], 1..7 tests --------- not ok  1  expecting cmd finishes successfully (bash way), 5ms not ok  2  expecting cmd finishes successfully (the same as above, bash way), 20ms ok  3  expecting cmd fails (same as above, bash way), 20ms ok  4  expecting cmd prints exact value (bash way), 10ms ok  5  expecting cmd prints exact value (the same as above, bash way), 10ms ok  6  expecting cmd prints some message (bash way), 15ms ok  7  expecting cmd prints some message (the same as above, bash way), 15ms --------- 5 (of 7) tests passed, 2 tests failed, rated as 71.43%, spent 95ms

Nysa-522 Over a year ago

Also updated the question with this example txt

glenn jackman Over a year ago

The new sample text has trailing whitespace. The 2 regexes at the start of the while loop needed [[:space:]]* before the $ anchor.

Collectives™ on Stack Overflow

Bash script for TXT file to JSON conversion parsing issue

3 Answers 3

Comments

Comments

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related