3

I receive files with lines representing datasets. Each line contains characters representing several fields of data without breaks and I need to break the lines to extract each data field eg

ABCD075BCD156300544E0001000900125349544520494

->

ABCD 075BCD 15630 0544 E00010009 00 12 5349544520494

I know the number of characters in each field, so I can do this in awk using FIELDWIDTHS. This works fine when all the datasets are the same, but they aren't. To make it more complicated, I can only tell which type of dataset it is by reading one of the fields.

So I think I need to apply a set of FIELDWIDTHS to read the data type. Then apply a different set of FIELDWIDTHS and reread the same line using getline to extract the data. But this doesn't work because the changed fieldwidths are never applied.

BEGIN {
  FIELDWIDTHS = "30 2";                  # set FIELDWIDTHS to read data type
}

{
  print $2;                              # print data type
  FIELDWIDTHS = "5 5 5 5 5 5 5 5 5 5 5"; # change fieldwidths to read data
  getline NF;                            # reread current line to use new fieldwidths
  print $2;                              # print data field
  FIELDWIDTHS = "30 2";                  # change fieldwidths to read next line
}

END {
}

Any advice would be appreciated.

Improve this question
1
  • 1
    It's also not clear if you'll have common fields to operate on that just happen to be at different character positions in each line or if each line should be treated completely differently based on it's type or even if your data really should be treated as fixed width fields or if a regexp match would bee a better approach. A few more lines of sample input and expected output would really help clarify the best approach. Commented Apr 23, 2021 at 14:20

4 Answers 4

Reset to default
4

with any awk:

awk 'NR%2   { fieldwidths="4 6 5 4 9 2 2 13" } # update fieldwidths on odd line numbers
    !(NR%2) { fieldwidths="4 5 4 2 3 9 7 11" } # update fieldwidths on even line numbers
    # condition { fieldwidths="# # #  ..." }   # whatever other condition you want...

{ fields=split(fieldwidths, fldwd); startPos=1;
  for(i=1; i<=fields; i++) {
      printf "%s", (i==1?"": OFS) substr($0, startPos, fldwd[i])
      startPos+=fldwd[i]
  }
  print ""
}' infile
Improve this answer
4

I'd think something like this (using GNU awk for FIELDWIDTHS) is what you want:

BEGIN {
    type2fw[10] = "7 3 6 8 9"
    type2fw[12] = "5 5 5 5 5 5 5 5 5 5 5"
    type2fw[53] = "1 1 1 17 29 31"
    ....
}
{
    FIELDWIDTHS = type2fw[substr($0,31,2)]
    $0 = $0
    do whatever you like with the fields
}

but that's kinda inefficient as it's doing field splitting twice (once when the record is read and then a 2nd time when you do $0=$0). You could make it more efficient by only re-splitting when the type changes:

BEGIN {
    type2fw[10] = "7 3 6 8 9"
    type2fw[12] = "5 5 5 5 5 5 5 5 5 5 5"
    type2fw[53] = "1 1 1 17 29 31"
    ....
}
{ type = substr($0,31,2) }
type != prev {
    FIELDWIDTHS = type2fw[type]
    $0 = $0
    prev = type
}
{
    do whatever you like with the fields
}

and you could sort the input first by that 31st/32nd character type field (e.g. sort -k1.31,1.32 file | awk '...') so you only have to change FIELDWIDTHS once per type.

Without seeing concise, testable multi-line/type sample input and expected output I can't be more concrete than that and that might even be the wrong approach and something using match($0,/(foo)(bar)(etc)/,a) or some other approach would be better.

Improve this answer
3
  • Inefficiencies apart, rather than hard coding type2fw array in the BEGIN block it's perhaps more agile to pass a string var from shell to Awk as awk -v b=$(for i in "${type2fw[@]}"; do printf "%s","$i";done | cut -c 2-) 'BEGIN{split(b,type2fw,",")} {...}' .... just a suggestion to automate the treatment by importing the fieldwidth info from elsewhere... Commented Apr 24, 2021 at 12:32
  • 1
    @Cbhihe it's not clear why hard-coding the array in shell and passing it to awk would be better than just hard-coding the array in awk. You'd need more code than that to do what you suggest too as you'd need to split b by newlines (not commas) to get the fieldwidths strings for every type of input line, but then also pass the type string (e.g. "12") associated with the fieldwidths string as the mapping doesn't appear to be numerically incremental indices (e.g. it seems to be 2-digit strings). You could define an associative array in the shell as a step but that's non-portable as well as slow Commented Apr 24, 2021 at 13:20
  • 1
    +1 I was just using the comma separation as an example, but I agree about the associative array being non-portable would be an issue. I really don't like the idea of hard coding something with external dependencies inside an awk script. Everything depends on use-case anyway and we're short on such info. Commented Apr 24, 2021 at 15:00
3

Using gnu awk, you can have the current line reparsed by assigning $0 = $0. For example,

echo '1 abcdefghij
2   abcdefghij' |
awk '
/^1/{ FIELDWIDTHS = "1 1 5 5"; $0 = $0; print $3; next }
/^2/{ FIELDWIDTHS = "1 3 3 3"; $0 = $0; print $3; next }
'

Or you can consider a more Unix-like solution, piping the data through one awk that handles one field format, marking the treated lines for example with a prefixed # character, then piping the result into a second awk. For example,

awk -v FIELDWIDTHS="1 1 5 5" '
/^1/{ print "#" $3; next }
    { print }
' |
awk -v FIELDWIDTHS="1 3 3 3" '
/^2/{ print $3; next }
/^#/{ print substr($0,2); next }
    { print }
'
Improve this answer
1
  • 2
    I wouldn't consider reading the input N times for N different types of line instead of just reading it once to handle all line types to be a more Unix-like solution. Commented Apr 23, 2021 at 14:12
0

Another approach using GNU sed with its /e modifier of the s/// command is presented here.

General idea is to store the space separated list of field widths in files whose names are the 31-32 characters of the current input record. The unpk function is supplied the filename holding the field widths pertinent to the current record. It then generates sed code that slices the current record based on these widths.

#--- edit this function to add the fieldwidths corresponding to
#--- the 2 characters in the 31st/32nd
#--- positions of the input record
_init_() {
  [ -s "$1" ] && return
  case $1 in
    */12) echo '4 6 5 4 9 2 2 13' ;;
    */96) echo '5 5 5 6 7 2 2 13' ;;
  esac > "$1"
}

_unpk_() {
  _init_ "$1"
< "$1" tr -s ' \t' '[\n*]' |
sed -Ee '
  1i\
$!d;H;z;x
  s|.*|s/\\n.{&}/\&\\n/|
  s|$|;s/\\n/ /|
  $a\
s/^.|.$//g
'
}

export -f _init_ _unpk_
tmpdir=$(mktemp -d)

sed -Ee "w $tmpdir/h
  s:.{30}(..).*:_unpk_ '$tmpdir/\\1' | sed -Ef - '$tmpdir/h':e
" file
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.