Skip to main content
Unix & Linux

Return to Question

added 48 characters in body
Source Link
Coder
Coder
  • 179
  • 1
  • 9

I have a big text file (roughly 2GB). I have a csv file that has the following fields:

rowID,pattern,other
1,abc_1z1,90
2,abc_1z2,90
3,abc_1z10,80
4,abc_3p1,77
...

My interest is: replace the content of the big file as follows. Whenever a string in the big file matches a "pattern" in my CSV (second field), it will replace that string by the corresponding "rowID" (first field).

This is what I have tried using sed, which is extremely slow (also due to in-place replacement of the file). But, is there any faster solution?

while read f1 f2 f3; 
do 
    sed -i "s/$f2/$f1/g" bigfile; 
done < map.csv

Note that map.csv contains over 500000 rows.

I have a big text file (roughly 2GB). I have a csv file that has the following fields:

rowID,pattern,other
1,abc_1z1,90
2,abc_1z2,90
3,abc_1z10,80
4,abc_3p1,77
...

My interest is: replace the content of the big file as follows. Whenever a string in the big file matches a "pattern" in my CSV (second field), it will replace that string by the corresponding "rowID" (first field).

This is what I have tried using sed, which is extremely slow (also due to in-place replacement of the file). But, is there any faster solution?

while read f1 f2 f3; 
do 
    sed -i "s/$f2/$f1/g" bigfile; 
done < map.csv

I have a big text file (roughly 2GB). I have a csv file that has the following fields:

rowID,pattern,other
1,abc_1z1,90
2,abc_1z2,90
3,abc_1z10,80
4,abc_3p1,77
...

My interest is: replace the content of the big file as follows. Whenever a string in the big file matches a "pattern" in my CSV (second field), it will replace that string by the corresponding "rowID" (first field).

This is what I have tried using sed, which is extremely slow (also due to in-place replacement of the file). But, is there any faster solution?

while read f1 f2 f3; 
do 
    sed -i "s/$f2/$f1/g" bigfile; 
done < map.csv

Note that map.csv contains over 500000 rows.

edited title
Link
Coder
Coder
  • 179
  • 1
  • 9

Replace strings in a big text file based on fields from a CSV file

Source Link
Coder
Coder
  • 179
  • 1
  • 9

Replace strings in a big file based on fields from a CSV file

I have a big text file (roughly 2GB). I have a csv file that has the following fields:

rowID,pattern,other
1,abc_1z1,90
2,abc_1z2,90
3,abc_1z10,80
4,abc_3p1,77
...

My interest is: replace the content of the big file as follows. Whenever a string in the big file matches a "pattern" in my CSV (second field), it will replace that string by the corresponding "rowID" (first field).

This is what I have tried using sed, which is extremely slow (also due to in-place replacement of the file). But, is there any faster solution?

while read f1 f2 f3; 
do 
    sed -i "s/$f2/$f1/g" bigfile; 
done < map.csv