12

This works

head -c 10 /dev/zero | sed 's/\x0/x/g'

and generates xxxxxxxxxx as expected.

Then I tried this:

sed 's/\x0/x/g' /dev/zero | head -c 10

which seems to just hang.

Then I thought, oh, sure, there's no newline in here and sed can't even finish take a line in the pattern space, so it hangs. Is that the reason why the command above hangs?

But wait, there's a -u option:

  -u, --unbuffered

         load minimal amounts of data from the input files and flush the output buffers more often

which made me think it would be the solution, but it also seems to block:

sed -u 's/\x0/x/g' /dev/zero | head -c 10

Does this mean that it is just not possible to operate with sed on an infinite stream?

Improve this question
7
  • /dev/zero is like /dev/null: there's no output from this. null, zero - data sink What are tou trying to achieve with /dev/zero at the beginning? Commented Feb 2 at 12:30
  • 4
    I have read that they can be used as data sinks, but also that /dev/zero can be used as an infinite source (which I think the first snippet shows), wheras /dev/null can't. What am I trying to achive? Nothing but understanding. The question popped to my mind when I tried using cat /dev/zero as something that doesn't return. Commented Feb 2 at 12:46
  • 25
    @GillesQuénot, no, while writing to either is the same (whatever written just gets succesfully ignored), reading is the difference between the two. Reading from /dev/zero produces an endless stream of zero/NUL bytes \x00. Reading from /dev/null produces just a zero-length read, an "EOF". Commented Feb 2 at 13:49
  • I remember now cases where it's useful: dd if=/dev/zero of=/dev/sdX. But what I don't get is why this don't work? : cat /dev/zero | head Commented Feb 2 at 19:13
  • 7
    @GillesQuénot: head without any arguments behaves like head -n 10, i.e., prints the first 10 lines. /dev/zero produces an infinite string of NUL octets, and never produces a line terminator. In other words, it produces a single infinite line of NUL bytes. Since you asked head to print the first 10 lines, it will need to wait for the tenth line terminator … which never comes. See pubs.opengroup.org/onlinepubs/9799919799 for details. Commented Feb 2 at 20:16

3 Answers 3

Reset to default
16

As it happens, by writing the question, I think I've maybe come up with the answer. I'll delete it if somebody else actually proves I'm wrong and gives the correct answer.

I suppose the problem is that sed is line-oriented, whether the line separators are \n or \x0 (the latter is the case when passing the -z option, I understand), so an infinite stream consisting of repetitions of one character is just not a valid input to sed, because it means one of two things:

  • if that infinitely repeated character is \n (resp. \x0), and sed was launched without (resp. with) the -z option, then the input stream is an infinite number of empty lines,
  • otherwise, the input stream is a single, infinitely long line composed of an infinite number of repetitions of that character.
Improve this answer
7
  • 4
    FWIW, standard sed cannot handle NUL-bytes nor \xhh escape sequences nor -u nor -z Commented Feb 2 at 12:52
  • 1
    And you're right, regex engine isn't meant to work with infinite strings. Commented Feb 2 at 12:58
  • 4
    @Enlico, I would expect you're exactly right. If you try reading an endless stream of characters that are not the line terminator, you might see sed allocating more and more memory until something hits the fan. Some implementations might just abort or misbehave when encountering an overly-long line, and there's allowances for that in POSIX, too. (Look for LINE_MAX in pubs.opengroup.org/onlinepubs/9799919799/utilities/…) Commented Feb 2 at 16:52
  • 7
    Since nobody's mentioned it, tr is byte-oriented and can work with /dev/zero. For example, tr '\000' 'x' < /dev/zero | hexdump -C produces a stream of 0x78 ASCII x bytes. Or tr '\000' '\n' is like yes '', a stream of newlines. Commented Feb 2 at 22:18
  • 2
    @PeterCordes Glad you mentioned tr, but it's worth noting that, perplexingly, not all versions of tr can work with nul bytes, either. Commented Feb 3 at 21:54
4

At least some versions of sed, such as GNU sed, can handle NUL characters.  This can be shown with a simple example:

/bin/echo -ne 'hi\x00bye\nend' | sed 's:\x00:-:'

This should produce the following output:

hi-bye
end

Where the final line is missing the newline and so the shell prompt will appear immediately afterwards. This shows both how sed handles NUL characters in the file stream and how it handles lines without a newline terminator. The key is, the stream hits an end-of-file condition after reading the last line.  sed is very much a line-based editor as it is based on ed, the original line-based editor for UNIX.

There is an alternative command that should fit the needs for the example given in the original question. The tr command just does single character (technically, single-byte) replacements or deletions. The following command should work for the original author's example:

tr '\0' x < /dev/zero
Improve this answer
2

I believe this may be a "X/YProblem" so I'll answer what I see as the need instead of the actual "how-to" question :

If you need a simple way to generate a N-length sequence of the same character, there are several easy ways:

# several ways to generate a 12 char long string with "x" characters:
x12=$( yes "" | head -n 12 | tr '\n' 'x' )
  # maximum size if only the limit you can put into a variable.
  # If you need too many characters, write to a file instead.
  # This version is easy to grok, and quite versatile:
  # for exemple if instead you want as a string of 12 consecutive "foo" :
  # foo12=$(  yes "foo" | head -n 12 | tr -d '\n'  )
x12=$( printf '%12s' " " | tr ' ' 'x' )
  # even better : printf is builtin, only 1 tr spawned. 
  # The maximum size depends on OS, shell version, ram, etc.
  # on some old unixes you would need '%12s\n' instead, 
  # and maybe `` instead of $()
# and probably tons of other ways
Improve this answer
2
  • Weirdly I couldn't post this for several minutes... I have no idea why. Commented Feb 25 at 16:42
  • 1
    There were (are?) some issues with posting answers today. I came across it too. Commented Feb 25 at 17:10

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.