Problems reading from socket

Sockets are a red herring here: this is an issue of READ vs. READ-PRESERVING-WHITESPACE:

(with-input-from-string (s (format nil "(this)~%"))
  (values (read s)
          (read-char s nil :eof)))

(with-input-from-string (s (format nil "(this)~%"))
  (values (read-preserving-whitespace s)
          (read-char s nil :eof)))

READ is allowed (possibly even required, not sure offhand) to throw away syntactically meaningless whitespace following the read object -- in this case the #\Newline character.

Using READ-PRESERVING-WHITESPACE instead your code should work on both Clisp and SBCL.

Also, I would advice against line-buffering if you wish portable parts of your application to be ... portable: it is better to FINISH-OUTPUT explicitly when you wish stuff to be actually sent to the other end -- it may be sent sooner, of course, FINISH-OUTPUT just flushes any pending output. (Note: FINISH-OUTPUT, not FORCE-OUTPUT.)

If portability is not a concern and you're always careful to terminate your messages with a newline, :LINE buffering should of course work just fine.

Thanks very much for this. Indeed, I had discovered read-preserving-whitespace as a _work-around_ for my immediate needs. However, that cannot be the "right" response and I consider it only a work-around because, in the case of the problem in question, there is no whitespace to be preserved. The problem is that SBCL's read hangs when no such following whitespace exists (over a socket).

Interestingly,
(with-input-from-string (s "(this)") (read s))

does work in SBCL as one would hope without hanging. So there does seem to be some issue with sockets vs. with arbitrary input streams, as SBCL hangs on the same read over a socket.

ps: I appreciate the comments on portability and have learned from them. However, please keep in mind that, in this case, I have no control over how the server sends data over the socket, and that the question really has to do with reading a non-whitespace-terminated expression over a socket.

You say:

> However, that cannot be the "right" response and I consider it only a work-around because, in
> the case of the problem in question, there is no whitespace to be preserved. The problem is
> that SBCL's read hangs when no such following whitespace exists (over a socket).

Which I don't understand at all.

When the first call to READ occurs, there is #\( #\t #\h #\i #\s #\) #\newline available on the socket. READ consumes __all__ of this, including the newline. Then there is nothing more to read, of READ-CHAR hangs instead of getting the newline.

I repeat that the socket stream is a non-issue here. You will have identical results on SBCL if you do

(with-open-file (f "/tmp/test.sexp"
                   :direction :output
                   :if-exists :supersede)
  (format f "(this)~%"))

(with-open-file (f "/tmp/test.sexp")
  (values (read f)
          (read-char f nil :eof))) ; => (THIS), :EOF

where READ has consumed the newline before READ-CHAR could get at it.

The only difference is that in case of a pipe or a socket instead of getting an EOF from READ-CHAR the call will block instead, waiting for more input. This would happen just the same way if you had opened a named pipe called instead.

I think you missed my followup example withOUT the #\Newline. When I create a file without newline at end (i.e., consisting entirely of the six characters: #\( #\t #\h #\i #\s #\) ), read from the file works just fine. However, when sent over a socket, the simple read hangs.

Note, with the socket receiving the non-newline-terminated list, I get the following:

* (loop while (listen mysock) collect (read-char mysock))

(#\( #\t #\h #\i #\s #\))

but,

* (read myclient)

on the same sent message hangs.

At this point, I'm convinced this is a bug but I'd prefer the two of us come to a common understanding before submitting. Common Lisp (read) allows the consumption of whitespace following numbers and symbols, but lists are self terminated. Like I said before, I can use read-preserving-whitespace as a work-around, but there is a difference here between the way read reads from a file and from a socket.

Thanks for thinking about this with me!

Oh, sorry! I missed that.

You're right. There's a bug in READ. It uses READ-CHAR without LISTEN to check for and discard following whitespace, and therefore can hang unless EOF has been reached.

Thanks so much!

Thanks Nikodemus Siivola, that solved my question.