CRLF with BlockBuffering not always correct

The following test case produces different results with System.IO and Data.Text.IO:

import Control.Monad (when)
import Data.Text qualified as T
import Data.Text.IO qualified as T
import Data.ByteString.Char8 qualified as BS
import System.Directory (removeFile)
import System.IO

main :: IO ()
main = do
  let
    test putFunc value = do
      let tempFile = "test-data.txt"
      withFile tempFile WriteMode $ \h -> do
        hSetEncoding h utf8
        hSetNewlineMode h (NewlineMode CRLF CRLF)
        hSetBuffering h (BlockBuffering $ Just 8192)
        putFunc h value
      BS.readFile tempFile

    testStr = replicate 2047 '_'

  resultStr <- test hPutStrLn testStr
  resultTxt <- test T.hPutStrLn (T.pack testStr)

  when (resultStr /= resultTxt) $ do
    putStrLn "String:"
    print resultStr
    putStrLn "Text:"
    print resultTxt

The results are:

String:
"_______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________\r\n"
Text:
"_______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________\n"

Text is using "\n" when it should be using "\r\n"

This doesn't happen with a different text size (eg 2046, 2048) and it doesn't happen with text-2.1.1

Uh oh. There is indeed a case where we write a newline as LF when it should be CRLF (below, when n+1 == len):

text/src/Data/Text/Internal/IO.hs

Lines 265 to 268 in eeeb893

    
           | isCRLF && x == '\n' && n + 1 < len -> do 
        
               n1 <- writeCharBuf' raw len n '\r' 
        
               writeCharBuf' raw len n1 '\n' >>= inner s' 
        
           | n < len -> writeCharBuf' raw len n x >>= inner s'

Now I am wondering why our tests did not catch this.

text/tests/Tests/QuickCheckUtils.hs

Lines 268 to 278 in eeeb893

    
           propTest enc genBufferMode nl d = forAll genBufferMode $ \mode -> ioProperty $ withTempFile $ \_ h -> do 
        
             let ts = modData d 
        
                 t = unline . map (filt (not . (`elem` "\r\n"))) $ ts 
        
             IO.hSetEncoding h enc 
        
             IO.hSetNewlineMode h nl 
        
             IO.hSetBuffering h mode 
        
             () <- writer h t 
        
             IO.hSeek h IO.AbsoluteSeek 0 
        
             r <- reader h 
        
             let isEq = r == t 
        
             seq isEq $ pure $ counterexample (show r ++ bool " /= " " == " isEq ++ show t) isEq

Now I am wondering why our tests did not catch this.

Presumably it's because the reader function is also doing CRLF conversion, and "\n" and "\r\n" both end up as "\n" after reading. It's a classic Catch-22 situation. Or, to put it another way, "Who watches the watchers?"

The reader is T.hGetContents or TL.hGetContents, BTW.

Ideally, the tests would use something more low-level to read back the contents, as I did with my example code (which uses ByteString).

Because this is a boot library, it can't be worked around using Cabal constraints on text and therefore affects all projects using ghc 10.2 and 12.x. Is there any chance of getting a fix for this into the upcoming ghc 10.3?

Although it seems like it would occur rarely in practice, it does break tests for the say package (fpco/say#5) which then has a knock-on effect on projects using say.

I'm going to work on this

Because this is a boot library, it can't be worked around using Cabal constraints on text

It's not correct. You can bump text. constraints: text >=2.1.3 would work just fine.

Add regression test for #648

Navigation Menu

Repository navigation

CRLF with BlockBuffering not always correct #648

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	\| isCRLF && x == '\n' && n + 1 < len -> do
	n1 <- writeCharBuf' raw len n '\r'
	writeCharBuf' raw len n1 '\n' >>= inner s'
	\| n < len -> writeCharBuf' raw len n x >>= inner s'

	propTest enc genBufferMode nl d = forAll genBufferMode $ \mode -> ioProperty $ withTempFile $ \_ h -> do
	let ts = modData d
	t = unline . map (filt (not . (`elem` "\r\n"))) $ ts
	IO.hSetEncoding h enc
	IO.hSetNewlineMode h nl
	IO.hSetBuffering h mode
	() <- writer h t
	IO.hSeek h IO.AbsoluteSeek 0
	r <- reader h
	let isEq = r == t
	seq isEq $ pure $ counterexample (show r ++ bool " /= " " == " isEq ++ show t) isEq

CRLF with BlockBuffering not always correct #648

Description

Activity

Lysxia commented on Jul 29, 2025

neilmayhew commented on Jul 29, 2025

neilmayhew commented on Jul 31, 2025

Lysxia commented on Jul 31, 2025

phadej commented on Jul 31, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions