Web lists-archives.com

Re: gawk Regression: CR characters are not stripped on Windows

[urrgh - Cygwin's list policy in supplying reply-to makes it difficult to reply-to-all]

On 02/27/2018 01:22 AM, Orgad Shaneh wrote:

Cross-posting per Eli Zaretskii's request.

CR characters used to be automatically stripped on Windows (MSYS2 and
Cygwin environments). This is broken in 4.2.0.

You should not think of Cygwin as a Windows environment, but as a Linux-alike environment. gawk on Linux does not automatically strip CRs, therefore gawk on Cygwin should not automatically strip CRs.

What MSYS2 does is different, and that environment is entitled to use patches to make interoperability with native windows program nicer, at the expense of being less like Linux.

Furthermore, the change in Cygwin predates the gawk 4.2.0 release, and was intentionally made in a coordinated release in Feb 2017 alongside sed and grep:


following on from discussions about bash after ShellShock:


Changing gawk back to automatically strip CRs on Cygwin would be a regression.

As Eli said, this change was deliberate. But this has several drawbacks.

1. The gawk info page states that:

Under MS-Windows, 'gawk' (and many other text programs) silently
translates end-of-line '\r\n' to '\n' on input and '\n' to '\r\n' on

and on Feb 8 the following section was added:

Recent versions of Cygwin open all files in binary mode.  This means
that you should use 'RS = "\r?\n"' in order to be able to handle
standard MS-Windows text files with carriage-return plus line-feed line

Or mount your Windows text files under a text mount in Cygwin (so that such files already have \r stripped), or add steps to your pipelines to strip CR before handing the data to gawk.

This breaks compatibility between different gawk versions. What were
the reasons for this change in cygwin, and why was it pushed upstream?

See the discussion in Feb 2017 for rationale, but the executive summary is that Cygwin attempts to emulate Linux, silent corruption of binary files was deemed worse than manually having to explicitly strip CR when dealing with Windows text output.

2. Git and other tools automatically convert text files to CRLF on

Not Cygwin git. The problems you are encountering are more likely to happen when you mix and match tools from disparate environments, rather than when you use all tools from the same source.

This means that any awk script that runs on both platforms
must use RS = "\r?\n".

or strip the CR in any other means. But the same is true of any script that must run on both Windows and Linux.

Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple