How to find and remove ^M Character from Unix/Linux text file

Control M or Ctrl-M or ^M character whatever you call them creates the problem when present in Unix or Linux text file. The presence of these characters may lead to unexpected results or interrupt the working of shell script, configuration files, etc.

Unix /Linux is the most widely used operating system for server and database hosting. Developers create code, text script on Windows, and move those to Unix environment where script execution fails with Command not found and Program exited with status 127 error.

What is ^M character in Unix/Linux?

Ctrl M or ^M is the carriage return character. Those come in the file because of different line termination characters used by Unix and Windows/DOS operating systems. Unix uses only line feed (LF) while windows use both carriage return (CR) and line feed (LF) as termination characters. This carriage return (CR) character becomes Ctrl M in Unix/Linux. 

See the below image for details.

Windows use CR LF while Unix use LF line termination character

Want to know more about line termination character, just head on to Wikipedia.

How to find ^M character

There are multiple ways to find out the CTRL-M character in a text file. You can use text editors like Notepad,  Notepad++, Textpad, VI, or even cat/grep Unix commands should also do the job.

Below we have covered the methods,

1. Using Notepad++ text editor

Open the file in Notepad++. Click on View -> Show Symbol -> Show All Characters or click this “¶” menu on the toolbar to displays all the characters including CR and LF. As mentioned above Unix only uses LF while Windows uses CRLF as a line termination chacter. While moving the Windows file to Unix, you need to remove this CR character.

line termination character windows and unix

2. cat -v command

cat command with -v option displays non-printing characters including ^M on the standard output as shown below.

$ cat -v texthost.prog
echo "hi how are you"^M
ls^M
^M

3. grep command

grep command allows you to search a string in a file. So run the grep command on the file as shown below to find out and display all the lines where the ^M character is present.

Note:- 

To type “^M” – click Ctrl+V and Ctrl+M i.e you can hold the CTRL key and press V and M sequentially. V should be first.

$ grep ^M texthost.prog
echo "hi how are you"
ls

4. vi text editor

-b option available in the VI Editor displays this special character. Simply run vi -b <file name> to display ^M on the screen as shown below.

$ vi -b texthost.prog
echo "hi how are you"^M
ls^M
^M

How to Avoid ^M character

You can use text editors like Notepad++ and TextPad to create files specific to the respective operating system. FTP software also helps to do explicit character conversion. Let’s discuss all these methods in detail and see how you can avoid Ctrl ^M character from a file.

1. Notepad++ or Textpad editor

Editors like Notepad++ or TextPad generally have a built-in option to save the file in Unix or Windows format. Depending on the target operating system you should save the file to avoid the character issue. For Notepad++, navigate to Edit->EOL Conversion and chose the respective file format. Here you need to choose UNIX/OSX format.

notepad eol conversion

2. FTP file in ASCII mode

sFTP or FTP software like WinSCP, FileZilla provides options to transfer files using either Binary or ASCII mode. Always, transfer text files in ASCII mode from Windows to Unit or vice-version to avoid this ^M character issue for explicit character conversion.

The ASCII Mode does explicit character conversion. Windows format file is automatically converted to Unix format file and vice versa. Binary Mode moves the file as is with all properties.

How to Remove ^M character

The best is to avoid it using the method mentioned above, but if you still end up with it, then use the below method to remove the ^M character.

1. Explicit character conversion using dos2unix and unix2dos

Unix operating system like Solaris provides built-in command/utility dos2unix and unix2dos to convert file from Unix to DOS and vice versa.

dos2unix converts the file from DOS to UNIX format and unix2dos converts the file from UNIX to DOS format. This is the way you can use it.

dos2unix dos_format.txt unix_format.txt
or
unix2dos unix_format.txt dos_format.txt

2. vi editor

Use the VI editor in binary mode to search and replace the ^M character with null as shown in the below code snippet.

$ vi -b texthost.prog
echo "hi how are you"^M
ls^M
^M

Press [ESC] Key and type below including : (do not copy type it)

:%s/^M//g

:wq

$ cat -v texthost.prog
echo "hi how are you"
ls

3. sed command

You can also use the stream editor (sed) to find and replace ^M with space as shown in the below example.

$  sed -e "s/^M//g" texthost.prog > texthost.prog2
$ cat -v texthost.prog2
echo "hi how are you"
ls

mv texthost.prog2 texthost.prog

4. Perl

You can use the Perl search and replace to remove ^M and replace it with blank space. Check the below example.

$perl -pe 's/^M//g' texthost.prog > dummy

$ cat -v dummy
echo "hi how are you"
ls

mv dummy texthost.prog

If you do not want to use the mv command, run below

perl -i -pe 's/^M//g' texthost.prog

You can also create a backup of the original file using the -i.bak option.

$perl -i.bak -pe 's/^M//g' texthost.prog

$ cat -v texthost.prog
echo "hi how are you"
ls

$ cat -v texthost.prog.bak
echo "hi how are you"^M
ls^M
^M

5. tr command

Tr command with ‘-d’ option deletes the pattern instead of translating. Here, we deleting the ‘\r’ (non-printable) character.

 tr -d '\r' < texthost.prog > texthost.prog2

Summary

I have covered most of the methods to find and remove m character. Do let me know if I missed any. You could also follow the discussion on StackExchange or StackOverflow for further details.

Share This: