Dec

Tip: Remove Duplicate Lines in File Using Perl

Rai

Linux

code, duplicate, file, i/o, Linux, perl, process, syntax, tip

Here is little nifty code I found to snip or remove duplicate lines in a file using Perl. An example of a file that has duplicate lines looks like this:

/home/index.php
/home/links.php
/home/index.php

The result we want to achieve is this:

/home/index.php
/home/links.php

And here goes the code:

$ftmp = 'file.txt'; my %match = (); { local @ARGV = ($ftmp); local $^I = '.tmp'; while(<>){ $match{$_}++; next if $match{$_} > 1; print; } }

The original file, file.txt, has its duplicate lines removed.

3 Comments to “Tip: Remove Duplicate Lines in File Using Perl”

R. Saravanan July 3, 2010 at 12:17 pm
hai,

good morning, anybody is there
R. Saravanan July 3, 2010 at 12:21 pm
hai

good morning, anyody is there
luchhe November 29, 2010 at 3:23 am
@Saravanan: go ahead dude, we’re all ears !