Tip: Remove Duplicate Lines in File Using Perl

Rai    Linux     code, duplicate, file, i/o, Linux, perl, process, syntax, tip

Here is little nifty code I found to snip or remove duplicate lines in a file using Perl. An example of a file that has duplicate lines looks like this:

/home/index.php
/home/links.php
/home/index.php

The result we want to achieve is this:

/home/index.php
/home/links.php

And here goes the code:

$ftmp = 'file.txt';
my %match = ();
{
local @ARGV = ($ftmp);
local $^I = '.tmp';
while(<>){
$match{$_}++;
next if $match{$_} > 1;
print;
}
}

The original file, file.txt, has its duplicate lines removed.