Python - Replace based on another file

$ cat main.txt
P|34|90
T|12
R|0|1291870414|ip1|890
R|1|1291870415|ip5|690
R|2|1291870415|ip1|899
R|3|1291870412|ip2|896
R|4|1291870418|ip3|999
R|5|1291870419|ip5|191

$ cat lookup.txt
ip7|172.17.4.8
ip1|172.17.4.3
ip5|172.17.4.9
ip4|172.17.4.2
ip3|172.17.4.1
ip2|172.17.4.6
ip6|172.17.4.7

Required Output:
Replace the 4th field (pipe delimited) of the 'R' lines of 'main.txt' with the corresponding lookup value from 'lookup.txt' i.e. 'ip1' to be replaced with '172.17.4.3', 'ip2' with '172.17.4.6' etc.

P|34|90
T|12
R|0|1291870414|172.17.4.3|890
R|1|1291870415|172.17.4.9|690
R|2|1291870415|172.17.4.3|899
R|3|1291870412|172.17.4.6|896
R|4|1291870418|172.17.4.1|999
R|5|1291870419|172.17.4.9|191

The python script:

import sys
d={}
for line in open("lookup.txt"):
    line=line.strip().split("|")
    d[line[0]]=line[-1]
for line in open(sys.argv[1]):
    if line.startswith('P'):
        print line,
    if line.startswith('T'):
        print line,
    if line.startswith('R'):
        line=line.strip().split("|")
        print '|'.join(line[0:3])+'|'+d[line[3]]+'|'+'|'.join(line[4:])

Executing it:

$ python replace-from-file.py main.txt
P|34|90
T|12
R|0|1291870414|172.17.4.3|890
R|1|1291870415|172.17.4.9|690
R|2|1291870415|172.17.4.3|899
R|3|1291870412|172.17.4.6|896
R|4|1291870418|172.17.4.1|999
R|5|1291870419|172.17.4.9|191

Related Posts:
- Lookup file operation using Python
- Lookup file in python using Dictionary
- Simple python file lookup function
- Find text string in file in Python

Python - Remove duplicate lines from file

Objective : Remove duplicate lines from a file (print first occurrence) which appeared exactly twice.

Input file:

$ cat file.txt
begin
ip 172.17.4.53
line 172.17.4.52
pl 172.17.4.51
pl 172.17.4.51
new 172.17.4.52
line 172.17.4.52
pl 172.17.4.51
end

Required: Remove duplicate lines from the above file i.e. print only the first occurrence of the lines which appeared exactly twice and for lines those appear more than twice or appeared only once, no action required.

i.e. Required output should look like this:

begin
ip 172.17.4.53
line 172.17.4.52
pl 172.17.4.51
pl 172.17.4.51
new 172.17.4.52
pl 172.17.4.51
end

The python script 'remove-duplicate.py' :

d = {}

fp = open("file.txt.nodup","w")
text_file = open("file.txt", "r")
lines = text_file.readlines()
for line in lines:
    if not line in d.keys():
        d[line] = 0
    d[line] = d[line] + 1

for line in lines:
    if d[line] == 0:
        continue
    elif d[line] == 2:
        fp.write(line)
        d[line] = 0
    else:
        fp.write(line)

Executing it:

$ python remove-duplicate.py
$ cat file.txt.nodup
begin
ip 172.17.4.53
line 172.17.4.52
pl 172.17.4.51
pl 172.17.4.51
new 172.17.4.52
pl 172.17.4.51
end

Remove duplicate based on field using python

Input file:

$ cat file.txt
DD:12
AA:11
EE:13
AA:11
BB:09
DD:13
AA:78

Required output: Keep only 1st occurrence of each unique first field. i.e. required output:

DD:12
AA:11
EE:13
BB:09

Python script:

d = {}

input = file('file.txt')
for line in input:
   ff = line.split(':',1)[0]
   if ff not in d:
      d[ff] = 1
      print line,

Awk alternative:

$ awk -F ":" '!x[$1]++' file.txt
DD:12
AA:11
EE:13
BB:09

Grouping related items using python dictionary

Thought of trying a awk post that I did someday back on my bash scripting blog.

Input file:

$ cat data.txt
Manager1|sw1
Manager3|sw5
Manager1|sw4
Manager2|sw9
Manager2|sw12
Manager1|sw2
Manager1|sw0

Required output: Group the similar engineers which are under common Manager. i.e. required output:

Manager3|sw5
Manager2|sw9,sw12
Manager1|sw1,sw4,sw2,sw0

The python program:

d={}

fp = open("grp.txt","w")
for line in open("data.txt"):
   line=line.strip().split("|")
   d.setdefault(line[0],[])
   d[line[0]].append(line[1])

print d
for i,j in d.iteritems():
   fp.write(i+"|"+','.join(j)+"\n")

Output file after executing above script:

$ cat grp.txt
Manager3|sw5
Manager2|sw9,sw12
Manager1|sw1,sw4,sw2,sw0

Related concepts:
setdefault(key[, default])
If key is in the dictionary, return its value. If not, insert key with a value of default and return default. default defaults to None.

Dictionary iteritems : Read here

Lookup file in python using dictionary

Input files:
- main.txt contains id:name details
- lkfile contains the result of a particular exam in the format pass/fail:id

$ cat main.txt
id341:Mr X
id990:Mr Y
id223:Mr P
id212:Mr N
id183:Mr L

$ cat lkfile
fail:id223
pass:id341
fail:id183
pass:id990
pass:id212
pass:id555

Required:
Update main.txt with the results from lkfile i.e. required output:

pass:Mr X
pass:Mr Y
fail:Mr P
pass:Mr N
fail:Mr L

The python script using python Dictionaries:

def lookupf(file1,file2,outfile):
    fp = open(outfile,"w")
    a={}
    for line in open(file1):
        f = line.strip().split(":")
        a[f[1]]=f[0]

    for line2 in open(file2):
        f2 = line2.strip().split(":")
        if len(f2) == 2:
            if a.has_key(f2[0]):
                fp.write(a[f2[0]] + ":" + f2[1]+"\n")
            else:
                fp.write(line2.strip())
    fp.close()

#Calling the function
lookupf("lkfile","main.txt","out.txt")

Executing:

$ python lookup.py
$ cat out.txt
pass:Mr X
pass:Mr Y
fail:Mr P
pass:Mr N
fail:Mr L

Related concepts:

- The awk alternative would be:

$ awk '
    BEGIN {FS=OFS=":"}
    NR==FNR{a[$2]=$1;next}a[$1]{$1=a[$1]}1
' lkfile main.txt

- More about python dictionaries
- Python mapping type has_key