Python list append example - divide by two

Input file:

$ cat file.txt
h1|u|1
h2|5|1|1
rec1|1239400800|Sat|fan1|AX|2|10035|-|2|50
rec2|1239400800|Sat|fan1|AX|2|-|-|2|17
rec5|1239400801|Sat|fan3|AY|5|10035|-|2|217
rec8|1239400804|Sat|fan5|AX|2|5|-|2|970

Required Output:
- Lines starting with "h1" or "h2", no action required, just print.
- Lines starting with "rec", divide the values starting from 6th field by 2.

Required output is:

h1|u|1
h2|5|1|1
rec1|1239400800|Sat|fan1|AX|1|5017|-|1|25
rec2|1239400800|Sat|fan1|AX|1|-|-|1|8
rec5|1239400801|Sat|fan3|AY|2|5017|-|1|108
rec8|1239400804|Sat|fan5|AX|1|2|-|1|485

The python script:

fp = open("file.txt", "rU")
lines = fp.readlines()
fp.close()

for line in lines:
       if line.startswith("h1"):
               print line,
       if line.startswith("h2"):
               print line,
       if line.startswith("rec"):
               f=line.split("|")
               r = f[5:]
               l = []
               for each in r:
                       try:
                               l.append(str(int(each)/2))
                       except ValueError:
                               l.append(each)

               t = "|".join(f[0:5]) + "|" + "|".join(l)
               print t.rstrip()

Python sort file based on last field

Input file:

$ cat file.txt
IN,90,453
US,12,1,120
NZ,89,200
WI,500
TS,12,124

Required output: Sort the above comma delimited file based on the last field (column). i.e. required output:

US,12,1,120
TS,12,124
NZ,89,200
IN,90,453
WI,500

Solution:
The solution using Awk in UNIX bash shell is here. And here is the python one:

$ python
Python 2.5.2 (r252:60911, Jan 20 2010, 21:48:48)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> d_list = [line.strip() for line in open("file.txt")]
>>> d_list
['IN,90,453', 'US,12,1,120', 'NZ,89,200', 'WI,500', 'TS,12,124']
>>> d_list.sort(key = lambda line: line.split(",")[-1])
>>> d_list
['US,12,1,120', 'TS,12,124', 'NZ,89,200', 'IN,90,453', 'WI,500']
>>> for line in d_list:
...     print line
...
US,12,1,120
TS,12,124
NZ,89,200
IN,90,453
WI,500
>>>

Some notes:
Accessing last element of a list in python:
A negative index accesses elements from the end of the list counting backwards. The last element of any non-empty list is always list[-1].

Simple python file lookup function for newbie

Config file 'ip-mapping.txt' is a file of the following format:

$ cat /home/testusr/work/ip-mapping.txt
#id:ip1,ip2,ip3
200:172.17.4.12,172.17.4.14,172.17.4.10
205:172.17.4.14,172.17.4.14,172.17.4.11
210:172.17.4.12,172.17.4.18,172.17.4.18
208:172.17.4.11,172.17.4.10,172.17.4.19

Required: Create a simple python function which will accept an 'id' and will return 'ip1' from the list of ips.

The python script:

import os,sys

config = '/home/testusr/work/ip-mapping.txt'
if not os.path.exists(config):
    print config+' file not present'
    sys.exit()

def getip(id):
   all = open(config).readlines()
   for line in all:
        if line.startswith('#'):
            continue
        f=line.split(":")
        if f[0]==id:
                return f[1].split(',')[0]

ip=getip('205')
print ip

Executing it:

$ python get-ip.py
172.17.4.14

I am sure there will be much better solutions to this problem, please comment, really appreciated.

The description about Exit function of 'sys' module (source) :

sys.exit([arg])
Exit from Python. This is implemented by raising the SystemExit exception, so cleanup actions
specified by finally clauses of try statements are honored, and it is possible to intercept the exit
attempt at an outer level. The optional argument arg can be an integer giving the exit status
(defaulting to zero), or another type of object. If it is an integer, zero is considered “successful
termination” and any nonzero value is considered “abnormal termination” by shells and the
like. Most systems require it to be in the range 0-127, and produce undefined results otherwise.

Some systems have a convention for assigning specific meanings to specific exit codes, but these
are generally underdeveloped; Unix programs generally use 2 for command line syntax errors
and 1 for all other kind of errors. If another type of object is passed, None is equivalent to
passing zero, and any other object is printed to sys.stderr and results in an exit code of 1. In
particular, sys.exit("some error message") is a quick way to exit a program when an error occurs.

Related posts on lookup on file using python:

• Lookup file in python using dictionary
• Find text string in file using python
• Lookup file operation in python
  

Split a file into sub files in python

Input file 'file.txt' is basically a log file containing running information of certain device interfaces in the following format:

$ cat file.txt
debug: on
max allowed connection: 3
tr#45
Starting: interface 78e23
Fan Status: On
Speed: -
sl no: 3431212-2323-90
vendor: aledaia
Stopping: interface 78e23
tr#90
newdebug received
Starting: interface 78e24
Fan Status: Off
Speed: 5670
sl no: 3431212-2323-90
vendor: aledaia
Stopping: interface 78e24
Starting: interface 68e73
Fan Status: On
Speed: 1200
sl no: 3431212-2323-90
vendor: aledaia
Stopping: interface 68e73
tr#99

Required:

Split the above file into sub-files such that
- Each sub file conatins information of an interface (basically information from 'Starting' and 'Stopping' of the interface)
- Sub-file name should be of the format: interface-name_someSLno.txt

The python script:

flag=0;c=0
for line in open("file.txt"):
    line=line.strip()
    if line.startswith("Stopping"):
        flag=0
        o.close()
    if line.startswith("Starting"):
        interface=line.split(" ")[2]
        flag=1;c=c+1
        o=open(interface+"_"+str(c)+".txt","w")
    if flag and not line.startswith("Starting"):
        print >>o, line

Output:

$ cat 78e23_1.txt
Fan Status: On
Speed: -
sl no: 3431212-2323-90
vendor: aledaia

$ cat 78e24_2.txt
Fan Status: Off
Speed: 5670
sl no: 3431212-2323-90
vendor: aledaia

$ cat 68e73_3.txt
Fan Status: On
Speed: 1200
sl no: 3431212-2323-90
vendor: aledaia

Remove all except digits using python

Input file:

$ cat file.txt
4590:21333 2ewwq13232
12ada1212w1 1
13224 9#09io#
qw2323000 9023

Required: From the above file only keep the digits (i.e. remove all other characters except digits)

Way1: Using python Regular Expression special character '\D' which matches any non-digit character (equivalent to the set [^0-9])

$ python
Python 2.5.2 (r252:60911, Jul 22 2009, 15:35:03)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
>>> import re
>>> for line in open('file.txt'):
...     re.sub("\D", "",line)
...
'459021333213232'
'12121211'
'13224909'
'23230009023'
>>>

Another way : Using python filter built-in function to iterate isdigit() on all lines of the file.

>>>
>>> for line in open('file.txt'):
...     filter(lambda x: x.isdigit(), line)
...
'459021333213232'
'12121211'
'13224909'
'23230009023'
>>>

Change file delimiter using Python

Input file is comma delimited:

$ cat /tmp/file.txt
5232,92338,84545,34,
2233,25644,23233,23,
6211,1212,4343,434,
2434,621171,9121,33,

Required:

Convert the above comma(,) delimited file to a colon(:) delimited file such that there is no colon at the end of each line.

Python solution:

$ python
Python 2.5.2 (r252:60911, Jul 22 2009, 15:35:03)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> fp = open("/tmp/file.txt.new","w")
>>> for line in open('/tmp/file.txt'):
...     fp.write(line.strip()[:-1].replace(',',':')+'\n')
...
>>>

Output:

$ cat /tmp/file.txt.new
5232:92338:84545:34
2233:25644:23233:23
6211:1212:4343:434
2434:621171:9121:33

Alternative solutions:

An alternative using UNIX sed will be:

$ sed -e 's/,/:/g' -e 's/:$//g' /tmp/file.txt

And a related post using UNIX awk can be found on my bash scripting blog here

Python - print section of file using line number

e.g. Print the section of input file 'input.txt' between line number 22 and 89.

Using python enumerate function sequence numbers:

for i,line in enumerate(open("file.txt")):
    if i >= 21 and i < 89 :
        print line,

And if you want to write the section to a new file say '/tmp/fileA'

fp = open("/tmp/fileA","w")
for i,line in enumerate(open("file.txt")):
    if i >= 21 and i < 89 :
        fp.write(line)

Another approach:

print(''.join(open('file.txt', 'r').readlines()[21:89])),

And if you wish to write the section to a new file say '/tmp/fileB'

fp = open("/tmp/fileB","w")
fp.write(''.join(open('file.txt', 'r').readlines()[21:89])),

Read about python enumerate function here and below is a small example using python enumerate function:

>>> for i, student in enumerate(['Alex', 'Ryan', 'Deb']):
...     print i, student
...
0 Alex
1 Ryan
2 Deb
>>>

Also find my other post on Extracting section of a file using line numbers applying awk, sed, Perl, vi editor and UNIX/Linux head and tail command techniques.

Print line above pattern in python

Input file: 'data.txt' contains results of a set of students in the following format.

$ cat data.txt
id:502
Result:Pass
id:909
Result:Fail
id:503
Result:Pass
id:501
Result:Pass
id:802
Result:Fail

Required:
Print the Ids of the students who have passed the exam.

The python program:

fp = open("passedids.txt","w")
data = open("data.txt").readlines()
for i in range(len(data)):
        if data[i].startswith("Result:Pass"):
                fp.write(data[i-1].split(":")[1])

Output:

$ cat passedids.txt
502
503
501

Python - delete lines between two pattern

Input file:

$ cat input.txt
test1
test2
test3
BEGIN
test4
test5
test6
END
test7
test8
test9
BEGIN
test10
test11
END
test12

Required:
From the above file delete the lines which are between a BEGIN-END block and print rest of the lines.

The python script deleteline.py:

flag = 1
linelist = open("input.txt").readlines()
for line in linelist:
    if line.startswith("BEGIN"):
        flag = 0
    if line.startswith("END"):
        flag = 1
    if flag and not line.startswith("END"):
       print line,

Executing it:

$ python deleteline.py
test1
test2
test3
test7
test8
test9
test12

Now if we need to print the lines which are between a BEGIN-END block.
Here is a modification of the above scirpt.

flag = 1
linelist = open("input.txt").readlines()
for line in linelist:
    if line.startswith("BEGIN"):
        flag = 0
    if line.startswith("END"):
        flag = 1
    if not flag and not line.startswith("BEGIN"):
       print line,

Executing it:

$ python printline.py
test4
test5
test6
test10
test11

Related post:

- Lookup file operation using python

Print last instance of file - python example

Input file:

$ cat data.txt
k:begin:0
i:0:66
i:1:76
k:end:0
k:begin:7
i:0:55
i:1:65
i:2:57
k:end:7
k:begin:2
i:0:10
i:1:0
k:end:7
k:begin:2
i:0:46
k:end:7
k:begin:9
i:0:66
i:1:56
i:2:46
i:3:26
k:end:7

Required: Print last instance of the above file. One instance being from "k:begin" to "k:end"

The python program:

result =[]
all = open("data.txt").readlines()
for line in all[::-1]: #start from last ;proceed up
        f=line.split(":")
        if f[0]=="k" and f[1]=="end":
                continue
        elif f[0]=="k" and f[1]=="begin":
                break
        else: result.append(line)
print result
print "\nlast instance is\n"
print ''.join(result[::-1]) #reverse

Executing it:

$ python ins.py
['i:3:26\n', 'i:2:46\n', 'i:1:56\n', 'i:0:66\n']

last instance is

i:0:66
i:1:56
i:2:46
i:3:26

Related post:
- Print first few instances from file using python

Truncate file extension using python glob

My current directory contains the following 2 files.

$ ls -1
20061117.dat.dat
details.dat.dat.dat

Required: Move(rename) the above files to single .dat extension (e.g. details.dat.dat.dat to details.dat)
The python code using glob module:

>>> import os,glob
>>> for file in glob.glob("*.dat"):
...     newF=".".join(file.split(".")[:2])
...     os.rename(file,newF)
...

Now:

$ ls -1
20061117.dat
details.dat

Using glob module we can use wildcards with Python according to the rules used by the Unix shell. More about it can be found here

Few more examples:

# lists all files in the current directory
glob.glob('*')

# returns all .dat extension files
glob.glob('*.dat')

# lists all files starting with a letter, followed by 3 characters (numbers, letters) and any ending.
glob.glob('[a-z]???.*')

Performing multiple split in python

This post is mainly for python newbies.
Input file:

$ cat data.txt
File Start
#Comment
Gid034:s9823,I1290,s9034,s1230
Gid309:s9034,I5678,s1293,s4590
Gid124:s2145,K9008,s2381,s0234
Gid213:s9012,N9034,s8913,s9063
#End

Required: Extract the 3 rd field (colored blue) from the above file. i.e. required output:

I1290
I5678
K9008
N9034

Here we would need to split the required lines twice (one for field separator : and then for comma) to extract the desired column.

Using bash cut command or awk, the solution would be:
$ grep ^Gid data.txt | cut -d":" -f2 | cut -d"," -f2
$ awk -F"[:,]" '/^Gid/{print $3}' data.txt

The python code:

for line in open("data.txt"):
    if line.startswith("Gid"):
        print line.split(":")[1].split(",")[1]

Move files based on condition in python

Contents of /tmp/mydir/

$ ls /tmp/mydir/ | paste -

logWA241.dat
logWA249.dat
logWA258.dat
logWA259.dat

Required: Move the above files to directories under /tmp/mydir such that logWA241.dat should go to dir /tmp/mydir/1, similarly logWA258.dat to /tmp/mydir/8 (i.e. dir name with last digit before .dat extn)

The python script:

import os
DIR="/tmp/mydir"
for file in os.listdir(DIR):
   Absfile = os.path.join(DIR,file)
   if os.path.isfile(Absfile) and file.endswith(".dat"):
       Dname = Absfile.split(".")[:-1][-1][-1:]
       Dname = os.path.join(DIR,Dname)
       if not os.path.exists(Dname):
           os.mkdir(Dname)
           os.system('mv '+Absfile+' '+Dname)
       else:
           os.system('mv '+Absfile+' '+Dname)

The contents of /tmp/mydir/ after exection of the above script.

$ ls -R /tmp/mydir/

o/p:

/tmp/mydir/:
1  8  9

/tmp/mydir/1:
logWA241.dat

/tmp/mydir/8:
logWA258.dat

/tmp/mydir/9:
logWA249.dat  logWA259.dat

Related concepts and modules:
- Python os module

Remove duplicate based on field using python

Input file:

$ cat file.txt
DD:12
AA:11
EE:13
AA:11
BB:09
DD:13
AA:78

Required output: Keep only 1st occurrence of each unique first field. i.e. required output:

DD:12
AA:11
EE:13
BB:09

Python script:

d = {}

input = file('file.txt')
for line in input:
   ff = line.split(':',1)[0]
   if ff not in d:
      d[ff] = 1
      print line,

Awk alternative:

$ awk -F ":" '!x[$1]++' file.txt
DD:12
AA:11
EE:13
BB:09

A lookup file operation using python

Details of input files:

details.txt contains the details of some students. The details of a student starts with its #ID,Name,Class,Year ans Status
idlist.txt contains the IDs of the students who has passed the exam.

$ cat idlist.txt
ID55
ID12
ID90

$ cat details.txt
#ID10
Name:Mr A
Class:IX
Year:1985
Satus=Nill
#ID12
Name:Mr B
Class:X
Year:1987
#ID10
Name:Mr X
Class:X
Year:1983
#ID90
Name:Mr Y
Class:IX
Year:1984
#ID55
Name:Mr Z
Class:X
Year:1985

Required: Pull out the details of the students(from details.txt) who has passed the exam(whose ID is present in idlist.txt).

idlist = open("idlist.txt").readlines()
idlist = [i.strip() for i in idlist]
detailslist = open("details.txt").readlines()
flag = 0
fp = open("filter.out", "w")
for id in idlist:
        for lines in detailslist:
                if lines.startswith("#") and id in lines:
                        flag = 1
                if lines.startswith("#") and not id in lines:
                        flag = 0
                if flag:
                        fp.write(lines)
fp.close

Executing the filter.py:

$ python filter.py
$ cat filter.out
#ID55
Name:Mr Z
Class:X
Year:1985
#ID12
Name:Mr B
Class:X
Year:1987
#ID90
Name:Mr Y
Class:IX
Year:1984

So filter.out contains the required output.

Related concepts and functions:

>>> idlist = open("idlist.txt").readlines()
>>> idlist
['ID55\n', 'ID12\n', 'ID90\n']
>>> idlist = [i.strip() for i in idlist]
>>> idlist
['ID55', 'ID12', 'ID90']

Python - append a field based on condition

Thought of solving the same problem that I post on awk in my bash scripting blog

Input file:

$ cat file.txt
ID5,17.95,107.0,Y
ID5,6.56,12.3,Y
ID5,7.36,22.5,Y
ID5,4.03,72.2,Y
ID6,282.8,134.1,Y
ID6,111.56,61.7,Y
ID6,171.24,72.4,Y
ID7,125.6,89,Y

Output required: Append a field with value "Agg line" if first field (ID field) is the first unique one, for rest of its (that ID) occurrences, append a field with text "sub-line". .i.e. required output:

Agg line,ID5,17.95,107.0,Y
sub-line,ID5,6.56,12.3,Y
sub-line,ID5,7.36,22.5,Y
sub-line,ID5,4.03,72.2,Y
Agg line,ID6,282.8,134.1,Y
sub-line,ID6,111.56,61.7,Y
sub-line,ID6,171.24,72.4,Y
Agg line,ID7,125.6,89,Y

The python program for solving the same:

fp = open("file.txt", "rU")
lines = fp.readlines()
fp.close()

f_f=" "
for line in lines:
    f=line.split(",")
    if f[0]==f_f:
        print "sub-line,"+line.rstrip()
    else:
        f_f=f[0]
        print "Agg line,"+line.rstrip()

Python - extract blocks of data from file

Input file contains addresses of 3 persons:

$ cat details.txt
Details:Mr X
Koramangala Post
3rd Cross, 17th Main
PIN: 12345
Details:Mr Y:details
NGV
PIN: 45678
Details:Mr Z:details
5th Ave, #23
NHM Post
LKV
PIN: 32456

Output required: We are required to divide/split the above file into 3 sub-files, each should contain one address.
The python program:

f=0
for line in open("details.txt"):
    line=line.strip()
    if "Details" in line:
        filename=line.split(":")[1]
        o=open(filename.replace(" ","_"),"w")
        f=1
    if f:print >>o, line

Output: Sub-files generated after execution of the above program:

$ cat Mr_X
Details:Mr X
Koramangala Post
3rd Cross, 17th Main
PIN: 12345

$ cat Mr_Y
Details:Mr Y:details
NGV
PIN: 45678

$ cat Mr_Z
Details:Mr Z:details
5th Ave, #23
NHM Post
LKV
PIN: 32456

- Related solution using awk from my bash scripting blog

Sum and average calculation using python

Input file:

$ cat data.txt
12313.23
4005.12
13434.12
2133.21
213123.21
9000.23

Required: Calculate simple sum and average of the above float values.
Python script:

data = open("data.txt").read().split()
s = sum([ float(i) for i in data ])
print "Sum=" , s
print "Avg="  , s/len(data)

Executing it:

$ python sum-avg.py
Sum= 254009.12
Avg= 42334.8533333

Awk solution for the same:

$ awk '
    {s+=$0}
    END {printf "Sum =%10.2f,Avg = %10.2f\n",s,s/NR}
' data.txt

Output:

Sum = 254009.12,Avg =   42334.85

Related functions and concepts:
a) Python for loop read here
b) The built-in function len() returns the length of a string
c) python numeric types read here