Python - unpack arguments from list with star operator

We can use *-operator (star operator) to unpack the arguments out of a list or tuple. Here is an example with python built in range() function :

>>> mylist = [3, 10]
>>> mylist
[3, 10]
>>> range(mylist[0], mylist[1])
[3, 4, 5, 6, 7, 8, 9]
>>> range(*mylist)
[3, 4, 5, 6, 7, 8, 9]
>>> 

Python merge line with line above

Input file:

$ cat file.txt
500:120:100:X
:100:120
200:900:125
120:120
:900
120:345
12:900:1234:34
:90

Required: Join the lines which startswith : with the previous line.
i.e. required output is:

500:120:100:X:100:120
200:900:125
120:120:900
120:345
12:900:1234:34:90

The python script to achieve this:

data=open("file.txt").read().split("\n")
for i,line in enumerate(data):
    if line.startswith(":"):
        data[i-1]= data[i-1]+line
        data.pop(i)
print '\n'.join(data),

Executing it:

$ python merge-lines.py
500:120:100:X:100:120
200:900:125
120:120:900
120:345
12:900:1234:34:90

The solution using UNIX Awk can be found here

More about python enumerate function can be found here. Mentioned below is a small example on python enumerate function

>>> for i, student in enumerate(['Alex', 'Ryan', 'Deb']):
...     print i, student
...
0 Alex
1 Ryan
2 Deb
>>>

Related Posts:
- Print section of file using line number - Python
- Print line next to pattern using Python
- Print line above pattern using Python

New learning:
Python list pop method:
list.pop([i])
Remove the item at the given position in the list, and return it. If no index is specified, a.pop() removes and returns the last item in the list.

Python list append example - divide by two

Input file:

$ cat file.txt
h1|u|1
h2|5|1|1
rec1|1239400800|Sat|fan1|AX|2|10035|-|2|50
rec2|1239400800|Sat|fan1|AX|2|-|-|2|17
rec5|1239400801|Sat|fan3|AY|5|10035|-|2|217
rec8|1239400804|Sat|fan5|AX|2|5|-|2|970

Required Output:
- Lines starting with "h1" or "h2", no action required, just print.
- Lines starting with "rec", divide the values starting from 6th field by 2.

Required output is:

h1|u|1
h2|5|1|1
rec1|1239400800|Sat|fan1|AX|1|5017|-|1|25
rec2|1239400800|Sat|fan1|AX|1|-|-|1|8
rec5|1239400801|Sat|fan3|AY|2|5017|-|1|108
rec8|1239400804|Sat|fan5|AX|1|2|-|1|485

The python script:

fp = open("file.txt", "rU")
lines = fp.readlines()
fp.close()

for line in lines:
       if line.startswith("h1"):
               print line,
       if line.startswith("h2"):
               print line,
       if line.startswith("rec"):
               f=line.split("|")
               r = f[5:]
               l = []
               for each in r:
                       try:
                               l.append(str(int(each)/2))
                       except ValueError:
                               l.append(each)

               t = "|".join(f[0:5]) + "|" + "|".join(l)
               print t.rstrip()

Python - Replace based on another file

$ cat main.txt
P|34|90
T|12
R|0|1291870414|ip1|890
R|1|1291870415|ip5|690
R|2|1291870415|ip1|899
R|3|1291870412|ip2|896
R|4|1291870418|ip3|999
R|5|1291870419|ip5|191

$ cat lookup.txt
ip7|172.17.4.8
ip1|172.17.4.3
ip5|172.17.4.9
ip4|172.17.4.2
ip3|172.17.4.1
ip2|172.17.4.6
ip6|172.17.4.7

Required Output:
Replace the 4th field (pipe delimited) of the 'R' lines of 'main.txt' with the corresponding lookup value from 'lookup.txt' i.e. 'ip1' to be replaced with '172.17.4.3', 'ip2' with '172.17.4.6' etc.

P|34|90
T|12
R|0|1291870414|172.17.4.3|890
R|1|1291870415|172.17.4.9|690
R|2|1291870415|172.17.4.3|899
R|3|1291870412|172.17.4.6|896
R|4|1291870418|172.17.4.1|999
R|5|1291870419|172.17.4.9|191

The python script:

import sys
d={}
for line in open("lookup.txt"):
    line=line.strip().split("|")
    d[line[0]]=line[-1]
for line in open(sys.argv[1]):
    if line.startswith('P'):
        print line,
    if line.startswith('T'):
        print line,
    if line.startswith('R'):
        line=line.strip().split("|")
        print '|'.join(line[0:3])+'|'+d[line[3]]+'|'+'|'.join(line[4:])

Executing it:

$ python replace-from-file.py main.txt
P|34|90
T|12
R|0|1291870414|172.17.4.3|890
R|1|1291870415|172.17.4.9|690
R|2|1291870415|172.17.4.3|899
R|3|1291870412|172.17.4.6|896
R|4|1291870418|172.17.4.1|999
R|5|1291870419|172.17.4.9|191

Related Posts:
- Lookup file operation using Python
- Lookup file in python using Dictionary
- Simple python file lookup function
- Find text string in file in Python

Python sort file based on last field

Input file:

$ cat file.txt
IN,90,453
US,12,1,120
NZ,89,200
WI,500
TS,12,124

Required output: Sort the above comma delimited file based on the last field (column). i.e. required output:

US,12,1,120
TS,12,124
NZ,89,200
IN,90,453
WI,500

Solution:
The solution using Awk in UNIX bash shell is here. And here is the python one:

$ python
Python 2.5.2 (r252:60911, Jan 20 2010, 21:48:48)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> d_list = [line.strip() for line in open("file.txt")]
>>> d_list
['IN,90,453', 'US,12,1,120', 'NZ,89,200', 'WI,500', 'TS,12,124']
>>> d_list.sort(key = lambda line: line.split(",")[-1])
>>> d_list
['US,12,1,120', 'TS,12,124', 'NZ,89,200', 'IN,90,453', 'WI,500']
>>> for line in d_list:
...     print line
...
US,12,1,120
TS,12,124
NZ,89,200
IN,90,453
WI,500
>>>

Some notes:
Accessing last element of a list in python:
A negative index accesses elements from the end of the list counting backwards. The last element of any non-empty list is always list[-1].

bsddb185 sunaudiodev - Python 2.6 Ubuntu installation

If you are arriving on this page looking for the solution of following error message during python2.6 installation (make) on your Ubuntu:

Failed to find the necessary bits to build these modules:
bsddb185           sunaudiodev                       
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

then here is the solution:

$ wget http://www.lysium.de/sw/python2.6-disable-old-modules.patch

$ patch -p1 < python2.6-disable-old-modules.patch

For a complete guide to install python 2.6 on your Ubuntu, you can check this page, its really useful.

Python - Remove duplicate lines from file

Objective : Remove duplicate lines from a file (print first occurrence) which appeared exactly twice.

Input file:

$ cat file.txt
begin
ip 172.17.4.53
line 172.17.4.52
pl 172.17.4.51
pl 172.17.4.51
new 172.17.4.52
line 172.17.4.52
pl 172.17.4.51
end

Required: Remove duplicate lines from the above file i.e. print only the first occurrence of the lines which appeared exactly twice and for lines those appear more than twice or appeared only once, no action required.

i.e. Required output should look like this:

begin
ip 172.17.4.53
line 172.17.4.52
pl 172.17.4.51
pl 172.17.4.51
new 172.17.4.52
pl 172.17.4.51
end

The python script 'remove-duplicate.py' :

d = {}

fp = open("file.txt.nodup","w")
text_file = open("file.txt", "r")
lines = text_file.readlines()
for line in lines:
    if not line in d.keys():
        d[line] = 0
    d[line] = d[line] + 1

for line in lines:
    if d[line] == 0:
        continue
    elif d[line] == 2:
        fp.write(line)
        d[line] = 0
    else:
        fp.write(line)

Executing it:

$ python remove-duplicate.py
$ cat file.txt.nodup
begin
ip 172.17.4.53
line 172.17.4.52
pl 172.17.4.51
pl 172.17.4.51
new 172.17.4.52
pl 172.17.4.51
end

Simple python file lookup function for newbie

Config file 'ip-mapping.txt' is a file of the following format:

$ cat /home/testusr/work/ip-mapping.txt
#id:ip1,ip2,ip3
200:172.17.4.12,172.17.4.14,172.17.4.10
205:172.17.4.14,172.17.4.14,172.17.4.11
210:172.17.4.12,172.17.4.18,172.17.4.18
208:172.17.4.11,172.17.4.10,172.17.4.19

Required: Create a simple python function which will accept an 'id' and will return 'ip1' from the list of ips.

The python script:

import os,sys

config = '/home/testusr/work/ip-mapping.txt'
if not os.path.exists(config):
    print config+' file not present'
    sys.exit()

def getip(id):
   all = open(config).readlines()
   for line in all:
        if line.startswith('#'):
            continue
        f=line.split(":")
        if f[0]==id:
                return f[1].split(',')[0]

ip=getip('205')
print ip

Executing it:

$ python get-ip.py
172.17.4.14

I am sure there will be much better solutions to this problem, please comment, really appreciated.

The description about Exit function of 'sys' module (source) :

sys.exit([arg])
Exit from Python. This is implemented by raising the SystemExit exception, so cleanup actions
specified by finally clauses of try statements are honored, and it is possible to intercept the exit
attempt at an outer level. The optional argument arg can be an integer giving the exit status
(defaulting to zero), or another type of object. If it is an integer, zero is considered “successful
termination” and any nonzero value is considered “abnormal termination” by shells and the
like. Most systems require it to be in the range 0-127, and produce undefined results otherwise.

Some systems have a convention for assigning specific meanings to specific exit codes, but these
are generally underdeveloped; Unix programs generally use 2 for command line syntax errors
and 1 for all other kind of errors. If another type of object is passed, None is equivalent to
passing zero, and any other object is printed to sys.stderr and results in an exit code of 1. In
particular, sys.exit("some error message") is a quick way to exit a program when an error occurs.

Related posts on lookup on file using python:

• Lookup file in python using dictionary
• Find text string in file using python
• Lookup file operation in python
  

Python - count instances without a specific line

Input file:

$ cat data.txt
k:begin:0
i:0:66
i:1:76
t:1:143
k:end:0
k:begin:7
i:0:55
i:1:65
i:2:57
k:end:7
k:begin:2
i:0:10
i:1:0
t:1:10
k:end:7
k:begin:2
i:0:46
t:0:46
k:end:7
k:begin:9
i:0:66
i:1:56
i:2:46
i:3:26
k:end:7

Required: Count total number of instances (one instance being from a 'k:begin' to 'k:end' line) which do not have a 't' line associated.

The python program:

import sys
count=0
data = open(sys.argv[1]).readlines()
for i in range(len(data)):
       if data[i].startswith("k:end") and data[i-1].split(":")[0]!="t":
               count=count+1
print count

Executing it:

$ python count_no_t.py data.txt
2

Related post:

- Print last instance of a file using Python
- Print line next to pattern in Python
- Print line above pattern in Python

Python convert string to tuple & list

Let's check the use of python 'tuple' and 'list' in-built functions.

tuple([iterable])

It returns a 'tuple' whose items are the same and in the same order as iterable‘s items. iterable may be a sequence, a container that supports iteration, or an iterator object.

tuple('xyz') returns ('x', 'y', 'z') and tuple([1, 2, 3]) returns (1, 2, 3)

e.g.

$ cat file.txt
Python Prog
Readline
Programming

Now:

>>> for line in open("file.txt"):
...     t = tuple(line)
...     print t
...
('P', 'y', 't', 'h', 'o', 'n', ' ', 'P', 'r', 'o', 'g', '\n')
('R', 'e', 'a', 'd', 'l', 'i', 'n', 'e', '\n')
('P', 'r', 'o', 'g', 'r', 'a', 'm', 'm', 'i', 'n', 'g', '\n')
>>>

list([iterable])

It returns a list whose items are the same and in the same order as iterable‘s items. iterable may be either a sequence, a container that supports iteration, or an iterator object. If iterable is already a list, a copy is made and returned, similar to iterable[:]. For instance, list('xyz') returns ['x', 'y', 'z'] and list( (1, 2, 3) ) returns [1, 2, 3].

>>>
>>> for line in open("file.txt"):
...     l = list(line)
...     print l
...
['P', 'y', 't', 'h', 'o', 'n', ' ', 'P', 'r', 'o', 'g', '\n']
['R', 'e', 'a', 'd', 'l', 'i', 'n', 'e', '\n']
['P', 'r', 'o', 'g', 'r', 'a', 'm', 'm', 'i', 'n', 'g', '\n']
>>>

Split a file into sub files in python

Input file 'file.txt' is basically a log file containing running information of certain device interfaces in the following format:

$ cat file.txt
debug: on
max allowed connection: 3
tr#45
Starting: interface 78e23
Fan Status: On
Speed: -
sl no: 3431212-2323-90
vendor: aledaia
Stopping: interface 78e23
tr#90
newdebug received
Starting: interface 78e24
Fan Status: Off
Speed: 5670
sl no: 3431212-2323-90
vendor: aledaia
Stopping: interface 78e24
Starting: interface 68e73
Fan Status: On
Speed: 1200
sl no: 3431212-2323-90
vendor: aledaia
Stopping: interface 68e73
tr#99

Required:

Split the above file into sub-files such that
- Each sub file conatins information of an interface (basically information from 'Starting' and 'Stopping' of the interface)
- Sub-file name should be of the format: interface-name_someSLno.txt

The python script:

flag=0;c=0
for line in open("file.txt"):
    line=line.strip()
    if line.startswith("Stopping"):
        flag=0
        o.close()
    if line.startswith("Starting"):
        interface=line.split(" ")[2]
        flag=1;c=c+1
        o=open(interface+"_"+str(c)+".txt","w")
    if flag and not line.startswith("Starting"):
        print >>o, line

Output:

$ cat 78e23_1.txt
Fan Status: On
Speed: -
sl no: 3431212-2323-90
vendor: aledaia

$ cat 78e24_2.txt
Fan Status: Off
Speed: 5670
sl no: 3431212-2323-90
vendor: aledaia

$ cat 68e73_3.txt
Fan Status: On
Speed: 1200
sl no: 3431212-2323-90
vendor: aledaia

Python - print last few characters

Input file:

$ cat file.txt
sldadop233masdsa213313131ada121
sltadop233masdsa813313133cso128
slyadop233masdsa11331313Kada134
slqadop233masdsa31331313tada162

Required: Print last 6 characters of each line of the above input file.

The python script:

$ cat extract-last.py
import sys
for line in sys.stdin:
    print '%s' % (line[-7:-1])

Executing it:

$ python extract-last.py < file.txt
ada121
cso128
ada134
ada162

Things to learn:
- How to read a file in python from stdin

Other alternatives in UNIX are:

#Using bash parameter substitution
$ while read line ; do echo ${line: -6}; done < file.txt

#Since all lines are of fixed length, we can use 'cut' command
$ cut -c26-31 file.txt

#Using sed
$ sed 's/^.*\(......\)$/\1/' file.txt

#Using awk
$ awk '{ print substr( $0, length($0) - 5, length($0) ) }'  file.txt

Remove all except digits using python

Input file:

$ cat file.txt
4590:21333 2ewwq13232
12ada1212w1 1
13224 9#09io#
qw2323000 9023

Required: From the above file only keep the digits (i.e. remove all other characters except digits)

Way1: Using python Regular Expression special character '\D' which matches any non-digit character (equivalent to the set [^0-9])

$ python
Python 2.5.2 (r252:60911, Jul 22 2009, 15:35:03)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
>>> import re
>>> for line in open('file.txt'):
...     re.sub("\D", "",line)
...
'459021333213232'
'12121211'
'13224909'
'23230009023'
>>>

Another way : Using python filter built-in function to iterate isdigit() on all lines of the file.

>>>
>>> for line in open('file.txt'):
...     filter(lambda x: x.isdigit(), line)
...
'459021333213232'
'12121211'
'13224909'
'23230009023'
>>>

Change file delimiter using Python

Input file is comma delimited:

$ cat /tmp/file.txt
5232,92338,84545,34,
2233,25644,23233,23,
6211,1212,4343,434,
2434,621171,9121,33,

Required:

Convert the above comma(,) delimited file to a colon(:) delimited file such that there is no colon at the end of each line.

Python solution:

$ python
Python 2.5.2 (r252:60911, Jul 22 2009, 15:35:03)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> fp = open("/tmp/file.txt.new","w")
>>> for line in open('/tmp/file.txt'):
...     fp.write(line.strip()[:-1].replace(',',':')+'\n')
...
>>>

Output:

$ cat /tmp/file.txt.new
5232:92338:84545:34
2233:25644:23233:23
6211:1212:4343:434
2434:621171:9121:33

Alternative solutions:

An alternative using UNIX sed will be:

$ sed -e 's/,/:/g' -e 's/:$//g' /tmp/file.txt

And a related post using UNIX awk can be found on my bash scripting blog here

Print line next to pattern in python

Input file: 'file.txt' contains results of a set of students in the following format (i.e. for any student result precedes the student id)

$ cat file.txt
Result:Pass
id:502
Result:Fail
id:909
Result:Pass
id:503
Result:Pass
id:501
Result:Fail
id:802

Required: Print the Ids of the students who have passed the exam.

The python program:

fp = open("passedids.txt","w")
data = open("file.txt").readlines()
for i in range(len(data)):
        if data[i].startswith("Result:Pass"):
                fp.write(data[i+1].split(":")[1])

Executing it:

$ python printnext.py
$ cat passedids.txt
502
503
501

Another python alternative:

fp=open('file.txt','r')
previous_line = ""

for current_line in fp:
    if 'Result:Pass' in previous_line:
        print current_line.split(":")[1],
    previous_line = current_line
fp.close()

Executing it:

$ python printnext1.py
502
503
501

Related post:

- Print line above pattern in python

Python - print section of file using line number

e.g. Print the section of input file 'input.txt' between line number 22 and 89.

Using python enumerate function sequence numbers:

for i,line in enumerate(open("file.txt")):
    if i >= 21 and i < 89 :
        print line,

And if you want to write the section to a new file say '/tmp/fileA'

fp = open("/tmp/fileA","w")
for i,line in enumerate(open("file.txt")):
    if i >= 21 and i < 89 :
        fp.write(line)

Another approach:

print(''.join(open('file.txt', 'r').readlines()[21:89])),

And if you wish to write the section to a new file say '/tmp/fileB'

fp = open("/tmp/fileB","w")
fp.write(''.join(open('file.txt', 'r').readlines()[21:89])),

Read about python enumerate function here and below is a small example using python enumerate function:

>>> for i, student in enumerate(['Alex', 'Ryan', 'Deb']):
...     print i, student
...
0 Alex
1 Ryan
2 Deb
>>>

Also find my other post on Extracting section of a file using line numbers applying awk, sed, Perl, vi editor and UNIX/Linux head and tail command techniques.

Python - Adding numbers in a list

Lets see some of ways in python to add the numbers present in a list.

Suppose:

>>> numlist = [10,20,5,30]
>>> numlist
[10, 20, 5, 30]
>>> print sum(numlist)
65

Using python built in function 'reduce'

>>> numlist
[10, 20, 5, 30]
>>> def add(x, y): return x + y
...
>>> sum = reduce(add, numlist)
>>> sum
65

Enhancing the above using python 'lambda' function

>>> numlist
[10, 20, 5, 30]
>>> reduce(lambda b,a: a+b, numlist)
65
>>>

Or using python for loop:

>>> numlist
[10, 20, 5, 30]
>>> sum = 0
>>> for i in numlist:
...     sum += i
...
>>> sum
65
>>>

Python - time difference between dates

Required:

Find the time difference between two dates (of following format) in seconds and in hh:mm:ss format.

e.g.

date1='Oct/09/2009 10:58:01' and
date2='Oct/10/2009 12:17:10'

find the difference between date1 and date2 in seconds(i.e. 91149 seconds) and later convert it to hh:mm:ss format (i.e. 25:19:09).

The complete python program:

import sys,time,string,getopt

def usage():
    print "Usage: adbtimediff.py -f <fromTime> -t <toTime> \n"
    sys.exit(2)


def parse_args():
    global fromTime,toTime
    fromTime = toTime = ""

    try:
        opts, args = getopt.getopt(sys.argv[1:], "f:t:", ["fromtime", "totime"])
    except getopt.GetoptError:
        print "Invalid arguments, exiting"
        sys.exit(2)

    for arg, val in opts:
        if  arg in ("-f","--fromtime"):
            fromTime = val
        elif arg in ("-t","--totime"):
            toTime = val

    if fromTime == toTime == "" :
        usage()

def compute_time(time1):
    t = time1.split(':')
    return time.mktime(time.strptime(":".join(t[0:len(t)]),"%b/%d/%Y %H:%M:%S"))

def subtract(list):
    return list[1] - list[0]

def time_convert(secs):
    secs = int(secs)
    mins = secs // 60
    hrs = mins // 60
    return "%02d:%02d:%02d" % (hrs, mins % 60, secs % 60)

def main():
    parse_args()
    print "Fromtime : " + str(fromTime) + '\n' + "Totime : " + str(toTime)
    timelist = [ fromTime, toTime ]
    s = map(compute_time,timelist)
    d = subtract(s)
    print "diff in seconds : " + str(d)
    f = str(d).split('.')
    final = time_convert(f[0])
    print "Total difference in required format : " + str(final)

main()

Executing the above script:

$ python timediff.py -f 'Oct/09/2009  10:58:01' -t 'Oct/10/2009  12:17:10'

Output:

Fromtime : Oct/09/2009  10:58:01
Totime : Oct/10/2009  12:17:10
diff in seconds : 91149.0
Total difference in required format : 25:19:09

Related concepts and posts:

- Convert seconds to hh:mm:ss format using python
- Python time.mktime
- Python time.strftime
- Python map
- Python getopt

Python - seconds to hh-mm-ss conversion

Solution1: Using python 'time' module strftime function.

 
Python 2.5.2 (r252:60911, Jul 22 2009, 15:35:03)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.strftime('%H:%M:%S', time.gmtime(7302))
'02:01:42'
>>> time.strftime('%H:%M:%S', time.gmtime(86399))
'23:59:59'
>>> time.strftime('%H:%M:%S', time.gmtime(86405))
'00:00:05'

So as seen above this solution works only for num seconds < 1 day (86400 seconds)

Solution2: Using python datetime module, timedelta object.

>>> import datetime
>>> x = datetime.timedelta(seconds=7302)
>>> str(x)
'2:01:42'
>>> x = datetime.timedelta(seconds=86399)
>>> str(x)
'23:59:59'
>>> x = datetime.timedelta(seconds=86405)
>>> str(x)
'1 day, 0:00:05'

Solution3: Using normal division in python

import sys

secs = int(sys.argv[1])
mins = secs // 60
hrs = mins // 60

#hh:mm:ss
print "%02d:%02d:%02d" % (hrs, mins % 60, secs % 60)

#mm:ss
print "%02d:%02d" % (mins, secs % 60)

Executing it:

$ python timeconv.py 7302
02:01:42
121:42

$ python timeconv.py 86399
23:59:59
1439:59

$ python timeconv.py 86405
24:00:05
1440:05

Print line above pattern in python

Input file: 'data.txt' contains results of a set of students in the following format.

$ cat data.txt
id:502
Result:Pass
id:909
Result:Fail
id:503
Result:Pass
id:501
Result:Pass
id:802
Result:Fail

Required:
Print the Ids of the students who have passed the exam.

The python program:

fp = open("passedids.txt","w")
data = open("data.txt").readlines()
for i in range(len(data)):
        if data[i].startswith("Result:Pass"):
                fp.write(data[i-1].split(":")[1])

Output:

$ cat passedids.txt
502
503
501