Python - unpack arguments from list with star operator

We can use *-operator (star operator) to unpack the arguments out of a list or tuple. Here is an example with python built in range() function :

>>> mylist = [3, 10]
>>> mylist
[3, 10]
>>> range(mylist[0], mylist[1])
[3, 4, 5, 6, 7, 8, 9]
>>> range(*mylist)
[3, 4, 5, 6, 7, 8, 9]
>>> 

Python merge line with line above

Input file:

$ cat file.txt
500:120:100:X
:100:120
200:900:125
120:120
:900
120:345
12:900:1234:34
:90

Required: Join the lines which startswith : with the previous line.
i.e. required output is:

500:120:100:X:100:120
200:900:125
120:120:900
120:345
12:900:1234:34:90

The python script to achieve this:

data=open("file.txt").read().split("\n")
for i,line in enumerate(data):
    if line.startswith(":"):
        data[i-1]= data[i-1]+line
        data.pop(i)
print '\n'.join(data),

Executing it:

$ python merge-lines.py
500:120:100:X:100:120
200:900:125
120:120:900
120:345
12:900:1234:34:90

The solution using UNIX Awk can be found here

More about python enumerate function can be found here. Mentioned below is a small example on python enumerate function

>>> for i, student in enumerate(['Alex', 'Ryan', 'Deb']):
...     print i, student
...
0 Alex
1 Ryan
2 Deb
>>>

Related Posts:
- Print section of file using line number - Python
- Print line next to pattern using Python
- Print line above pattern using Python

New learning:
Python list pop method:
list.pop([i])
Remove the item at the given position in the list, and return it. If no index is specified, a.pop() removes and returns the last item in the list.

Python list append example - divide by two

Input file:

$ cat file.txt
h1|u|1
h2|5|1|1
rec1|1239400800|Sat|fan1|AX|2|10035|-|2|50
rec2|1239400800|Sat|fan1|AX|2|-|-|2|17
rec5|1239400801|Sat|fan3|AY|5|10035|-|2|217
rec8|1239400804|Sat|fan5|AX|2|5|-|2|970

Required Output:
- Lines starting with "h1" or "h2", no action required, just print.
- Lines starting with "rec", divide the values starting from 6th field by 2.

Required output is:

h1|u|1
h2|5|1|1
rec1|1239400800|Sat|fan1|AX|1|5017|-|1|25
rec2|1239400800|Sat|fan1|AX|1|-|-|1|8
rec5|1239400801|Sat|fan3|AY|2|5017|-|1|108
rec8|1239400804|Sat|fan5|AX|1|2|-|1|485

The python script:

fp = open("file.txt", "rU")
lines = fp.readlines()
fp.close()

for line in lines:
       if line.startswith("h1"):
               print line,
       if line.startswith("h2"):
               print line,
       if line.startswith("rec"):
               f=line.split("|")
               r = f[5:]
               l = []
               for each in r:
                       try:
                               l.append(str(int(each)/2))
                       except ValueError:
                               l.append(each)

               t = "|".join(f[0:5]) + "|" + "|".join(l)
               print t.rstrip()

Python - Replace based on another file

$ cat main.txt
P|34|90
T|12
R|0|1291870414|ip1|890
R|1|1291870415|ip5|690
R|2|1291870415|ip1|899
R|3|1291870412|ip2|896
R|4|1291870418|ip3|999
R|5|1291870419|ip5|191

$ cat lookup.txt
ip7|172.17.4.8
ip1|172.17.4.3
ip5|172.17.4.9
ip4|172.17.4.2
ip3|172.17.4.1
ip2|172.17.4.6
ip6|172.17.4.7

Required Output:
Replace the 4th field (pipe delimited) of the 'R' lines of 'main.txt' with the corresponding lookup value from 'lookup.txt' i.e. 'ip1' to be replaced with '172.17.4.3', 'ip2' with '172.17.4.6' etc.

P|34|90
T|12
R|0|1291870414|172.17.4.3|890
R|1|1291870415|172.17.4.9|690
R|2|1291870415|172.17.4.3|899
R|3|1291870412|172.17.4.6|896
R|4|1291870418|172.17.4.1|999
R|5|1291870419|172.17.4.9|191

The python script:

import sys
d={}
for line in open("lookup.txt"):
    line=line.strip().split("|")
    d[line[0]]=line[-1]
for line in open(sys.argv[1]):
    if line.startswith('P'):
        print line,
    if line.startswith('T'):
        print line,
    if line.startswith('R'):
        line=line.strip().split("|")
        print '|'.join(line[0:3])+'|'+d[line[3]]+'|'+'|'.join(line[4:])

Executing it:

$ python replace-from-file.py main.txt
P|34|90
T|12
R|0|1291870414|172.17.4.3|890
R|1|1291870415|172.17.4.9|690
R|2|1291870415|172.17.4.3|899
R|3|1291870412|172.17.4.6|896
R|4|1291870418|172.17.4.1|999
R|5|1291870419|172.17.4.9|191

Related Posts:
- Lookup file operation using Python
- Lookup file in python using Dictionary
- Simple python file lookup function
- Find text string in file in Python

Python sort file based on last field

Input file:

$ cat file.txt
IN,90,453
US,12,1,120
NZ,89,200
WI,500
TS,12,124

Required output: Sort the above comma delimited file based on the last field (column). i.e. required output:

US,12,1,120
TS,12,124
NZ,89,200
IN,90,453
WI,500

Solution:
The solution using Awk in UNIX bash shell is here. And here is the python one:

$ python
Python 2.5.2 (r252:60911, Jan 20 2010, 21:48:48)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> d_list = [line.strip() for line in open("file.txt")]
>>> d_list
['IN,90,453', 'US,12,1,120', 'NZ,89,200', 'WI,500', 'TS,12,124']
>>> d_list.sort(key = lambda line: line.split(",")[-1])
>>> d_list
['US,12,1,120', 'TS,12,124', 'NZ,89,200', 'IN,90,453', 'WI,500']
>>> for line in d_list:
...     print line
...
US,12,1,120
TS,12,124
NZ,89,200
IN,90,453
WI,500
>>>

Some notes:
Accessing last element of a list in python:
A negative index accesses elements from the end of the list counting backwards. The last element of any non-empty list is always list[-1].

Python - Remove duplicate lines from file

Objective : Remove duplicate lines from a file (print first occurrence) which appeared exactly twice.

Input file:

$ cat file.txt
begin
ip 172.17.4.53
line 172.17.4.52
pl 172.17.4.51
pl 172.17.4.51
new 172.17.4.52
line 172.17.4.52
pl 172.17.4.51
end

Required: Remove duplicate lines from the above file i.e. print only the first occurrence of the lines which appeared exactly twice and for lines those appear more than twice or appeared only once, no action required.

i.e. Required output should look like this:

begin
ip 172.17.4.53
line 172.17.4.52
pl 172.17.4.51
pl 172.17.4.51
new 172.17.4.52
pl 172.17.4.51
end

The python script 'remove-duplicate.py' :

d = {}

fp = open("file.txt.nodup","w")
text_file = open("file.txt", "r")
lines = text_file.readlines()
for line in lines:
    if not line in d.keys():
        d[line] = 0
    d[line] = d[line] + 1

for line in lines:
    if d[line] == 0:
        continue
    elif d[line] == 2:
        fp.write(line)
        d[line] = 0
    else:
        fp.write(line)

Executing it:

$ python remove-duplicate.py
$ cat file.txt.nodup
begin
ip 172.17.4.53
line 172.17.4.52
pl 172.17.4.51
pl 172.17.4.51
new 172.17.4.52
pl 172.17.4.51
end

Python - count instances without a specific line

Input file:

$ cat data.txt
k:begin:0
i:0:66
i:1:76
t:1:143
k:end:0
k:begin:7
i:0:55
i:1:65
i:2:57
k:end:7
k:begin:2
i:0:10
i:1:0
t:1:10
k:end:7
k:begin:2
i:0:46
t:0:46
k:end:7
k:begin:9
i:0:66
i:1:56
i:2:46
i:3:26
k:end:7

Required: Count total number of instances (one instance being from a 'k:begin' to 'k:end' line) which do not have a 't' line associated.

The python program:

import sys
count=0
data = open(sys.argv[1]).readlines()
for i in range(len(data)):
       if data[i].startswith("k:end") and data[i-1].split(":")[0]!="t":
               count=count+1
print count

Executing it:

$ python count_no_t.py data.txt
2

Related post:

- Print last instance of a file using Python
- Print line next to pattern in Python
- Print line above pattern in Python

Python convert string to tuple & list

Let's check the use of python 'tuple' and 'list' in-built functions.

tuple([iterable])

It returns a 'tuple' whose items are the same and in the same order as iterable‘s items. iterable may be a sequence, a container that supports iteration, or an iterator object.

tuple('xyz') returns ('x', 'y', 'z') and tuple([1, 2, 3]) returns (1, 2, 3)

e.g.

$ cat file.txt
Python Prog
Readline
Programming

Now:

>>> for line in open("file.txt"):
...     t = tuple(line)
...     print t
...
('P', 'y', 't', 'h', 'o', 'n', ' ', 'P', 'r', 'o', 'g', '\n')
('R', 'e', 'a', 'd', 'l', 'i', 'n', 'e', '\n')
('P', 'r', 'o', 'g', 'r', 'a', 'm', 'm', 'i', 'n', 'g', '\n')
>>>

list([iterable])

It returns a list whose items are the same and in the same order as iterable‘s items. iterable may be either a sequence, a container that supports iteration, or an iterator object. If iterable is already a list, a copy is made and returned, similar to iterable[:]. For instance, list('xyz') returns ['x', 'y', 'z'] and list( (1, 2, 3) ) returns [1, 2, 3].

>>>
>>> for line in open("file.txt"):
...     l = list(line)
...     print l
...
['P', 'y', 't', 'h', 'o', 'n', ' ', 'P', 'r', 'o', 'g', '\n']
['R', 'e', 'a', 'd', 'l', 'i', 'n', 'e', '\n']
['P', 'r', 'o', 'g', 'r', 'a', 'm', 'm', 'i', 'n', 'g', '\n']
>>>

Python - print last few characters

Input file:

$ cat file.txt
sldadop233masdsa213313131ada121
sltadop233masdsa813313133cso128
slyadop233masdsa11331313Kada134
slqadop233masdsa31331313tada162

Required: Print last 6 characters of each line of the above input file.

The python script:

$ cat extract-last.py
import sys
for line in sys.stdin:
    print '%s' % (line[-7:-1])

Executing it:

$ python extract-last.py < file.txt
ada121
cso128
ada134
ada162

Things to learn:
- How to read a file in python from stdin

Other alternatives in UNIX are:

#Using bash parameter substitution
$ while read line ; do echo ${line: -6}; done < file.txt

#Since all lines are of fixed length, we can use 'cut' command
$ cut -c26-31 file.txt

#Using sed
$ sed 's/^.*\(......\)$/\1/' file.txt

#Using awk
$ awk '{ print substr( $0, length($0) - 5, length($0) ) }'  file.txt

Python - time difference between dates

Required:

Find the time difference between two dates (of following format) in seconds and in hh:mm:ss format.

e.g.

date1='Oct/09/2009 10:58:01' and
date2='Oct/10/2009 12:17:10'

find the difference between date1 and date2 in seconds(i.e. 91149 seconds) and later convert it to hh:mm:ss format (i.e. 25:19:09).

The complete python program:

import sys,time,string,getopt

def usage():
    print "Usage: adbtimediff.py -f <fromTime> -t <toTime> \n"
    sys.exit(2)


def parse_args():
    global fromTime,toTime
    fromTime = toTime = ""

    try:
        opts, args = getopt.getopt(sys.argv[1:], "f:t:", ["fromtime", "totime"])
    except getopt.GetoptError:
        print "Invalid arguments, exiting"
        sys.exit(2)

    for arg, val in opts:
        if  arg in ("-f","--fromtime"):
            fromTime = val
        elif arg in ("-t","--totime"):
            toTime = val

    if fromTime == toTime == "" :
        usage()

def compute_time(time1):
    t = time1.split(':')
    return time.mktime(time.strptime(":".join(t[0:len(t)]),"%b/%d/%Y %H:%M:%S"))

def subtract(list):
    return list[1] - list[0]

def time_convert(secs):
    secs = int(secs)
    mins = secs // 60
    hrs = mins // 60
    return "%02d:%02d:%02d" % (hrs, mins % 60, secs % 60)

def main():
    parse_args()
    print "Fromtime : " + str(fromTime) + '\n' + "Totime : " + str(toTime)
    timelist = [ fromTime, toTime ]
    s = map(compute_time,timelist)
    d = subtract(s)
    print "diff in seconds : " + str(d)
    f = str(d).split('.')
    final = time_convert(f[0])
    print "Total difference in required format : " + str(final)

main()

Executing the above script:

$ python timediff.py -f 'Oct/09/2009  10:58:01' -t 'Oct/10/2009  12:17:10'

Output:

Fromtime : Oct/09/2009  10:58:01
Totime : Oct/10/2009  12:17:10
diff in seconds : 91149.0
Total difference in required format : 25:19:09

Related concepts and posts:

- Convert seconds to hh:mm:ss format using python
- Python time.mktime
- Python time.strftime
- Python map
- Python getopt

Print line above pattern in python

Input file: 'data.txt' contains results of a set of students in the following format.

$ cat data.txt
id:502
Result:Pass
id:909
Result:Fail
id:503
Result:Pass
id:501
Result:Pass
id:802
Result:Fail

Required:
Print the Ids of the students who have passed the exam.

The python program:

fp = open("passedids.txt","w")
data = open("data.txt").readlines()
for i in range(len(data)):
        if data[i].startswith("Result:Pass"):
                fp.write(data[i-1].split(":")[1])

Output:

$ cat passedids.txt
502
503
501

Print last instance of file - python example

Input file:

$ cat data.txt
k:begin:0
i:0:66
i:1:76
k:end:0
k:begin:7
i:0:55
i:1:65
i:2:57
k:end:7
k:begin:2
i:0:10
i:1:0
k:end:7
k:begin:2
i:0:46
k:end:7
k:begin:9
i:0:66
i:1:56
i:2:46
i:3:26
k:end:7

Required: Print last instance of the above file. One instance being from "k:begin" to "k:end"

The python program:

result =[]
all = open("data.txt").readlines()
for line in all[::-1]: #start from last ;proceed up
        f=line.split(":")
        if f[0]=="k" and f[1]=="end":
                continue
        elif f[0]=="k" and f[1]=="begin":
                break
        else: result.append(line)
print result
print "\nlast instance is\n"
print ''.join(result[::-1]) #reverse

Executing it:

$ python ins.py
['i:3:26\n', 'i:2:46\n', 'i:1:56\n', 'i:0:66\n']

last instance is

i:0:66
i:1:56
i:2:46
i:3:26

Related post:
- Print first few instances from file using python