REGEX
No account is required. Learn the material, try the examples, and mark it complete locally when you are ready to move on.
Text Processing
Lets say our objective is to find some patterns in textual data. Then it becomes really important for us to understand how python stores and search for text inside the strings.
Just for the reference I'm including useful patterns on top of the file.
import re # regular expressionstext1 = "This is a sample email sent by ashishchaudhary62@gmail.com"
text1'This is a sample email sent by ashishchaudhary62@gmail.com'Lets say our objective is to extract just the email from the string
re.findall("\w+@\w+?.com", text1)['ashishchaudhary62@gmail.com']# 0,1
text2 = "It is a sample string 123445554 444 string"
re.findall("[0-9]+?4", text2)['1234', '45554', '44']re.findall("[0-9]+?4", text2)['1234', '45554', '44']re.findall("[a-z]+ish", text1)['ashish']re.findall("[\w]+@gmail.com", text1)['ashishchaudhary62@gmail.com']Wow, what just happened?
Lets understand what exactly happened with the help of simple examples.
text1'This is a sample email sent by ashishchaudhary62@gmail.com'Lets imagine first that our objective is to extract just the word "email" from the string.
Python has an in-built module for text pattern matching.
import re
re stands for regular expressions. Its the re module which gives users the ability to extract patterns.
We have inside re some useful methods. We are going to look at re.findall, re.search, re.sub
re.findall("pattern", string)
re.findall("email", text1)['email']Yep, that's how easy things are with re. Just specify the pattern to match as the first argument of re.findall method and as the second argument pass the string in which you want to find that pattern.
re.findall(".com", text1)['.com']Our string does not contain "google.com" so nothing will be fetched out.
re.findall("google.com", text1)[]text1'This is a sample email sent by ashishchaudhary62@gmail.com'# capture all alphabet a inside the text
re.findall("a", text1)
['a', 'a', 'a', 'a', 'a', 'a', 'a']re.findall("c", text1)['c', 'c']re.findall("sample", text1)['sample']text2 = """ This is a multi line string which contains absolutely
random data. Our objective is to focus on text pattern matching.
I'm creating a random text string here.
"""
print(text2) This is a multi line string which contains absolutely
random data. Our objective is to focus on text pattern matching.
I'm creating a random text string here.
re.findall("\n", text2)
# \n is a newline character['\n', '\n', '\n']re.findall("text", text2)['text', 'text']Let's look at some challenging examples
\w inside the re language means any word character
To shorten the output I'll only display first 10 characters
re.findall("\w", text2)[:10]['T', 'h', 'i', 's', 'i', 's', 'a', 'm', 'u', 'l']re has different patterns using which we can catch different types of patterns
- character means keep matching the pattern till it matches
re.findall("\w+", text2)['This',
'is',
'a',
'multi',
'line',
'string',
'which',
'contains',
'absolutely',
'random',
'data',
'Our',
'objective',
'is',
'to',
'focus',
'on',
'text',
'pattern',
'matching',
'I',
'm',
'creating',
'a',
'random',
'text',
'string',
'here']In the above case, we kept matching word characters (a-z A-Z 0-9) till the pattern matched, till the pattern encountered whitespace or till a different character set not specified in pattern gets encountered.
some_str = "this is A random strinG"re.findall("a", some_str)['a']re.findall iterates over a string, character by character and searches for a given pattern. So, in this case. In the first iteration of re.findall("a", some_str), re will try to match the pattern "a" with the first character of our string, that is "t"
The string "this is A random strinG" starts from "t" right...
first iteration: pattern is matched with "t", pattern is "a", no result, pattern didn't get matched
Second iteration: pattern is match with "h" and so on till the length of the whole string till all "a" are extracted out
pattern = 'r'
re.findall(pattern, some_str)
# Extracted are all the character set matched to the pattern['r', 'r']re.findall("this", some_str)['this']re.findall("thiS", some_str) # thiS does not exist
# this do exist but
# this and thiS are different as python is case sensitive[]"this" == "thiS"False"this" == "this"True# In every iteration, check if the text string is a
# character pattern inside []
some_str = "G this is A random strinG"
re.findall("[Gabcg]", some_str)['G', 'a', 'G']# In every iteration, check if the character is either a or r
some_str = "this is A random strinG"
re.findall("[ar]", some_str)['r', 'a', 'r']# In every iteration, check if the character is either a, b or r
re.findall("[abr]", some_str)['r', 'a', 'r']In every iteration, check if the character is either a, b or t
order wouldn't matter
The search would go like
In first iteration of search
is the first character of text string "t"
is the first character of text string "b"
is the first character of text string "a"
and so on for the second and the rest of the iterations
re.findall("[tba]", some_str)['t', 'a', 't']some_integers = "1234 431 776 2341"
pattern = "123"
re.findall(pattern, some_integers)['123']some_integers = "1234 431 776 2341"
pattern = "23"
re.findall(pattern, some_integers)['23', '23']some_integers = "1234 431 776 2341"
# search if each character is either 2 or 3
pattern = "[23]"
re.findall(pattern, some_integers)['2', '3', '3', '2', '3']some_integers = "1234 431 776 2341 33223"
# + requires that the pattern will be matched one or more time
pattern = "[23]+" # "23", "2", and so on combinations of 2 or 3
# So long as the pattern keeps matching,
# the pattern will be part of the
# same string in the output
re.findall(pattern, some_integers)['23', '3', '23', '33223']
some_integers = "1234 431 776 2341"
# Find whitespaces 1 and 2
# \s stands for whitespace
pattern = "[1\s2]"
re.findall(pattern, some_integers)['1', '2', ' ', '1', ' ', ' ', '2', '1']print("a\tb") # \t is tab charactera b
some_integers = "1234 431 776 23417"
pattern = "[71][7]" # 17 77
re.findall(pattern, some_integers)['77', '17']some_integers = "1234 431 776 2341"
pattern = "[23][23]" # 23 32 22 33
re.findall(pattern, some_integers)['23', '23']some_integers = "1234 431 776 2341"
pattern = "[23][41]" # feched out string should be of length 2
# first character should be either 2 or 3,
# while second character should be either 4 or 1
re.findall(pattern, some_integers)['34', '31', '34']# I'll change the string a little
some_integers = "123333224 43321 7372326 2341"
# + means that the pattern will be fetched out one or more time
pattern = "[23]+"
# So long as the pattern keeps matching, the pattern will be part of the
# same string in the output
re.findall(pattern, some_integers)['2333322', '332', '3', '232', '23']# I'll change the string a little
some_integers = "123333224 43321 7372326 2341"
# * means that the pattern will be fetched out zero or more time
# if there's no pattern in the string, empty string is returned
pattern = "[23]*"
# So long as the pattern keeps matching, the pattern will be part of the
# same string in the output
re.findall(pattern, some_integers)['',
'2333322',
'',
'',
'',
'332',
'',
'',
'',
'3',
'',
'232',
'',
'',
'23',
'',
'',
'']some_integers = "123333224 43321 7372326 2341"
pattern = "[23]3"
re.findall(pattern, some_integers)
['23', '33', '33', '23', '23']Notice how the second character of the returned string is always 3 while the first character could take either 2 or 3 So, only 2 possible outputs: 33 or 23
some_integers = "123333224 43321 7372326 2341"
pattern = "[23][3]+"
# is not same as pattern: "[23]+"
# pattern "[23][3]+" means match 2 or 3 at first match and
# after that keep match any 3 present in the string
re.findall(pattern, some_integers)['23333', '33', '23', '23']some_integers = "123333224 43321 7372326 2341"
pattern = "[23]+"
re.findall(pattern, some_integers)['2333322', '332', '3', '232', '23']Lets try different patterns now
Just for demonstration purpose I'll introduce a couple patterns There could be many ways to go about this
str1 = "$45,000,675.00%"
pattern = "[0123456789]"
re.findall(pattern, str1)['4', '5', '0', '0', '0', '6', '7', '5', '0', '0']
# This following pattern will fetch out those demical zeros as well
pattern = "[0123456789]" # "[0-9]" also does the same thing as [0123456789]
re.findall(pattern, str1)['4', '5', '0', '0', '0', '6', '7', '5', '0', '0']pattern = "[0-9]" # [0-9] does the same thing as [0123456789]
re.findall(pattern, str1)['4', '5', '0', '0', '0', '6', '7', '5', '0', '0']Join method
We can join any list or tuple using the string method join
"_".join(("this", "is", "a", "string"))'this_is_a_string'"_".join(["this", "is", "a", "string"])'this_is_a_string'"".join(re.findall(pattern, str1))'4500067500'The output means 4.5 billions while our number should have been 45 million
str1'$45,000,675.00%'pattern = "."
re.findall(pattern, str1)['$', '4', '5', ',', '0', '0', '0', ',', '6', '7', '5', '.', '0', '0', '%']pattern = "\."
re.findall(pattern, str1)['.']pattern = "[0-9]+\."
re.findall(pattern, str1)['675.']pattern = "[0-9,]+\."
re.findall(pattern, str1)['45,000,675.']str1'$45,000,675.00%'pattern2 = "([0-9,]+)\."
re.findall(pattern2, str1)['45,000,675']This could work. Now we have our number in millions.
pattern2 = "([0-9,]+)\."
list1 = re.findall(pattern2, str1)
list1
['45,000,675']list1[0]'45,000,675'output = list1[0]
output'45,000,675're.findall("[0-9]+", output)['45', '000', '675']output_list = re.findall("[0-9]+", output)
output_list['45', '000', '675']"".join(output_list)'45000675'output_str = "".join(output_list)
output_str'45000675'Quite tedious I know. I made it tedious for learning perspective.
Lets make it easy
string split method
str3 = "this is a random string"
str3'this is a random string'str3.split(" ")['this', 'is', 'a', 'random', 'string']str4 = "This_is_a_random_string"
str4'This_is_a_random_string'str4.split("_")['This', 'is', 'a', 'random', 'string']str5 = "this,is,a,random,string"
str5'this,is,a,random,string'str5.split(sep=",")['this', 'is', 'a', 'random', 'string']split method of the class string just splits the string based on a separator.
I hope, you now know what split method does.
str1 = '$45,000,675.00%'
str1'$45,000,675.00%'str1.split(sep=".")['$45,000,675', '00%']str1.split(sep=".")[0]'$45,000,675'new_str = str1.split(".")[0]
new_str'$45,000,675're.findall("[0-9]", new_str)['4', '5', '0', '0', '0', '6', '7', '5']new_list_of_numbers = re.findall("[0-9]", new_str)
new_list_of_numbers['4', '5', '0', '0', '0', '6', '7', '5']"".join(new_list_of_numbers)'45000675'new_str = "".join(new_list_of_numbers)
new_str'45000675'int(new_str)45000675One liner
str1 = '$45,000,675.00%'
int("".join(re.findall("[0-9]", str1.split(".")[0])))45000675dollar character:
should be searched as "\$" in the pattern we have to escape $ if we want to specifically search for $ as $ has a special meaning in python $ means search at the end of line \$ will literally search for the character $
str5 = """Our string has some prices of
products $45000 $675.00 $300"""
re.findall("\$", str5)['$', '$', '$']re.findall("\$[0-9]+", str5)['$45000', '$675', '$300']re.findall("\$[0-9\.]+", str5)['$45000', '$675.00', '$300']Objective
Find unique emails in a specified text string
text1 = """This is a group email conversation in a company
ashishchaudhary62@gmail.com: Hey, how are you all? I just joined.
I hope to learn new things under guidance of you seniors.
somerandomemail@gmail.com: Who are you? Don't text at late hours.
Be professional, if you've nothing to offer, then don't unnecessarily text.
tickles62@gmail.com: Chill somerandomemail, he is a new joinee, don't
pull his legs.
Hi. Ashish. Welcome to Tickles private. We are happy to have you
as our partner.
ashishchaudhary62@gmail.com: Hello sir. So sorry I didn't know
you were the founder. I thought, this was just the employee group.
somerandomemail@gmail.com: Don't assume Tickles sir will reply to your
messages junior.
tickles62@gmail.com: somerandomemail I didn't mean to reply, but now
you've forced me to *facepalm
somerandomemail@gmail.com: ashish, this place is like our family. You don't
have to be so uptight. See the reaction of sir, he's so fed up of us.
Maybe that's why we needed new people.
ashishchaudhary62@gmail.com: Wow, I don't know if this is a
sarcastic conversation or an honest one. But this place is fun.
Thank you all
"""
print(text1)This is a group email conversation in a company
ashishchaudhary62@gmail.com: Hey, how are you all? I just joined.
I hope to learn new things under guidance of you seniors.
somerandomemail@gmail.com: Who are you? Don't text at late hours.
Be professional, if you've nothing to offer, then don't unnecessarily text.
tickles62@gmail.com: Chill somerandomemail, he is a new joinee, don't
pull his legs.
Hi. Ashish. Welcome to Tickles private. We are happy to have you
as our partner.
ashishchaudhary62@gmail.com: Hello sir. So sorry I didn't know
you were the founder. I thought, this was just the employee group.
somerandomemail@gmail.com: Don't assume Tickles sir will reply to your
messages junior.
tickles62@gmail.com: somerandomemail I didn't mean to reply, but now
you've forced me to *facepalm
somerandomemail@gmail.com: ashish, this place is like our family. You don't
have to be so uptight. See the reaction of sir, he's so fed up of us.
Maybe that's why we needed new people.
ashishchaudhary62@gmail.com: Wow, I don't know if this is a
sarcastic conversation or an honest one. But this place is fun.
Thank you all
re.findall("[\w]+@[\w]+.com", text1)['ashishchaudhary62@gmail.com',
'somerandomemail@gmail.com',
'tickles62@gmail.com',
'ashishchaudhary62@gmail.com',
'somerandomemail@gmail.com',
'tickles62@gmail.com',
'somerandomemail@gmail.com',
'ashishchaudhary62@gmail.com']set() function
Converts a list or tuple like sequence to a set. A set contains only unique values.
set(re.findall("[\w]+@[\w]+.com", text1)){'ashishchaudhary62@gmail.com',
'somerandomemail@gmail.com',
'tickles62@gmail.com'}I'm all for challenging patterns
x = """From corrupted@data@spam.com sample.email@gmail.com
Sat Feb 5 19:14:16 2012"""
print(x)From corrupted@data@spam.com sample.email@gmail.com
Sat Feb 5 19:14:16 2012
y = re.findall('\S+?@\S+', x)
print(y)
# corrupt email also got fetched!['corrupted@data@spam.com', 'sample.email@gmail.com']
Greedy matching
If we have two character in a series of pattern matches, say the symbol appears two or more times. Where to end the pattern match now, at the first encounter of underscore or the last encounter of underscore character
re.findall(".+", "this\nis")['this', 'is']# re tends to get greedy by default
import re
x = 'Lets _ use the _ underscore character'
y = re.findall('.+_', x)
print(y)['Lets _ use the _']
re extracted the _ character from the end
import re
x = 'Lets _ use the _ underscore character'
y = re.findall('.+?_', x)
print(y)['Lets _', ' use the _']
x = """From corrupted@data@spam.com sample.email@gmail.com
Sat Feb 5 19:14:16 2012"""
x'From corrupted@data@spam.com sample.email@gmail.com\nSat Feb 5 19:14:16 2012're.findall("\s[\w\.]+@[\.\w]+.com", x)[' sample.email@gmail.com']re.findall("\s([\w\.]+@[\.\w]+.com)", x)['sample.email@gmail.com']Lets look at a very practical case of greedy matching
HTML tags
sample_html = """
<body>
<h1>Heading 1</h1>
<p>This is some random paragraph.
<br>I have forgotten a lot about how to write html.</p>
<p>But that's not the objective. Objective is to get specific tags.
Anyhow, if I had to work in HTML I'd just google things that I may have forgotten.
</p>
</body>
"""
print(sample_html)
<body>
<h1>Heading 1</h1>
<p>This is some random paragraph.
<br>I have forgotten a lot about how to write html.</p>
<p>But that's not the objective. Objective is to get specific tags.
Anyhow, if I had to work in HTML I'd just google things that I may have forgotten.
</p>
</body>
If we search for the html tag pattern in our usual way, we'll encounter the greedy behaviour of re
re.findall("<.+>", sample_html)['<body>',
'<h1>Heading 1</h1>',
'<p>',
'<br>I have forgotten a lot about how to write html.</p>',
'<p>',
'</p>',
'</body>']In the above multi line string we said, match the character < and then match any non line character till we encounter the character >
Our objective is also to find these < tags > but we need some way to tell re, wait stop at the first > character you encounter.
re.findall("<.+?>", sample_html)['<body>', '<h1>', '</h1>', '<p>', '<br>', '</p>', '<p>', '</p>', '</body>']re.sub()
substitute
some_str = "I use this string too much"
some_str'I use this string too much're.sub("string", "super_string", some_str)'I use this super_string too much'print(sample_html)
<body>
<h1>Heading 1</h1>
<p>This is some random paragraph.
<br>I have forgotten a lot about how to write html.</p>
<p>But that's not the objective. Objective is to get specific tags.
Anyhow, if I had to work in HTML I'd just google things that I may have forgotten.
</p>
</body>
print(re.sub("<.+?>", "<HeheTag>", sample_html))
<HeheTag>
<HeheTag>Heading 1<HeheTag>
<HeheTag>This is some random paragraph.
<HeheTag>I have forgotten a lot about how to write html.<HeheTag>
<HeheTag>But that's not the objective. Objective is to get specific tags.
Anyhow, if I had to work in HTML I'd just google things that I may have forgotten.
<HeheTag>
<HeheTag>
re.search()
re.search() returns a yes (True) or no (False) depending on if the search argument was found or not
if re.search("a", "sample string"):
print("This block was executed")
else:
print("pattern not found")This block was executed
if re.search("at", "sample string"):
print("This block was executed")
else:
print("pattern not found")pattern not found
Sample Patterns
str10 = """This is a string.
This 123 is a number.
This is an email: tookie@gmail.com
_ _ _
* *
"""
print(str10)This is a string.
This 123 is a number.
This is an email: tookie@gmail.com
_ _ _
* *
re.findall("[a-z]+", str10)['his',
'is',
'a',
'string',
'his',
'is',
'a',
'number',
'his',
'is',
'an',
'email',
'tookie',
'gmail',
'com']re.findall("[a-zA-Z]+", str10)['This',
'is',
'a',
'string',
'This',
'is',
'a',
'number',
'This',
'is',
'an',
'email',
'tookie',
'gmail',
'com']re.findall("[a-zA-Z0-9]+", str10)['This',
'is',
'a',
'string',
'This',
'123',
'is',
'a',
'number',
'This',
'is',
'an',
'email',
'tookie',
'gmail',
'com']re.findall("\w+", str10)['This',
'is',
'a',
'string',
'This',
'123',
'is',
'a',
'number',
'This',
'is',
'an',
'email',
'tookie',
'gmail',
'com',
'_',
'_',
'_']re.findall("[\w]+", str10)['This',
'is',
'a',
'string',
'This',
'123',
'is',
'a',
'number',
'This',
'is',
'an',
'email',
'tookie',
'gmail',
'com',
'_',
'_',
'_']Lesson resources
Supporting code and files from this part of the curriculum.
Finished this lesson?
Completion is saved in this browser without creating an account.