HTML Scraping Walkthrough

What is HTML scraping?

taking the the text that makes up a website, taking what you want from it and ordering for yourself.

When do I want to do this are there any criteria?

When you are on a website and there is a table of some sort or a list of links to other pages that have
information that you want.

What tools exist to help me?

I immediately looked into using python to break into scraping.
I was not disappointed

lxml

Beautiful Soup

csv

Regular Expressions

I eventually decided to go with regular expressions. Why?

probably the biggest reason was regex101.com it made testing my regular expressions much easier that it would have been without.
second regular expressions are everywhere so looking up any problems I was having wasn't as bad as
any of the other choices I tried.
thirdly setup was very easy on any computer all the libraries that I needed to import were already installed
with the python package.

Here is a very Simple code example

         import requests
         import re

         page = requests.get("http://ephe.github.io/grimoire/").text

         first_page_pattern = re.compile('spells/[\w\-\s]*">([\w\s]*)')

         for i in re.finditer(first_page_pattern,text):
            print i.group(1)
               print '\n'

         

How do I set this up?

The first line of code after imports is getting the html of the website and turning it into a string.
this is where youo would use surf to that website and copy the page source into regex101.com
and play with your pattern until it captures most if not everything you need to capture.
re.finditer(pattern,string,flags=0) creates an array of MatchObjects that you iterate through in a for loop
the number of groups each MatchObject has depends on how many captures you setup in your pattern