Intro to XPath

In the previous sections, we used a lot of different methods to find needed data without knowing the HTML way to the tag. To extract desired parts of code describing where the elements exactly are (in your HTML structure), you can use XPath.

XPath provides us with simple Python-friendly syntax to navigate through the HTML file. The notation is familiar with folder organization on your computer. For example, in the syntax of the XPathes / means move forward to one generation, like in the directory hierarchy where the same symbol moves us deeper into the folder's content. For instance:

python
xpath = "/html/body/div"

Here we define the path to all div tags in body tags of html tags.

To specify which div from a variety of tags you want, enclose its number in square brackets []:

python
xpath = "/html/head/div[2]"

This path for the following tree will detect the second p of the body (numeration starts from 1):

If you want to detect all p tags everywhere, use //. This symbol is used to pass tags. For example, you want to detect all p blocks but don’t know the whole path to each one:

python
xpath = "//p"

It also can be used when you know the beginning or part of your path:

python
xpath = "/html/body//p"

The code above directs all p tags in the body tag.