markup = "
I want pizza and more pizza!
" soup = BeautifulSoup(markup, 'html.parser') first_b, second_b = soup.find_all('b') print first_b == second_b # True print first_b.previous_element == second_b.previous_element # False 

Если вы хотите выяснить, указывают ли две переменные на один и тот же
объект, используйте is :

print first_b is second_b # False 

Копирование объектов Beautiful Soup¶

Вы можете использовать copy.copy() для создания копии любого Tag или
NavigableString :

import copy p_copy = copy.copy(soup.p) print p_copy # 
I want pizza and more pizza! 

Копия считается равной оригиналу, так как у нее
такая же разметка, что и у оригинала, но это другой объект:

print soup.p == p_copy # True print soup.p is p_copy # False 

Единственная настоящая разница в том, что копия полностью отделена от
исходного дерева объекта Beautiful Soup, как если бы в отношении нее вызвали
метод extract() :

print p_copy.parent # None 

Это потому, что два разных объекта Tag не могут занимать одно и то же
пространство в одно и то же время.

Разбор части документа¶

(Обратите внимание, что эта функция не будет работать, если вы используете парсер html5lib.
Если вы используете html5lib, будет разобран весь документ, независимо
от обстоятельств. Это потому что html5lib постоянно переставляет части дерева разбора
в процессе работы, и если какая-то часть документа не
попала в дерево разбора, все рухнет. Чтобы избежать путаницы, в
примерах ниже я принудительно использую встроенный в Python
парсер HTML.)

SoupStrainer ¶

Класс SoupStrainer принимает те же аргументы, что и типичный
метод из раздела Поиск по дереву: name , attrs , string и **kwargs . Вот
три объекта SoupStrainer :

from bs4 import SoupStrainer only_a_tags = SoupStrainer("a") only_tags_with_id_link2 = SoupStrainer(id="link2") def is_short_string(string): return len(string)  10 only_short_strings = SoupStrainer(string=is_short_string) 

Вернемся к фрагменту из «Алисы в стране чудес»
и увидим, как выглядит документ, когда он разобран с этими
тремя объектами SoupStrainer :

html_doc = """  
The Dormouse's story 
Once upon a time there were three little sisters; and their names were Elsie, Lacie and Tillie; and they lived at the bottom of a well. 
.
  """ print(BeautifulSoup(html_doc, "html.parser", parse_only=only_a_tags).prettify()) #  # Elsie #  #  # Lacie #  #  # Tillie #  print(BeautifulSoup(html_doc, "html.parser", parse_only=only_tags_with_id_link2).prettify()) #  # Lacie #  print(BeautifulSoup(html_doc, "html.parser", parse_only=only_short_strings).prettify()) # Elsie # , # Lacie # and # Tillie # . # 

Вы также можете передать SoupStrainer в любой из методов. описанных в разделе
Поиск по дереву. Может, это не безумно полезно, но я
решил упомянуть:

soup = BeautifulSoup(html_doc) soup.find_all(only_short_strings) # [u'nn', u'nn', u'Elsie', u',n', u'Lacie', u' andn', u'Tillie', # u'nn', u'. ', u'n'] 

Устранение неисправностей¶

diagnose() ¶

Если у вас возникли проблемы с пониманием того, что Beautiful Soup делает с
документом, передайте документ в функцию Diagnose() . (Новое в
Beautiful Soup 4.2.0.) Beautiful Soup выведет отчет, показывающий,
как разные парсеры обрабатывают документ, и сообщит вам, если
отсутствует парсер, который Beautiful Soup мог бы использовать:

from bs4.diagnose import diagnose with open("bad.html") as fp: data = fp.read() diagnose(data) # Diagnostic running on Beautiful Soup 4.2.0 # Python version 2.7.3 (default, Aug 1 2012, 05:16:07) # I noticed that html5lib is not installed. Installing it may help. # Found lxml version 2.3.2.0 # # Trying to parse your data with html.parser # Here's what html.parser did with the document: # . 

Простой взгляд на вывод diagnose() может показать, как решить
проблему. Если это и не поможет, вы можете скопировать вывод Diagnose() , когда
обратитесь за помощью.

Ошибки при разборе документа¶

Существует два вида ошибок разбора. Есть сбои,
когда вы подаете документ в Beautiful Soup, и это поднимает
исключение, обычно HTMLParser.HTMLParseError . И есть
неожиданное поведение, когда дерево разбора Beautiful Soup сильно
отличается от документа, использованного для создания дерева.

Практически никогда источником этих проблемы не бывает Beautiful
Soup. Это не потому, что Beautiful Soup так прекрасно
написан. Это потому, что Beautiful Soup не содержит
кода, который бы разбирал документ. Beautiful Soup опирается на внешние парсеры. Если один парсер
не подходит для разбора документа, лучшим решением будет попробовать
другой парсер. В разделе Установка парсера вы найдете больше информации
и таблицу сравнения парсеров.

Наиболее распространенные ошибки разбора — это HTMLParser.HTMLParseError:
malformed start tag и HTMLParser.HTMLParseError: bad end
tag . Они оба генерируются встроенным в Python парсером HTML,
и решением будет установить lxml или
html5lib.

Наиболее распространенный тип неожиданного поведения — когда вы не можете найти
тег, который точно есть в документе. Вы видели его на входе, но
find_all() возвращает [] , или find() возвращает None . Это
еще одна распространенная проблема со встроенным в Python парсером HTML, который
иногда пропускает теги, которые он не понимает. Опять же, решение заключается в
установке lxml или html5lib .

Проблемы несоответствия версий¶

• SyntaxError: Invalid syntax (в строке ROOT_TAG_NAME =
  u'[document]’ ) — вызвано запуском версии Beautiful Soup на Python 2
  под Python 3 без конвертации кода.
• ImportError: No module named HTMLParser — вызвано запуском
  версия Beautiful Soup на Python 3 под Python 2.
• ImportError: No module named html.parser — вызвано запуском
  версия Beautiful Soup на Python 2 под Python 3.
• ImportError: No module named BeautifulSoup — вызвано запуском
  кода Beautiful Soup 3 в системе, где BS3
  не установлен. Или код писали на Beautiful Soup 4, не зная, что
  имя пакета сменилось на bs4 .
• ImportError: No module named bs4 — вызвано запуском
  кода Beautiful Soup 4 в системе, где BS4 не установлен.

Разбор XML¶

По умолчанию Beautiful Soup разбирает документы как HTML. Чтобы разобрать
документ в виде XML, передайте “xml” в качестве второго аргумента
в конструктор BeautifulSoup :

soup = BeautifulSoup(markup, "xml") 

Вам также нужно будет установить lxml .

Другие проблемы с парсерами¶

• Если ваш скрипт работает на одном компьютере, но не работает на другом, или работает в одной
  виртуальной среде, но не в другой, или работает вне виртуальной
  среды, но не внутри нее, это, вероятно, потому что в двух
  средах разные библиотеки парсеров. Например,
  вы могли разработать скрипт на компьютере с установленным lxml,
  а затем попытались запустить его на компьютере, где установлен только
  html5lib. Читайте в разделе Различия между парсерами, почему это
  важно, и исправляйте проблемы, указывая конкретную библиотеку парсера
  в конструкторе BeautifulSoup .
• Поскольку HTML-теги и атрибуты нечувствительны к регистру, все три HTML-
  парсера конвертируют имена тегов и атрибутов в нижний регистр. Таким образом,
  разметка преобразуется в . Если вы хотите
  сохранить смешанный или верхний регистр тегов и атрибутов, вам нужно
  разобрать документ как XML .

Прочие ошибки¶

• UnicodeEncodeError: ‘charmap’ codec can’t encode character
  u’xfoo’ in position bar (или практически любая другая ошибка
  UnicodeEncodeError ) — это не проблема с Beautiful Soup.
  Эта проблема проявляется в основном в двух ситуациях. Во-первых, когда вы пытаетесь
  вывести символ Unicode, который ваша консоль не может отобразить, потому что не знает, как.
  (Смотрите эту страницу в Python вики.) Во-вторых, когда
  вы пишете в файл и передаете символ Unicode, который
  не поддерживается вашей кодировкой по умолчанию. В этом случае самым простым
  решением будет явное кодирование строки Unicode в UTF-8 с помощью
  u.encode(«utf8») .
• KeyError: [attr] — вызывается при обращении к tag[‘attr’] , когда
  в искомом теге не определен атрибут attr . Наиболее
  типичны ошибки KeyError: ‘href’ и KeyError:
  ‘class’ . Используйте tag.get(‘attr’) , если вы не уверены, что attr
  определен — так же, как если бы вы работали со словарем Python.
• AttributeError: ‘ResultSet’ object has no attribute ‘foo’ — это
  обычно происходит тогда, когда вы ожидаете, что find_all() вернет
  один тег или строку. Но find_all() возвращает список тегов
  и строк в объекте ResultSet . Вам нужно перебрать
  список и поискать .foo в каждом из элементов. Или, если вам действительно
  нужен только один результат, используйте find() вместо
  find_all() .
• AttributeError: ‘NoneType’ object has no attribute ‘foo’ — это
  обычно происходит, когда вы вызываете find() и затем пытаетесь
  получить доступ к атрибуту .foo . Но в вашем случае
  find() не нашел ничего, поэтому вернул None вместо
  того, чтобы вернуть тег или строку. Вам нужно выяснить, почему
  find() ничего не возвращает.

Повышение производительности¶

Beautiful Soup никогда не будет таким же быстрым, как парсеры, на основе которых он
работает. Если время отклика критично, если вы платите за компьютерное время
по часам, или если есть какая-то другая причина, почему компьютерное время
важнее программистского, стоит забыть о Beautiful Soup
и работать непосредственно с lxml.

Тем не менее, есть вещи, которые вы можете сделать, чтобы ускорить Beautiful Soup. Если
вы не используете lxml в качестве основного парсера, самое время
начать . Beautiful Soup разбирает документы
значительно быстрее с lxml, чем с html.parser или html5lib.

Вы можете значительно ускорить распознавание кодировок, установив
библиотеку cchardet.

Разбор части документа не сэкономит много времени в процессе разбора,
но может сэкономить много памяти, что сделает
поиск по документу намного быстрее.

Beautiful Soup 3¶

Beautiful Soup 3 — предыдущая версия, и она больше
активно не развивается. На текущий момент Beautiful Soup 3 поставляется со всеми основными
дистрибутивами Linux:

$ apt-get install python-beautifulsoup

Он также публикуется через PyPi как BeautifulSoup :

$ pip install BeautifulSoup

Вы можете скачать tar-архив Beautiful Soup 3.2.0.

Если вы запустили easy_install beautifulsoup или easy_install
BeautifulSoup , но ваш код не работает, значит, вы ошибочно установили Beautiful
Soup 3. Вам нужно запустить easy_install beautifulsoup4 .

Архивная документация для Beautiful Soup 3 доступна онлайн.

Перенос кода на BS4¶

Большая часть кода, написанного для Beautiful Soup 3, будет работать и в Beautiful
Soup 4 с одной простой заменой. Все, что вам нужно сделать, это изменить
имя пакета c BeautifulSoup на bs4 . Так что это:

from BeautifulSoup import BeautifulSoup 

from bs4 import BeautifulSoup 

• Если выводится сообщение ImportError “No module named BeautifulSoup”, ваша
  проблема в том, что вы пытаетесь запустить код Beautiful Soup 3, в то время как
  у вас установлен Beautiful Soup 4.
• Если выводится сообщение ImportError “No module named bs4”, ваша проблема
  в том, что вы пытаетесь запустить код Beautiful Soup 4, в то время как
  у вас установлен Beautiful Soup 3.

Хотя BS4 в основном обратно совместим с BS3, большинство
методов BS3 устарели и получили новые имена, чтобы соответствовать PEP 8. Некоторые
из переименований и изменений нарушают обратную совместимость.

Вот что нужно знать, чтобы перейти с BS3 на BS4:

Вам нужен парсер¶

Beautiful Soup 3 использовал модуль Python SGMLParser , который теперь
устарел и был удален в Python 3.0. Beautiful Soup 4 по умолчанию использует
html.parser , но вы можете подключить lxml или html5lib
вместо него. Вы найдете таблицу сравнения парсеров в разделе Установка парсера.

Поскольку html.parser — это не то же, что SGMLParser , вы
можете обнаружить, что Beautiful Soup 4 дает другое дерево разбора, чем
Beautiful Soup 3. Если вы замените html.parser
на lxml или html5lib, может оказаться, что дерево разбора опять
изменилось. Если такое случится, вам придется обновить код,
чтобы разобраться с новым деревом.

Имена методов¶

• renderContents -> encode_contents
• replaceWith -> replace_with
• replaceWithChildren -> unwrap
• findAll -> find_all
• findAllNext -> find_all_next
• findAllPrevious -> find_all_previous
• findNext -> find_next
• findNextSibling -> find_next_sibling
• findNextSiblings -> find_next_siblings
• findParent -> find_parent
• findParents -> find_parents
• findPrevious -> find_previous
• findPreviousSibling -> find_previous_sibling
• findPreviousSiblings -> find_previous_siblings
• getText -> get_text
• nextSibling -> next_sibling
• previousSibling -> previous_sibling

Некоторые аргументы конструктора Beautiful Soup были переименованы по
той же причине:

• BeautifulSoup(parseOnlyThese=. ) -> BeautifulSoup(parse_only=. )
• BeautifulSoup(fromEncoding=. ) -> BeautifulSoup(from_encoding=. )

Я переименовал один метод для совместимости с Python 3:

• Tag.has_key() -> Tag.has_attr()

Я переименовал один атрибут, чтобы использовать более точную терминологию:

• Tag.isSelfClosing -> Tag.is_empty_element

Я переименовал три атрибута, чтобы избежать использования зарезервированных слов
в Python. В отличие от других, эти изменения не являются обратно
совместимыми. Если вы использовали эти атрибуты в BS3, ваш код не сработает
на BS4, пока вы их не измените.

• UnicodeDammit.unicode -> UnicodeDammit.unicode_markup
• Tag.next -> Tag.next_element
• Tag.previous -> Tag.previous_element

Генераторы¶

Я дал генераторам PEP 8-совместимые имена и преобразовал их в
свойства:

• childGenerator() -> children
• nextGenerator() -> next_elements
• nextSiblingGenerator() -> next_siblings
• previousGenerator() -> previous_elements
• previousSiblingGenerator() -> previous_siblings
• recursiveChildGenerator() -> descendants
• parentGenerator() -> parents

Так что вместо этого:

for parent in tag.parentGenerator(): . 

Вы можете написать это:

for parent in tag.parents: . 

(Хотя старый код тоже будет работать.)

Некоторые генераторы выдавали None после их завершения и
останавливались. Это была ошибка. Теперь генераторы просто останавливаются.

Добавились два генератора: .strings и
.stripped_strings .

.strings выдает
объекты NavigableString, а .stripped_strings выдает строки Python,
у которых удалены пробелы.

XML¶

Больше нет класса BeautifulStoneSoup для разбора XML. Чтобы
разобрать XML, нужно передать “xml” в качестве второго аргумента
в конструктор BeautifulSoup . По той же причине
конструктор BeautifulSoup больше не распознает
аргумент isHTML .

Улучшена обработка пустых тегов
XML. Ранее при разборе XML нужно было явно указать,
какие теги считать пустыми элементами. Аргумент SelfClosingTags
больше не распознается. Вместо этого
Beautiful Soup считает пустым элементом любой тег без содержимого. Если
вы добавляете в тег дочерний элемент, тег больше не считается
пустым элементом.

Мнемоники¶

Входящие мнемоники HTML или XML всегда преобразуются в
соответствующие символы Unicode. В Beautiful Soup 3 было несколько
перекрывающих друг друга способов взаимодействия с мнемониками. Эти способы
удалены. Конструктор BeautifulSoup больше не распознает
аргументы smartQuotesTo и convertEntities . (В Unicode,
Dammit все еще присутствует smart_quotes_to , но по умолчанию парные кавычки
преобразуются в Unicode). Константы HTML_ENTITIES ,
XML_ENTITIES и XHTML_ENTITIES были удалены, так как они
служили для настройки функции, которой больше нет (преобразование отдельных мнемоник в
символы Unicode).

Если вы хотите на выходе преобразовать символы Unicode обратно в мнемоники HTML,
а не превращать Unicode в символы UTF-8, вам нужно
использовать средства форматирования вывода .

Прочее¶

Tag.string теперь работает рекурсивно. Если тег А
содержит только тег B и ничего больше, тогда значение A.string будет таким же, как
B.string. (Раньше это был None.)

Многозначные атрибуты, такие как class , теперь в качестве значений имеют списки строк,
а не строки. Это может повлиять на поиск
по классу CSS.

Если вы передадите в один из методов find* одновременно string и
специфичный для тега аргумент, такой как name , Beautiful Soup будет
искать теги, которые, во-первых, соответствуют специфичным для тега критериям, и, во-вторых, имеют
Tag.string , соответствующий заданному вами значению string . Beautiful Soup не найдет сами строки. Ранее
Beautiful Soup игнорировал аргументы, специфичные для тегов, и искал
строки.

Конструктор BeautifulSoup больше не распознает
аргумент markupMassage . Теперь это задача парсера —
обрабатывать разметку правильно.

Редко используемые альтернативные классы парсеров, такие как
ICantBelieveItsBeautifulSoup и BeautifulSOAP ,
удалены. Теперь парсер решает, что делать с неоднозначной
разметкой.

Метод prettify() теперь возвращает строку Unicode, а не байтовую строку.

Перевод документации¶

Переводы документации Beautiful Soup очень
приветствуются. Перевод должен быть лицензирован по лицензии MIT,
так же, как сам Beautiful Soup и англоязычная документация к нему.

Есть два способа передать ваш перевод:

1. Создайте ветку репозитория Beautiful Soup, добавьте свой
   перевод и предложите слияние с основной веткой — так же,
   как вы предложили бы изменения исходного кода.
2. Отправьте в дискуссионную группу Beautiful Soup
   сообщение со ссылкой на
   ваш перевод, или приложите перевод к сообщению.

Используйте существующие переводы документации на китайский или португальский в качестве образца. В
частности, переводите исходный файл doc/source/index.rst вместо
того, чтобы переводить HTML-версию документации. Это позволяет
публиковать документацию в разных форматах, не
только в HTML.

Beautiful Soup is a Python library for pulling data out of HTML and XML files. It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work. The latest Version of Beautifulsoup is v4.9.3 as of now.

Prerequisites

How to install Beautifulsoup?

To install Beautifulsoup on Windows, Linux, or any operating system, one would need pip package. To check how to install pip on your operating system, check out – PIP Installation – Windows || Linux.
Now, run a simple command,

pip install beautifulsoup4

Wait and relax, Beautifulsoup would be installed shortly.

Install Beautifulsoup4 using Source code

One can install beautifulsoup, using source code directly, install beautifulsoup tarball from here – download the Beautiful Soup 4 source tarball
after downloading cd into the directory and run,

Python setup.py install

Verifying Installation

To check whether the installation is complete or not, let’s try implementing it using python

As BeautifulSoup is not a standard python library, we need to install it first. We are going to install the BeautifulSoup 4 library (also known as BS4), which is the latest one.

To isolate our working environment so as not to disturb the existing setup, let us first create a virtual environment.

Creating a virtual environment (optional)

A virtual environment allows us to create an isolated working copy of python for a specific project without affecting the outside setup.

Best way to install any python package machine is using pip, however, if pip is not installed already (you can check it using – “pip –version” in your command or shell prompt), you can install by giving below command −

Linux environment

$sudo apt-get install python-pip

Windows environment

To install pip in windows, do the following −

• Download the get-pip.py from https://bootstrap.pypa.io/get-pip.py or from the github to your computer.
• Open the command prompt and navigate to the folder containing get-pip.py file.
• Run the following command −

>python get-pip.py

That’s it, pip is now installed in your windows machine.

You can verify your pip installed by running below command −

>pip --version pip 19.2.3 from c:usersyadurappdatalocalprogramspythonpython37libsite-packagespip (python 3.7)

Installing virtual environment

Run the below command in your command prompt −

>pip install virtualenv

After running, you will see the below screenshot −

Below command will create a virtual environment (“myEnv”) in your current directory −

>virtualenv myEnv

Screenshot

To activate your virtual environment, run the following command −

>myEnvScriptsactivate

In the above screenshot, you can see we have “myEnv” as prefix which tells us that we are under virtual environment “myEnv”.

To come out of virtual environment, run deactivate.

(myEnv) C:Usersyadur>deactivate C:Usersyadur>

As our virtual environment is ready, now let us install beautifulsoup.

Installing BeautifulSoup

As BeautifulSoup is not a standard library, we need to install it. We are going to use the BeautifulSoup 4 package (known as bs4).

Linux Machine

To install bs4 on Debian or Ubuntu linux using system package manager, run the below command −

$sudo apt-get install python-bs4 (for python 2.x) $sudo apt-get install python3-bs4 (for python 3.x)

You can install bs4 using easy_install or pip (in case you find problem in installing using system packager).

$easy_install beautifulsoup4 $pip install beautifulsoup4

(You may need to use easy_install3 or pip3 respectively if you’re using python3)

Windows Machine

To install beautifulsoup4 in windows is very simple, especially if you have pip already installed.

>pip install beautifulsoup4

So now beautifulsoup4 is installed in our machine. Let us talk about some problems encountered after installation.

Problems after installation

On windows machine you might encounter, wrong version being installed error mainly through −

• error: ImportError “No module named HTMLParser”, then you must be running python 2 version of the code under Python 3.
• error: ImportError “No module named html.parser”error, then you must be running Python 3 version of the code under Python 2.

Best way to get out of above two situations is to re-install the BeautifulSoup again, completely removing existing installation.

If you get the SyntaxError “Invalid syntax” on the line ROOT_TAG_NAME = u’[document]’, then you need to convert the python 2 code to python 3, just by either installing the package −

$ python3 setup.py install

or by manually running python’s 2 to 3 conversion script on the bs4 directory −

$ 2to3-3.2 -w bs4

Installing a Parser

By default, Beautiful Soup supports the HTML parser included in Python’s standard library, however it also supports many external third party python parsers like lxml parser or html5lib parser.

To install lxml or html5lib parser, use the command −

Linux Machine

$apt-get install python-lxml $apt-get insall python-html5lib

Windows Machine

$pip install lxml $pip install html5lib

Generally, users use lxml for speed and it is recommended to use lxml or html5lib parser if you are using older version of python 2 (before 2.7.3 version) or python 3 (before 3.2.2) as python’s built-in HTML parser is not very good in handling older version.

Running Beautiful Soup

It is time to test our Beautiful Soup package in one of the html pages (taking web page – https://www.tutorialspoint.com/index.htm, you can choose any-other web page you want) and extract some information from it.

In the below code, we are trying to extract the title from the webpage −

from bs4 import BeautifulSoup import requests url = "https://www.tutorialspoint.com/index.htm" req = requests.get(url) soup = BeautifulSoup(req.text, "html.parser") print(soup.title)

Output

One common task is to extract all the URLs within a webpage. For that we just need to add the below line of code −

for link in soup.find_all('a'): print(link.get('href'))

Output

https://www.tutorialspoint.com/index.htm https://www.tutorialspoint.com/about/about_careers.htm https://www.tutorialspoint.com/questions/index.php https://www.tutorialspoint.com/online_dev_tools.htm https://www.tutorialspoint.com/codingground.htm https://www.tutorialspoint.com/current_affairs.htm https://www.tutorialspoint.com/upsc_ias_exams.htm https://www.tutorialspoint.com/tutor_connect/index.php https://www.tutorialspoint.com/whiteboard.htm https://www.tutorialspoint.com/netmeeting.php https://www.tutorialspoint.com/index.htm https://www.tutorialspoint.com/tutorialslibrary.htm https://www.tutorialspoint.com/videotutorials/index.php https://store.tutorialspoint.com https://www.tutorialspoint.com/gate_exams_tutorials.htm https://www.tutorialspoint.com/html_online_training/index.asp https://www.tutorialspoint.com/css_online_training/index.asp https://www.tutorialspoint.com/3d_animation_online_training/index.asp https://www.tutorialspoint.com/swift_4_online_training/index.asp https://www.tutorialspoint.com/blockchain_online_training/index.asp https://www.tutorialspoint.com/reactjs_online_training/index.asp https://www.tutorix.com https://www.tutorialspoint.com/videotutorials/top-courses.php https://www.tutorialspoint.com/the_full_stack_web_development/index.asp …. …. https://www.tutorialspoint.com/online_dev_tools.htm https://www.tutorialspoint.com/free_web_graphics.htm https://www.tutorialspoint.com/online_file_conversion.htm https://www.tutorialspoint.com/netmeeting.php https://www.tutorialspoint.com/free_online_whiteboard.htm http://www.tutorialspoint.com https://www.facebook.com/tutorialspointindia https://plus.google.com/u/0/+tutorialspoint Tweets by tutorialspoint http://www.linkedin.com/company/tutorialspoint https://www.youtube.com/channel/UCVLbzhxVTiTLiVKeGV7WEBg https://www.tutorialspoint.com/index.htm /about/about_privacy.htm#cookies /about/faq.htm /about/about_helping.htm /about/contact_us.htm

Similarly, we can extract useful information using beautifulsoup4.

Now let us understand more about “soup” in above example.

Beautifulsoup is a python package that is useful in scrapping web content from a particular URL. It parses HTML and XML documents and creates a parse tree that is used to extract data from HTML. But how you will install it in your system. In this entire tutorial you will know how to pip install Beautifulsoup, means how to install Beautifulsoup using the pip command.

Steps to Install Beautifulsoup using PIP

In this section, you will know all the steps required to install beautifulsoup in your system. I will tell you how to install in both windows and linux operating system.

Install Beautifulsoup on Windows

Step 1: Open your command prompt

Step 2: Check the version of the python by typing the following command.

python --version

Checking the version of python on windows

Step 3: Install the beautifulsoup using pip

After checking the version of the python now you can install beautifusoup for different python versions.

For python 3.xx

pip3 install beautifulsoup4

For python 2.xx

pip install beautifulsoup4

In my system, the python version is 3.xx. So I will use the pip3 command.

Installed beautiulsoup on windows

Install Beautifulsoup on Linux

Now let’s install Beautifulsoup on Linux. You have to follow the below steps to install it.

Step 1: Update the Linux

To update the Linux use the following command.

sudo apt-get update

Update you linux os

Step 2: Check the python version

To check the python version. If it is installed then you will get the version number. And if are getting the following error then you have to install it.

Python not found an error in Linux

To install python use the pip3 command.

pip3 install python

It will install the python 3.xx version.

Step 3: Install the Beautifulsoup

After the installation of the python install the Beautifulsoup using the pip command. Run the following bash command to install it.

pip3 install beautifulsoup4

It will successfully install the beautifulsoup on the Linux OS.

Conclusion

If you want to make your own scrapper then beautifulsoup python package is very useful. These are the steps to install beautifulsoup using the pip command for Linux and Window OS. Hope this tutorial has solved your queries. If you want to get more help then you can contact us anytime. We are always ready to help you. Apart from installation, there are so many operations you should explore with beautifulsoap package like extracting data using findall() function , parsing HTML and select.

Join our list

Subscribe to our mailing list and get interesting stuff and updates to your email inbox.

We respect your privacy and take protecting it seriously

Thank you for signup. A Confirmation Email has been sent to your Email Address.

Something went wrong.

Beautiful Soup Documentation

Beautiful Soup is a
Python library for pulling data out of HTML and XML files. It works
with your favorite parser to provide idiomatic ways of navigating,
searching, and modifying the parse tree. It commonly saves programmers
hours or days of work.

These instructions illustrate all major features of Beautiful Soup 4,
with examples. I show you what the library is good for, how it works,
how to use it, how to make it do what you want, and what to do when it
violates your expectations.

The examples in this documentation should work the same way in Python
2.7 and Python 3.2.

You might be looking for the documentation for Beautiful Soup 3.
If so, you should know that Beautiful Soup 3 is no longer being
developed, and that Beautiful Soup 4 is recommended for all new
projects. If you want to learn about the differences between Beautiful
Soup 3 and Beautiful Soup 4, see Porting code to BS4.

This documentation has been translated into other languages by its users.

• 이 문서는 한국어 번역도 가능합니다. (외부 링크)

Getting help

If you have questions about Beautiful Soup, or run into problems,
send mail to the discussion group. If
your problem involves parsing an HTML document, be sure to mention
:ref:`what the diagnose() function says ` about
that document.

Quick Start

Here’s an HTML document I’ll be using as an example throughout this
document. It’s part of a story from Alice in Wonderland:

html_doc = """  
The Dormouse's story
 
Once upon a time there were three little sisters; and their names were Elsie, Lacie and Tillie; and they lived at the bottom of a well.
 
.
 """

Running the «three sisters» document through Beautiful Soup gives us a
BeautifulSoup object, which represents the document as a nested
data structure:

from bs4 import BeautifulSoup soup = BeautifulSoup(html_doc) print(soup.prettify()) # # #  # # # 
# # The Dormouse's story # #
 # 
# Once upon a time there were three little sisters; and their names were # # Elsie # # , # # Lacie # # and # # Tillie # # ; and they lived at the bottom of a well. #
 # 
# . #
 # # 

Here are some simple ways to navigate that data structure:

soup.title #  soup.title.name # u'title' soup.title.string # u'The Dormouse's story' soup.title.parent.name # u'head' soup.p # 
The Dormouse's story
 soup.p['class'] # u'title' soup.a # Elsie soup.find_all('a') # [Elsie, # Lacie, # Tillie] soup.find(id="link3") # Tillie 

One common task is extracting all the URLs found within a page’s tags:

for link in soup.find_all('a'): print(link.get('href')) # http://example.com/elsie # http://example.com/lacie # http://example.com/tillie

Another common task is extracting all the text from a page:

print(soup.get_text()) # The Dormouse's story # # The Dormouse's story # # Once upon a time there were three little sisters; and their names were # Elsie, # Lacie and # Tillie; # and they lived at the bottom of a well. # # .

Does this look like what you need? If so, read on.

Installing Beautiful Soup

If you’re using a recent version of Debian or Ubuntu Linux, you can
install Beautiful Soup with the system package manager:

$ apt-get install python-bs4

Beautiful Soup 4 is published through PyPi, so if you can’t install it
with the system packager, you can install it with easy_install or
pip . The package name is beautifulsoup4 , and the same package
works on Python 2 and Python 3.

$ pip install beautifulsoup4

(The BeautifulSoup package is probably not what you want. That’s
the previous major release, Beautiful Soup 3. Lots of software uses
BS3, so it’s still available, but if you’re writing new code you
should install beautifulsoup4 .)

If you don’t have easy_install or pip installed, you can
download the Beautiful Soup 4 source tarball and
install it with setup.py .

$ python setup.py install

If all else fails, the license for Beautiful Soup allows you to
package the entire library with your application. You can download the
tarball, copy its bs4 directory into your application’s codebase,
and use Beautiful Soup without installing it at all.

I use Python 2.7 and Python 3.2 to develop Beautiful Soup, but it
should work with other recent versions.

Problems after installation

Beautiful Soup is packaged as Python 2 code. When you install it for
use with Python 3, it’s automatically converted to Python 3 code. If
you don’t install the package, the code won’t be converted. There have
also been reports on Windows machines of the wrong version being
installed.

If you get the ImportError «No module named HTMLParser», your
problem is that you’re running the Python 2 version of the code under
Python 3.

If you get the ImportError «No module named html.parser», your
problem is that you’re running the Python 3 version of the code under
Python 2.

In both cases, your best bet is to completely remove the Beautiful
Soup installation from your system (including any directory created
when you unzipped the tarball) and try the installation again.

If you get the SyntaxError «Invalid syntax» on the line
ROOT_TAG_NAME = u'[document]’ , you need to convert the Python 2
code to Python 3. You can do this either by installing the package:

$ python3 setup.py install

or by manually running Python’s 2to3 conversion script on the
bs4 directory:

Installing a parser

Beautiful Soup supports the HTML parser included in Python’s standard
library, but it also supports a number of third-party Python parsers.
One is the lxml parser. Depending on your setup,
you might install lxml with one of these commands:

$ apt-get install python-lxml

$ pip install lxml

Another alternative is the pure-Python html5lib parser, which parses HTML the way a
web browser does. Depending on your setup, you might install html5lib
with one of these commands:

$ apt-get install python-html5lib

$ pip install html5lib

This table summarizes the advantages and disadvantages of each parser library:

• Batteries included
• Decent speed
• Lenient (as of Python 2.7.3
  and 3.2.)

• Not very lenient
  (before Python 2.7.3
  or 3.2.2)

• Very fast
• Lenient

• External C dependency

• Very fast
• The only currently supported
  XML parser

• External C dependency

• Extremely lenient
• Parses pages the same way a
  web browser does
• Creates valid HTML5

• Very slow
• External Python
  dependency

If you can, I recommend you install and use lxml for speed. If you’re
using a version of Python 2 earlier than 2.7.3, or a version of Python
3 earlier than 3.2.2, it’s essential that you install lxml or
html5lib—Python’s built-in HTML parser is just not very good in older
versions.

Note that if a document is invalid, different parsers will generate
different Beautiful Soup trees for it. See Differences
between parsers for details.

Making the soup

To parse a document, pass it into the BeautifulSoup
constructor. You can pass in a string or an open filehandle:

from bs4 import BeautifulSoup soup = BeautifulSoup(open("index.html")) soup = BeautifulSoup("data")

First, the document is converted to Unicode, and HTML entities are
converted to Unicode characters:

BeautifulSoup("Sacré bleu!") Sacré bleu! 

Beautiful Soup then parses the document using the best available
parser. It will use an HTML parser unless you specifically tell it to
use an XML parser. (See Parsing XML.)

Kinds of objects

Beautiful Soup transforms a complex HTML document into a complex tree
of Python objects. But you’ll only ever have to deal with about four
kinds of objects.

Tag

A Tag object corresponds to an XML or HTML tag in the original document:

soup = BeautifulSoup(‘Extremely bold‘) tag = soup.b type(tag) #

Tags have a lot of attributes and methods, and I’ll cover most of them
in Navigating the tree and Searching the tree. For now, the most
important features of a tag are its name and attributes.

Name

Every tag has a name, accessible as .name :

tag.name # u'b'

If you change a tag’s name, the change will be reflected in any HTML
markup generated by Beautiful Soup:

tag.name = "blockquote" tag # 
Extremely bold
 

Attributes

A tag may have any number of attributes. The tag > has an attribute «class» whose value is
«boldest». You can access a tag’s attributes by treating the tag like
a dictionary:

tag['class'] # u'boldest'

You can access that dictionary directly as .attrs :

tag.attrs #

You can add, remove, and modify a tag’s attributes. Again, this is
done by treating the tag as a dictionary:

tag['class'] = 'verybold' tag['id'] = 1 tag # 
Extremely bold
 del tag['class'] del tag['id'] tag # 
Extremely bold
 tag['class'] # KeyError: 'class' print(tag.get('class')) # None

Multi-valued attributes

HTML 4 defines a few attributes that can have multiple values. HTML 5
removes a couple of them, but defines a few more. The most common
multi-valued attribute is class (that is, a tag can have more than
one CSS class). Others include rel , rev , accept-charset ,
headers , and accesskey . Beautiful Soup presents the value(s)
of a multi-valued attribute as a list:

css_soup = BeautifulSoup(' ') css_soup.p['class'] # ["body", "strikeout"] css_soup = BeautifulSoup(' ') css_soup.p['class'] # ["body"]

If an attribute looks like it has more than one value, but it’s not
a multi-valued attribute as defined by any version of the HTML
standard, Beautiful Soup will leave the attribute alone:

id_soup = BeautifulSoup(' ') id_soup.p['id'] # 'my id'

When you turn a tag back into a string, multiple attribute values are
consolidated:

rel_soup = BeautifulSoup('
Back to the homepage
') rel_soup.a['rel'] # ['index'] rel_soup.a['rel'] = ['index', 'contents'] print(rel_soup.p) # 
Back to the homepage
 

If you parse a document as XML, there are no multi-valued attributes:

xml_soup = BeautifulSoup(' ', 'xml') xml_soup.p['class'] # u'body strikeout'

NavigableString

A string corresponds to a bit of text within a tag. Beautiful Soup
uses the NavigableString class to contain these bits of text:

tag.string # u’Extremely bold’ type(tag.string) #

A NavigableString is just like a Python Unicode string, except
that it also supports some of the features described in Navigating
the tree and Searching the tree. You can convert a
NavigableString to a Unicode string with unicode() :

unicode_string = unicode(tag.string) unicode_string # u’Extremely bold’ type(unicode_string) #

You can’t edit a string in place, but you can replace one string with
another, using :ref:`replace_with` :

tag.string.replace_with("No longer bold") tag # 
No longer bold
 

NavigableString supports most of the features described in
Navigating the tree and Searching the tree, but not all of
them. In particular, since a string can’t contain anything (the way a
tag may contain a string or another tag), strings don’t support the
.contents or .string attributes, or the find() method.

If you want to use a NavigableString outside of Beautiful Soup,
you should call unicode() on it to turn it into a normal Python
Unicode string. If you don’t, your string will carry around a
reference to the entire Beautiful Soup parse tree, even when you’re
done using Beautiful Soup. This is a big waste of memory.

BeautifulSoup

The BeautifulSoup object itself represents the document as a
whole. For most purposes, you can treat it as a :ref:`Tag`
object. This means it supports most of the methods described in
Navigating the tree and Searching the tree.

Since the BeautifulSoup object doesn’t correspond to an actual
HTML or XML tag, it has no name and no attributes. But sometimes it’s
useful to look at its .name , so it’s been given the special
.name «[document]»:

soup.name # u'[document]'

Comments and other special strings

Tag , NavigableString , and BeautifulSoup cover almost
everything you’ll see in an HTML or XML file, but there are a few
leftover bits. The only one you’ll probably ever need to worry about
is the comment:

markup = » » soup = BeautifulSoup(markup) comment = soup.b.string type(comment) #

The Comment object is just a special type of NavigableString :

comment # u'Hey, buddy. Want to buy a used parser'

But when it appears as part of an HTML document, a Comment is
displayed with special formatting:

print(soup.b.prettify()) # # # 

Beautiful Soup defines classes for anything else that might show up in
an XML document: CData , ProcessingInstruction ,
Declaration , and Doctype . Just like Comment , these classes
are subclasses of NavigableString that add something extra to the
string. Here’s an example that replaces the comment with a CDATA
block:

from bs4 import CData cdata = CData("A CDATA block") comment.replace_with(cdata) print(soup.b.prettify()) # #  

html_doc = """  
The Dormouse's story
 
Once upon a time there were three little sisters; and their names were Elsie, Lacie and Tillie; and they lived at the bottom of a well.
 
.
 """ from bs4 import BeautifulSoup soup = BeautifulSoup(html_doc)

soup.head #  soup.title #  

soup.body.b # The Dormouse's story 

soup.find_all('a') # [Elsie, # Lacie, # Tillie]

head_tag = soup.head head_tag #  head_tag.contents [] title_tag = head_tag.contents[0] title_tag #  title_tag.contents # [u'The Dormouse's story']

len(soup.contents) # 1 soup.contents[0].name # u'html'

text = title_tag.contents[0] text.contents # AttributeError: 'NavigableString' object has no attribute 'contents'

for child in title_tag.children: print(child) # The Dormouse's story

head_tag.contents # []

for child in head_tag.descendants: print(child) #  # The Dormouse's story

len(list(soup.children)) # 1 len(list(soup.descendants)) # 25

title_tag.string # u'The Dormouse's story'

head_tag.contents # [] head_tag.string # u'The Dormouse's story'

print(soup.html.string) # None

for string in soup.strings: print(repr(string)) # u"The Dormouse's story" # u'nn' # u"The Dormouse's story" # u'nn' # u'Once upon a time there were three little sisters; and their names weren' # u'Elsie' # u',n' # u'Lacie' # u' andn' # u'Tillie' # u';nand they lived at the bottom of a well.' # u'nn' # u'. ' # u'n'

for string in soup.stripped_strings: print(repr(string)) # u"The Dormouse's story" # u"The Dormouse's story" # u'Once upon a time there were three little sisters; and their names were' # u'Elsie' # u',' # u'Lacie' # u'and' # u'Tillie' # u';nand they lived at the bottom of a well.' # u'. '

title_tag = soup.title title_tag #  title_tag.parent #  

title_tag.string.parent #  

print(soup.parent) # None

link = soup.a link # Elsie for parent in link.parents: if parent is None: print(parent) else: print(parent.name) # p # body # html # [document] # None

sibling_soup = BeautifulSoup("text1text2") print(sibling_soup.prettify()) # # # # # text1 # # # text2 # # # # 

The tag and the tag are at the same level: they’re both direct
children of the same tag. We call them siblings. When a document is
pretty-printed, siblings show up at the same indentation level. You
can also use this relationship in the code you write.

.next_sibling and .previous_sibling

You can use .next_sibling and .previous_sibling to navigate
between page elements that are on the same level of the parse tree:

sibling_soup.b.next_sibling # text2 sibling_soup.c.previous_sibling # text1 

The tag has a .next_sibling , but no .previous_sibling ,
because there’s nothing before the tag on the same level of the
tree. For the same reason, the tag has a .previous_sibling
but no .next_sibling :

print(sibling_soup.b.previous_sibling) # None print(sibling_soup.c.next_sibling) # None

The strings «text1» and «text2» are not siblings, because they don’t
have the same parent:

sibling_soup.b.string # u'text1' print(sibling_soup.b.string.next_sibling) # None

In real documents, the .next_sibling or .previous_sibling of a
tag will usually be a string containing whitespace. Going back to the
«three sisters» document:

Elsie Lacie Tillie 

link = soup.a link # Elsie link.next_sibling # u',n'

.next_siblings and .previous_siblings

You can iterate over a tag’s siblings with .next_siblings or
.previous_siblings :

for sibling in soup.a.next_siblings: print(repr(sibling)) # u',n' # Lacie # u' andn' # Tillie # u'; and they lived at the bottom of a well.' # None for sibling in soup.find(id="link3").previous_siblings: print(repr(sibling)) # ' andn' # Lacie # u',n' # Elsie # u'Once upon a time there were three little sisters; and their names weren' # None

Going back and forth

Take a look at the beginning of the «three sisters» document:

 
The Dormouse's story
 

An HTML parser takes this string of characters and turns it into a
series of events: «open an tag», «open a tag», «open a
tag», «add a string», «close the tag», «open a

tag», and so on. Beautiful Soup offers tools for reconstructing the
initial parse of the document.

.next_element and .previous_element

The .next_element attribute of a string or tag points to whatever
was parsed immediately afterwards. It might be the same as
.next_sibling , but it’s usually drastically different.

last_a_tag = soup.find("a", ) last_a_tag # Tillie last_a_tag.next_sibling # '; and they lived at the bottom of a well.'

last_a_tag.next_element # u'Tillie'

That’s because in the original markup, the word «Tillie» appeared
before that semicolon. The parser encountered an tag, then the
word «Tillie», then the closing tag, then the semicolon and rest of
the sentence. The semicolon is on the same level as the tag, but the
word «Tillie» was encountered first.

The .previous_element attribute is the exact opposite of
.next_element . It points to whatever element was parsed
immediately before this one:

.next_elements and .previous_elements

You should get the idea by now. You can use these iterators to move
forward or backward in the document as it was parsed:

for element in last_a_tag.next_elements: print(repr(element)) # u'Tillie' # u';nand they lived at the bottom of a well.' # u'nn' # 
.
 # u'. ' # u'n' # None

Searching the tree

Beautiful Soup defines a lot of methods for searching the parse tree,
but they’re all very similar. I’m going to spend a lot of time explaining
the two most popular methods: find() and find_all() . The other
methods take almost exactly the same arguments, so I’ll just cover
them briefly.

Once again, I’ll be using the «three sisters» document as an example:

html_doc = """  
The Dormouse's story
 
Once upon a time there were three little sisters; and their names were Elsie, Lacie and Tillie; and they lived at the bottom of a well.
 
.
 """ from bs4 import BeautifulSoup soup = BeautifulSoup(html_doc)

By passing in a filter to an argument like find_all() , you can
zoom in on the parts of the document you’re interested in.

Kinds of filters

Before talking in detail about find_all() and similar methods, I
want to show examples of different filters you can pass into these
methods. These filters show up again and again, throughout the
search API. You can use them to filter based on a tag’s name,
on its attributes, on the text of a string, or on some combination of
these.

A string

The simplest filter is a string. Pass a string to a search method and
Beautiful Soup will perform a match against that exact string. This
code finds all the tags in the document:

soup.find_all('b') # [The Dormouse's story]

If you pass in a byte string, Beautiful Soup will assume the string is
encoded as UTF-8. You can avoid this by passing in a Unicode string instead.

A regular expression

If you pass in a regular expression object, Beautiful Soup will filter
against that regular expression using its match() method. This code
finds all the tags whose names start with the letter «b»; in this
case, the tag and the tag:

import re for tag in soup.find_all(re.compile("^b")): print(tag.name) # body # b

This code finds all the tags whose names contain the letter ‘t’:

for tag in soup.find_all(re.compile("t")): print(tag.name) # html # title

A list

If you pass in a list, Beautiful Soup will allow a string match
against any item in that list. This code finds all the tags
and all the tags:

soup.find_all(["a", "b"]) # [The Dormouse's story, # Elsie, # Lacie, # Tillie]

True

The value True matches everything it can. This code finds all
the tags in the document, but none of the text strings:

for tag in soup.find_all(True): print(tag.name) # html # head # title # body # p # b # p # a # a # a # p

A function

If none of the other matches work for you, define a function that
takes an element as its only argument. The function should return
True if the argument matches, and False otherwise.

Here’s a function that returns True if a tag defines the «class»
attribute but doesn’t define the «id» attribute:

def has_class_but_no_id(tag): return tag.has_attr('class') and not tag.has_attr('id')

Pass this function into find_all() and you’ll pick up all the

tags:

soup.find_all(has_class_but_no_id) # [
The Dormouse's story
, # 
Once upon a time there were.
 , # 
.
 ]

This function only picks up the

tags. It doesn’t pick up the
tags, because those tags define both «class» and «id». It doesn’t pick
up tags like and , because those tags don’t define
«class».

Here’s a function that returns True if a tag is surrounded by
string objects:

from bs4 import NavigableString def surrounded_by_strings(tag): return (isinstance(tag.next_element, NavigableString) and isinstance(tag.previous_element, NavigableString)) for tag in soup.find_all(surrounded_by_strings): print tag.name # p # a # a # a # p

Now we’re ready to look at the search methods in detail.

find_all()

Signature: find_all( :ref:`name ` , :ref:`attrs ` , :ref:`recursive
` , :ref:`text ` , :ref:`limit ` , :ref:`**kwargs ` )

The find_all() method looks through a tag’s descendants and
retrieves all descendants that match your filters. I gave several
examples in Kinds of filters, but here are a few more:

soup.find_all("title") # [] soup.find_all("p", "title") # [
The Dormouse's story
] soup.find_all("a") # [Elsie, # Lacie, # Tillie] soup.find_all(id="link2") # [Lacie] import re soup.find(text=re.compile("sisters")) # u'Once upon a time there were three little sisters; and their names weren'

Some of these should look familiar, but others are new. What does it
mean to pass in a value for text , or id ? Why does
find_all(«p», «title») find a

tag with the CSS class «title»?
Let’s look at the arguments to find_all() .

The name argument

Pass in a value for name and you’ll tell Beautiful Soup to only
consider tags with certain names. Text strings will be ignored, as
will tags whose names that don’t match.

This is the simplest usage:

soup.find_all("title") # []

Recall from Kinds of filters that the value to name can be a
string, a regular expression, a list, a function, or the value
True.

The keyword arguments

Any argument that’s not recognized will be turned into a filter on one
of a tag’s attributes. If you pass in a value for an argument called id ,
Beautiful Soup will filter against each tag’s ‘id’ attribute:

If you pass in a value for href , Beautiful Soup will filter
against each tag’s ‘href’ attribute:

You can filter an attribute based on a string, a regular
expression, a list, a function, or the value True.

This code finds all tags whose id attribute has a value,
regardless of what the value is:

soup.find_all(id=True) # [Elsie, # Lacie, # Tillie]

You can filter multiple attributes at once by passing in more than one
keyword argument:

Some attributes, like the data-* attributes in HTML 5, have names that
can’t be used as the names of keyword arguments:

data_soup = BeautifulSoup('
foo!
') data_soup.find_all(data-foo="value") # SyntaxError: keyword can't be an expression

You can use these attributes in searches by putting them into a
dictionary and passing the dictionary into find_all() as the
attrs argument:

data_soup.find_all(attrs=) # [
foo!
]

Searching by CSS class

It’s very useful to search for a tag that has a certain CSS class, but
the name of the CSS attribute, «class», is a reserved word in
Python. Using class as a keyword argument will give you a syntax
error. As of Beautiful Soup 4.1.2, you can search by CSS class using
the keyword argument class_ :

soup.find_all("a", class_="sister") # [Elsie, # Lacie, # Tillie]

As with any keyword argument, you can pass class_ a string, a regular
expression, a function, or True :

soup.find_all(class_=re.compile("itl")) # [
The Dormouse's story
] def has_six_characters(css_class): return css_class is not None and len(css_class) == 6 soup.find_all(class_=has_six_characters) # [Elsie, # Lacie, # Tillie]

:ref:`Remember ` that a single tag can have multiple
values for its «class» attribute. When you search for a tag that
matches a certain CSS class, you’re matching against any of its CSS
classes:

css_soup = BeautifulSoup(' ') css_soup.find_all("p", class_="strikeout") # [ ] css_soup.find_all("p", class_="body") # [ ]

You can also search for the exact string value of the class attribute:

css_soup.find_all("p", class_="body strikeout") # [ ]

But searching for variants of the string value won’t work:

css_soup.find_all("p", class_="strikeout body") # []

If you want to search for tags that match two or more CSS classes, you
should use a CSS selector:

css_soup.select("p.strikeout.body") # [ ]

In older versions of Beautiful Soup, which don’t have the class_
shortcut, you can use the attrs trick mentioned above. Create a
dictionary whose value for «class» is the string (or regular
expression, or whatever) you want to search for:

soup.find_all("a", attrs=) # [Elsie, # Lacie, # Tillie]

The text argument

With text you can search for strings instead of tags. As with
name and the keyword arguments, you can pass in a string, a
regular expression, a list, a function, or the value True.
Here are some examples:

soup.find_all(text="Elsie") # [u'Elsie'] soup.find_all(text=["Tillie", "Elsie", "Lacie"]) # [u'Elsie', u'Lacie', u'Tillie'] soup.find_all(text=re.compile("Dormouse")) [u"The Dormouse's story", u"The Dormouse's story"] def is_the_only_string_within_a_tag(s): """Return True if this string is the only child of its parent tag.""" return (s == s.parent.string) soup.find_all(text=is_the_only_string_within_a_tag) # [u"The Dormouse's story", u"The Dormouse's story", u'Elsie', u'Lacie', u'Tillie', u'. ']

Although text is for finding strings, you can combine it with
arguments that find tags: Beautiful Soup will find all tags whose
.string matches your value for text . This code finds the
tags whose .string is «Elsie»:

The limit argument

find_all() returns all the tags and strings that match your
filters. This can take a while if the document is large. If you don’t
need all the results, you can pass in a number for limit . This
works just like the LIMIT keyword in SQL. It tells Beautiful Soup to
stop gathering results after it’s found a certain number.

There are three links in the «three sisters» document, but this code
only finds the first two:

soup.find_all("a", limit=2) # [Elsie, # Lacie]

The recursive argument

If you call mytag.find_all() , Beautiful Soup will examine all the
descendants of mytag : its children, its children’s children, and
so on. If you only want Beautiful Soup to consider direct children,
you can pass in recursive=False . See the difference here:

soup.html.find_all("title") # [] soup.html.find_all("title", recursive=False) # []

Here’s that part of the document:

     .

The tag is beneath the tag, but it’s not directly
beneath the tag: the tag is in the way. Beautiful Soup
finds the tag when it’s allowed to look at all descendants of
the tag, but when recursive=False restricts it to the
tag’s immediate children, it finds nothing.

Beautiful Soup offers a lot of tree-searching methods (covered below),
and they mostly take the same arguments as find_all() : name ,
attrs , text , limit , and the keyword arguments. But the
recursive argument is different: find_all() and find() are
the only methods that support it. Passing recursive=False into a
method like find_parents() wouldn’t be very useful.

Calling a tag is like calling find_all()

Because find_all() is the most popular method in the Beautiful
Soup search API, you can use a shortcut for it. If you treat the
BeautifulSoup object or a Tag object as though it were a
function, then it’s the same as calling find_all() on that
object. These two lines of code are equivalent:

soup.find_all("a") soup("a")

These two lines are also equivalent:

soup.title.find_all(text=True) soup.title(text=True)

find()

Signature: find( :ref:`name ` , :ref:`attrs ` , :ref:`recursive
` , :ref:`text ` , :ref:`**kwargs ` )

The find_all() method scans the entire document looking for
results, but sometimes you only want to find one result. If you know a
document only has one tag, it’s a waste of time to scan the
entire document looking for more. Rather than passing in limit=1
every time you call find_all , you can use the find()
method. These two lines of code are nearly equivalent:

soup.find_all('title', limit=1) # [] soup.find('title') #  

The only difference is that find_all() returns a list containing
the single result, and find() just returns the result.

If find_all() can’t find anything, it returns an empty list. If
find() can’t find anything, it returns None :

print(soup.find("nosuchtag")) # None

Remember the soup.head.title trick from Navigating using tag
names? That trick works by repeatedly calling find() :

soup.head.title #  soup.find("head").find("title") #  

find_parents() and find_parent()

Signature: find_parents( :ref:`name ` , :ref:`attrs ` , :ref:`text ` , :ref:`limit ` , :ref:`**kwargs ` )

Signature: find_parent( :ref:`name ` , :ref:`attrs ` , :ref:`text ` , :ref:`**kwargs ` )

I spent a lot of time above covering find_all() and
find() . The Beautiful Soup API defines ten other methods for
searching the tree, but don’t be afraid. Five of these methods are
basically the same as find_all() , and the other five are basically
the same as find() . The only differences are in what parts of the
tree they search.

First let’s consider find_parents() and
find_parent() . Remember that find_all() and find() work
their way down the tree, looking at tag’s descendants. These methods
do the opposite: they work their way up the tree, looking at a tag’s
(or a string’s) parents. Let’s try them out, starting from a string
buried deep in the «three daughters» document:

a_string = soup.find(text="Lacie") a_string # u'Lacie' a_string.find_parents("a") # [Lacie] a_string.find_parent("p") # 
Once upon a time there were three little sisters; and their names were # Elsie, # Lacie and # Tillie; # and they lived at the bottom of a well.
 a_string.find_parents("p", ) # []

You may have made the connection between find_parent() and
find_parents() , and the .parent and .parents attributes
mentioned earlier. The connection is very strong. These search methods
actually use .parents to iterate over all the parents, and check
each one against the provided filter to see if it matches.

find_next_siblings() and find_next_sibling()

Signature: find_next_siblings( :ref:`name ` , :ref:`attrs ` , :ref:`text ` , :ref:`limit ` , :ref:`**kwargs ` )

Signature: find_next_sibling( :ref:`name ` , :ref:`attrs ` , :ref:`text ` , :ref:`**kwargs ` )

These methods use :ref:`.next_siblings ` to
iterate over the rest of an element’s siblings in the tree. The
find_next_siblings() method returns all the siblings that match,
and find_next_sibling() only returns the first one:

first_link = soup.a first_link # Elsie first_link.find_next_siblings("a") # [Lacie, # Tillie] first_story_paragraph = soup.find("p", "story") first_story_paragraph.find_next_sibling("p") # 
.
 

find_previous_siblings() and find_previous_sibling()

Signature: find_previous_siblings( :ref:`name ` , :ref:`attrs ` , :ref:`text ` , :ref:`limit ` , :ref:`**kwargs ` )

Signature: find_previous_sibling( :ref:`name ` , :ref:`attrs ` , :ref:`text ` , :ref:`**kwargs ` )

These methods use :ref:`.previous_siblings ` to iterate over an element’s
siblings that precede it in the tree. The find_previous_siblings()
method returns all the siblings that match, and
find_previous_sibling() only returns the first one:

last_link = soup.find("a", ) last_link # Tillie last_link.find_previous_siblings("a") # [Lacie, # Elsie] first_story_paragraph = soup.find("p", "story") first_story_paragraph.find_previous_sibling("p") # 
The Dormouse's story
 

find_all_next() and find_next()

Signature: find_all_next( :ref:`name ` , :ref:`attrs ` , :ref:`text ` , :ref:`limit ` , :ref:`**kwargs ` )

Signature: find_next( :ref:`name ` , :ref:`attrs ` , :ref:`text ` , :ref:`**kwargs ` )

These methods use :ref:`.next_elements ` to
iterate over whatever tags and strings that come after it in the
document. The find_all_next() method returns all matches, and
find_next() only returns the first match:

first_link = soup.a first_link # Elsie first_link.find_all_next(text=True) # [u'Elsie', u',n', u'Lacie', u' andn', u'Tillie', # u';nand they lived at the bottom of a well.', u'nn', u'. ', u'n'] first_link.find_next("p") # 
.
 

find_all_previous() and find_previous()

Signature: find_all_previous( :ref:`name ` , :ref:`attrs ` , :ref:`text ` , :ref:`limit ` , :ref:`**kwargs ` )

Signature: find_previous( :ref:`name ` , :ref:`attrs ` , :ref:`text ` , :ref:`**kwargs ` )

These methods use :ref:`.previous_elements ` to
iterate over the tags and strings that came before it in the
document. The find_all_previous() method returns all matches, and
find_previous() only returns the first match:

first_link = soup.a first_link # Elsie first_link.find_all_previous("p") # [
Once upon a time there were three little sisters; .
 , # 
The Dormouse's story
] first_link.find_previous("title") #  

CSS selectors

Beautiful Soup supports the most commonly-used CSS selectors. Just pass a string into
the .select() method of a Tag object or the BeautifulSoup
object itself.

You can find tags:

soup.select("title") # [] soup.select("p nth-of-type(3)") # [
.
 ]

Find tags beneath other tags:

soup.select("body a") # [Elsie, # Lacie, # Tillie] soup.select("html head title") # []

Find tags directly beneath other tags:

soup.select("head > title") # [] soup.select("p > a") # [Elsie, # Lacie, # Tillie] soup.select("p > a:nth-of-type(2)") # [Lacie] soup.select("p > #link1") # [Elsie] soup.select("body > a") # []

Find the siblings of tags:

soup.select("#link1 ~ .sister") # [Lacie, # Tillie] soup.select("#link1 + .sister") # [Lacie]

Find tags by CSS class:

soup.select(".sister") # [Elsie, # Lacie, # Tillie] soup.select("[class~=sister]") # [Elsie, # Lacie, # Tillie]

Find tags by ID:

soup.select("#link1") # [Elsie] soup.select("a#link2") # [Lacie]

Test for the existence of an attribute:

soup.select('a[href]') # [Elsie, # Lacie, # Tillie]

Find tags by attribute value:

soup.select('a[href="http://example.com/elsie"]') # [Elsie] soup.select('a[href^="http://example.com/"]') # [Elsie, # Lacie, # Tillie] soup.select('a[href$="tillie"]') # [Tillie] soup.select('a[href*=".com/el"]') # [Elsie]

Match language codes:

multilingual_markup = """ 
Hello
 
Howdy, y'all
 
Pip-pip, old fruit
 
Bonjour mes amis
 """ multilingual_soup = BeautifulSoup(multilingual_markup) multilingual_soup.select('p[lang|=en]') # [
Hello
, # 
Howdy, y'all
, # 
Pip-pip, old fruit
]

This is a convenience for users who know the CSS selector syntax. You
can do all this stuff with the Beautiful Soup API. And if CSS
selectors are all you need, you might as well use lxml directly,
because it’s faster. But this lets you combine simple CSS selectors
with the Beautiful Soup API.

Modifying the tree

Beautiful Soup’s main strength is in searching the parse tree, but you
can also modify the tree and write your changes as a new HTML or XML
document.

Changing tag names and attributes

I covered this earlier, in Attributes, but it bears repeating. You
can rename a tag, change the values of its attributes, add new
attributes, and delete attributes:

soup = BeautifulSoup('Extremely bold') tag = soup.b tag.name = "blockquote" tag['class'] = 'verybold' tag['id'] = 1 tag # 
Extremely bold
 del tag['class'] del tag['id'] tag # 
Extremely bold
 

Modifying .string

If you set a tag’s .string attribute, the tag’s contents are
replaced with the string you give:

markup = 'I linked to example.com' soup = BeautifulSoup(markup) tag = soup.a tag.string = "New link text." tag # New link text. 

Be careful: if the tag contained other tags, they and all their
contents will be destroyed.

append()

You can add to a tag’s contents with Tag.append() . It works just
like calling .append() on a Python list:

soup = BeautifulSoup("Foo") soup.a.append("Bar") soup # FooBar soup.a.contents # [u'Foo', u'Bar']

BeautifulSoup.new_string() and .new_tag()

If you need to add a string to a document, no problem—you can pass a
Python string in to append() , or you can call the factory method
BeautifulSoup.new_string() :

soup = BeautifulSoup(" ") tag = soup.b tag.append("Hello") new_string = soup.new_string(" there") tag.append(new_string) tag # Hello there. tag.contents # [u'Hello', u' there']

If you want to create a comment or some other subclass of
NavigableString , pass that class as the second argument to
new_string() :

from bs4 import Comment new_comment = soup.new_string("Nice to see you.", Comment) tag.append(new_comment) tag # Hello there tag.contents # [u'Hello', u' there', u'Nice to see you.']

(This is a new feature in Beautiful Soup 4.2.1.)

What if you need to create a whole new tag? The best solution is to
call the factory method BeautifulSoup.new_tag() :

soup = BeautifulSoup(" ") original_tag = soup.b new_tag = soup.new_tag("a", href="http://www.example.com") original_tag.append(new_tag) original_tag #  new_tag.string = "Link text." original_tag # Link text. 

Only the first argument, the tag name, is required.

insert()

Tag.insert() is just like Tag.append() , except the new element
doesn’t necessarily go at the end of its parent’s
.contents . It’ll be inserted at whatever numeric position you
say. It works just like .insert() on a Python list:

markup = 'I linked to example.com' soup = BeautifulSoup(markup) tag = soup.a tag.insert(1, "but did not endorse ") tag # I linked to but did not endorse example.com tag.contents # [u'I linked to ', u'but did not endorse', example.com]

insert_before() and insert_after()

The insert_before() method inserts a tag or string immediately
before something else in the parse tree:

soup = BeautifulSoup("stop") tag = soup.new_tag("i") tag.string = "Don't" soup.b.string.insert_before(tag) soup.b # Don'tstop 

The insert_after() method moves a tag or string so that it
immediately follows something else in the parse tree:

soup.b.i.insert_after(soup.new_string(" ever ")) soup.b # Don't ever stop soup.b.contents # [Don't, u' ever ', u'stop']

clear()

Tag.clear() removes the contents of a tag:

markup = 'I linked to example.com' soup = BeautifulSoup(markup) tag = soup.a tag.clear() tag #  

extract()

PageElement.extract() removes a tag or string from the tree. It
returns the tag or string that was extracted:

markup = 'I linked to example.com' soup = BeautifulSoup(markup) a_tag = soup.a i_tag = soup.i.extract() a_tag # I linked to i_tag # example.com print(i_tag.parent) None

At this point you effectively have two parse trees: one rooted at the
BeautifulSoup object you used to parse the document, and one rooted
at the tag that was extracted. You can go on to call extract on
a child of the element you extracted:

my_string = i_tag.string.extract() my_string # u'example.com' print(my_string.parent) # None i_tag #

decompose()

Tag.decompose() removes a tag from the tree, then completely
destroys it and its contents:

markup = 'I linked to example.com' soup = BeautifulSoup(markup) a_tag = soup.a soup.i.decompose() a_tag # I linked to 

replace_with()

PageElement.replace_with() removes a tag or string from the tree,
and replaces it with the tag or string of your choice:

markup = 'I linked to example.com' soup = BeautifulSoup(markup) a_tag = soup.a new_tag = soup.new_tag("b") new_tag.string = "example.net" a_tag.i.replace_with(new_tag) a_tag # I linked to example.net 

replace_with() returns the tag or string that was replaced, so
that you can examine it or add it back to another part of the tree.

wrap()

PageElement.wrap() wraps an element in the tag you specify. It
returns the new wrapper:

soup = BeautifulSoup("
I wish I was bold.
") soup.p.string.wrap(soup.new_tag("b")) # I wish I was bold. soup.p.wrap(soup.new_tag("div") # 
I wish I was bold.
 

This method is new in Beautiful Soup 4.0.5.

unwrap()

Tag.unwrap() is the opposite of wrap() . It replaces a tag with
whatever’s inside that tag. It’s good for stripping out markup:

markup = 'I linked to example.com' soup = BeautifulSoup(markup) a_tag = soup.a a_tag.i.unwrap() a_tag # I linked to example.com 

Like replace_with() , unwrap() returns the tag
that was replaced.

Output

Pretty-printing

The prettify() method will turn a Beautiful Soup parse tree into a
nicely formatted Unicode string, with each HTML/XML tag on its own line:

markup = 'I linked to example.com' soup = BeautifulSoup(markup) soup.prettify() # 'n n n n n. ' print(soup.prettify()) # # # # # # I linked to # # example.com # # # # 

You can call prettify() on the top-level BeautifulSoup object,
or on any of its Tag objects:

Non-pretty printing

If you just want a string, with no fancy formatting, you can call
unicode() or str() on a BeautifulSoup object, or a Tag
within it:

str(soup) # ' I linked to example.com' unicode(soup.a) # u'I linked to example.com'

The str() function returns a string encoded in UTF-8. See
Encodings for other options.

You can also call encode() to get a bytestring, and decode()
to get Unicode.

Output formatters

If you give Beautiful Soup a document that contains HTML entities like
«&lquot;», they’ll be converted to Unicode characters:

soup = BeautifulSoup("“Dammit!” he said.") unicode(soup) # u' u201cDammit!u201d he said.'

If you then convert the document to a string, the Unicode characters
will be encoded as UTF-8. You won’t get the HTML entities back:

str(soup) # ' xe2x80x9cDammit!xe2x80x9d he said.'

By default, the only characters that are escaped upon output are bare
ampersands and angle brackets. These get turned into «&», «<»,
and «>», so that Beautiful Soup doesn’t inadvertently generate
invalid HTML or XML:

soup = BeautifulSoup("
The law firm of Dewey, Cheatem, & Howe
") soup.p # 
The law firm of Dewey, Cheatem, & Howe
 soup = BeautifulSoup('A link') soup.a # A link 

You can change this behavior by providing a value for the
formatter argument to prettify() , encode() , or
decode() . Beautiful Soup recognizes four possible values for
formatter .

The default is formatter=»minimal» . Strings will only be processed
enough to ensure that Beautiful Soup generates valid HTML/XML:

french = "
Il a dit <<Sacré bleu!>>
" soup = BeautifulSoup(french) print(soup.prettify(formatter="minimal")) # # # 
# Il a dit <<Sacré bleu!>> #
 # # 

If you pass in formatter=»html» , Beautiful Soup will convert
Unicode characters to HTML entities whenever possible:

print(soup.prettify(formatter="html")) # # # 
# Il a dit <<Sacré bleu!>> #
 # # 

If you pass in formatter=None , Beautiful Soup will not modify
strings at all on output. This is the fastest option, but it may lead
to Beautiful Soup generating invalid HTML/XML, as in these examples:

print(soup.prettify(formatter=None)) # # # 
# Il a dit > #
 # # link_soup = BeautifulSoup('A link') print(link_soup.a.encode(formatter=None)) # A link 

Finally, if you pass in a function for formatter , Beautiful Soup
will call that function once for every string and attribute value in
the document. You can do whatever you want in this function. Here’s a
formatter that converts strings to uppercase and does absolutely
nothing else:

def uppercase(str): return str.upper() print(soup.prettify(formatter=uppercase)) # # # 
# IL A DIT > #
 # # print(link_soup.a.prettify(formatter=uppercase)) # # A LINK # 

If you’re writing your own function, you should know about the
EntitySubstitution class in the bs4.dammit module. This class
implements Beautiful Soup’s standard formatters as class methods: the
«html» formatter is EntitySubstitution.substitute_html , and the
«minimal» formatter is EntitySubstitution.substitute_xml . You can
use these functions to simulate formatter=html or
formatter==minimal , but then do something extra.

Here’s an example that replaces Unicode characters with HTML entities
whenever possible, but also converts all strings to uppercase:

from bs4.dammit import EntitySubstitution def uppercase_and_substitute_html_entities(str): return EntitySubstitution.substitute_html(str.upper()) print(soup.prettify(formatter=uppercase_and_substitute_html_entities)) # # # 
# IL A DIT <<SACRÉ BLEU!>> #
 # # 

One last caveat: if you create a CData object, the text inside
that object is always presented exactly as it appears, with no
formatting. Beautiful Soup will call the formatter method, just in
case you’ve written a custom method that counts all the strings in the
document or something, but it will ignore the return value:

from bs4.element import CData soup = BeautifulSoup(" ") soup.a.string = CData("one < three") print(soup.a.prettify(formatter="xml")) # # 
get_text()
If you only want the text part of a document or tag, you can use the 
get_text() method. It returns all the text in a document or 
beneath a tag, as a single Unicode string:
markup = 'nI linked to example.comn' soup = BeautifulSoup(markup) soup.get_text() u'nI linked to example.comn' soup.i.get_text() u'example.com'
You can specify a string to be used to join the bits of text 
together:
# soup.get_text("|") u'nI linked to |example.com|n'
You can tell Beautiful Soup to strip whitespace from the beginning and 
end of each bit of text:
# soup.get_text("|", strip=True) u'I linked to|example.com'
But at that point you might want to use the :ref:`.stripped_strings ` 
generator instead, and process the text yourself:
[text for text in soup.stripped_strings] # [u'I linked to', u'example.com']
Specifying the parser to use
If you just need to parse some HTML, you can dump the markup into the 
BeautifulSoup constructor, and it’ll probably be fine. Beautiful 
Soup will pick a parser for you and parse the data. But there are a 
few additional arguments you can pass in to the constructor to change 
which parser is used.
The first argument to the BeautifulSoup constructor is a string or 
an open filehandle—the markup you want parsed. The second argument is 
how you’d like the markup parsed.
If you don’t specify anything, you’ll get the best HTML parser that’s 
installed. Beautiful Soup ranks lxml’s parser as being the best, then 
html5lib’s, then Python’s built-in parser. You can override this by 
specifying one of the following:

What type of markup you want to parse. Currently supported are 
«html», «xml», and «html5».
The name of the parser library you want to use. Currently supported 
options are «lxml», «html5lib», and «html.parser» (Python’s 
built-in HTML parser).

The section Installing a parser contrasts the supported parsers.
If you don’t have an appropriate parser installed, Beautiful Soup will 
ignore your request and pick a different parser. Right now, the only 
supported XML parser is lxml. If you don’t have lxml installed, asking 
for an XML parser won’t give you one, and asking for «lxml» won’t work 
either.
Differences between parsers
Beautiful Soup presents the same interface to a number of different 
parsers, but each parser is different. Different parsers will create 
different parse trees from the same document. The biggest differences 
are between the HTML parsers and the XML parsers. Here’s a short 
document, parsed as HTML:
BeautifulSoup("") #
Since an empty tag is not valid HTML, the parser turns it into a 
tag pair.
Here’s the same document parsed as XML (running this requires that you 
have lxml installed). Note that the empty tag is left alone, and 
that the document is given an XML declaration instead of being put 
into an tag.:
BeautifulSoup("", "xml") # #  
There are also differences between HTML parsers. If you give Beautiful 
Soup a perfectly-formed HTML document, these differences won’t 
matter. One parser will be faster than another, but they’ll all give 
you a data structure that looks exactly like the original HTML 
document.
But if the document is not perfectly-formed, different parsers will 
give different results. Here’s a short, invalid document parsed using 
lxml’s HTML parser. Note that the dangling
 tag is simply 
ignored:
BeautifulSoup(" ", "lxml") #
Here’s the same document parsed using html5lib:
BeautifulSoup(" ", "html5lib") #
Instead of ignoring the dangling
 tag, html5lib pairs it with an 
opening 
tag. This parser also adds an empty tag to the 
document.
Here’s the same document parsed with Python’s built-in HTML 
parser:
BeautifulSoup(" ", "html.parser") #
Like html5lib, this parser ignores the closing
 tag. Unlike 
html5lib, this parser makes no attempt to create a well-formed HTML 
document by adding a tag. Unlike lxml, it doesn’t even bother 
to add an tag.
Since the document « » is invalid, none of these techniques is 
the «correct» way to handle it. The html5lib parser uses techniques 
that are part of the HTML5 standard, so it has the best claim on being 
the «correct» way, but all three techniques are legitimate.
Differences between parsers can affect your script. If you’re planning 
on distributing your script to other people, or running it on multiple 
machines, you should specify a parser in the BeautifulSoup 
constructor. That will reduce the chances that your users parse a 
document differently from the way you parse it.
Encodings
Any HTML or XML document is written in a specific encoding like ASCII 
or UTF-8. But when you load that document into Beautiful Soup, you’ll 
discover it’s been converted to Unicode:
markup = "
Sacrxc3xa9 bleu!
" soup = BeautifulSoup(markup) soup.h1 # 
Sacré bleu!
 soup.h1.string # u'Sacrxe9 bleu!'
It’s not magic. (That sure would be nice.) Beautiful Soup uses a 
sub-library called Unicode, Dammit to detect a document’s encoding 
and convert it to Unicode. The autodetected encoding is available as 
the .original_encoding attribute of the BeautifulSoup object:
soup.original_encoding 'utf-8'
Unicode, Dammit guesses correctly most of the time, but sometimes it 
makes mistakes. Sometimes it guesses correctly, but only after a 
byte-by-byte search of the document that takes a very long time. If 
you happen to know a document’s encoding ahead of time, you can avoid 
mistakes and delays by passing it to the BeautifulSoup constructor 
as from_encoding .
Here’s a document written in ISO-8859-8. The document is so short that 
Unicode, Dammit can’t get a good lock on it, and misidentifies it as 
ISO-8859-7:
markup = b"
xedxe5xecxf9
" soup = BeautifulSoup(markup) soup.h1 
νεμω
 soup.original_encoding 'ISO-8859-7'
We can fix this by passing in the correct from_encoding :
soup = BeautifulSoup(markup, from_encoding="iso-8859-8") soup.h1 
םולש
 soup.original_encoding 'iso8859-8'
In rare cases (usually when a UTF-8 document contains text written in 
a completely different encoding), the only way to get Unicode may be 
to replace some characters with the special Unicode character 
«REPLACEMENT CHARACTER» (U+FFFD, �). If Unicode, Dammit needs to do 
this, it will set the .contains_replacement_characters attribute 
to True on the UnicodeDammit or BeautifulSoup object. This 
lets you know that the Unicode representation is not an exact 
representation of the original—some data was lost. If a document 
contains �, but .contains_replacement_characters is False , 
you’ll know that the � was there originally (as it is in this 
paragraph) and doesn’t stand in for missing data.
Output encoding
When you write out a document from Beautiful Soup, you get a UTF-8 
document, even if the document wasn’t in UTF-8 to begin with. Here’s a 
document written in the Latin-1 encoding:
markup = b'''    
Sacrxe9 bleu!
   ''' soup = BeautifulSoup(markup) print(soup.prettify()) # # # # # # 
# Sacré bleu! #
 # # 
Note that the tag has been rewritten to reflect the fact that 
the document is now in UTF-8.
If you don’t want UTF-8, you can pass an encoding into prettify() :
print(soup.prettify("latin-1")) # # # # .
You can also call encode() on the BeautifulSoup object, or any 
element in the soup, just as if it were a Python string:
soup.p.encode("latin-1") # '
Sacrxe9 bleu!
' soup.p.encode("utf-8") # '
Sacrxc3xa9 bleu!
'
Any characters that can’t be represented in your chosen encoding will 
be converted into numeric XML entity references. Here’s a document 
that includes the Unicode character SNOWMAN:
markup = u"N " snowman_soup = BeautifulSoup(markup) tag = snowman_soup.b
The SNOWMAN character can be part of a UTF-8 document (it looks like 
☃), but there’s no representation for that character in ISO-Latin-1 or 
ASCII, so it’s converted into «&#9731» for those encodings:
print(tag.encode("utf-8")) # ☃ print tag.encode("latin-1") # ☃ print tag.encode("ascii") # ☃ 
Unicode, Dammit
You can use Unicode, Dammit without using Beautiful Soup. It’s useful 
whenever you have data in an unknown encoding and you just want it to 
become Unicode:
from bs4 import UnicodeDammit dammit = UnicodeDammit("Sacrxc3xa9 bleu!") print(dammit.unicode_markup) # Sacré bleu! dammit.original_encoding # 'utf-8'
Unicode, Dammit’s guesses will get a lot more accurate if you install 
the chardet or cchardet Python libraries. The more data you 
give Unicode, Dammit, the more accurately it will guess. If you have 
your own suspicions as to what the encoding might be, you can pass 
them in as a list:
dammit = UnicodeDammit("Sacrxe9 bleu!", ["latin-1", "iso-8859-1"]) print(dammit.unicode_markup) # Sacré bleu! dammit.original_encoding # 'latin-1'
Unicode, Dammit has two special features that Beautiful Soup doesn’t 
use.
Smart quotes
You can use Unicode, Dammit to convert Microsoft smart quotes to HTML or XML 
entities:
markup = b"
I just x93lovex94 Microsoft Wordx92s smart quotes
" UnicodeDammit(markup, ["windows-1252"], smart_quotes_to="html").unicode_markup # u'
I just “love” Microsoft Word’s smart quotes
' UnicodeDammit(markup, ["windows-1252"], smart_quotes_to="xml").unicode_markup # u'
I just “love” Microsoft Word’s smart quotes
'
You can also convert Microsoft smart quotes to ASCII quotes:
UnicodeDammit(markup, ["windows-1252"], smart_quotes_to="ascii").unicode_markup # u'
I just "love" Microsoft Word's smart quotes
'
Hopefully you’ll find this feature useful, but Beautiful Soup doesn’t 
use it. Beautiful Soup prefers the default behavior, which is to 
convert Microsoft smart quotes to Unicode characters along with 
everything else:
UnicodeDammit(markup, ["windows-1252"]).unicode_markup # u'
I just u201cloveu201d Microsoft Wordu2019s smart quotes
'
Inconsistent encodings
Sometimes a document is mostly in UTF-8, but contains Windows-1252 
characters such as (again) Microsoft smart quotes. This can happen 
when a website includes data from multiple sources. You can use 
UnicodeDammit.detwingle() to turn such a document into pure 
UTF-8. Here’s a simple example:
snowmen = (u"N" * 3) quote = (u"NI like snowmen!N") doc = snowmen.encode("utf8") + quote.encode("windows_1252")
This document is a mess. The snowmen are in UTF-8 and the quotes are 
in Windows-1252. You can display the snowmen or the quotes, but not 
both:
print(doc) # ☃☃☃�I like snowmen!� print(doc.decode("windows-1252")) # ☃☃☃“I like snowmen!”
Decoding the document as UTF-8 raises a UnicodeDecodeError , and 
decoding it as Windows-1252 gives you gibberish. Fortunately, 
UnicodeDammit.detwingle() will convert the string to pure UTF-8, 
allowing you to decode it to Unicode and display the snowmen and quote 
marks simultaneously:
new_doc = UnicodeDammit.detwingle(doc) print(new_doc.decode("utf8")) # ☃☃☃“I like snowmen!”
UnicodeDammit.detwingle() only knows how to handle Windows-1252 
embedded in UTF-8 (or vice versa, I suppose), but this is the most 
common case.
Note that you must know to call UnicodeDammit.detwingle() on your 
data before passing it into BeautifulSoup or the UnicodeDammit 
constructor. Beautiful Soup assumes that a document has a single 
encoding, whatever it might be. If you pass it a document that 
contains both UTF-8 and Windows-1252, it’s likely to think the whole 
document is Windows-1252, and the document will come out looking like 
` ☃☃☃“I like snowmen!” ` .
UnicodeDammit.detwingle() is new in Beautiful Soup 4.1.0.
Parsing only part of a document
(Note that this feature won’t work if you’re using the html5lib parser. 
If you use html5lib, the whole document will be parsed, no 
matter what. This is because html5lib constantly rearranges the parse 
tree as it works, and if some part of the document didn’t actually 
make it into the parse tree, it’ll crash. To avoid confusion, in the 
examples below I’ll be forcing Beautiful Soup to use Python’s 
built-in parser.)
SoupStrainer
The SoupStrainer class takes the same arguments as a typical 
method from Searching the tree: :ref:`name ` , :ref:`attrs 
` , :ref:`text ` , and :ref:`**kwargs ` . Here are 
three SoupStrainer objects:
from bs4 import SoupStrainer only_a_tags = SoupStrainer("a") only_tags_with_id_link2 = SoupStrainer(id="link2") def is_short_string(string): return len(string) < 10 only_short_strings = SoupStrainer(text=is_short_string)
I’m going to bring back the «three sisters» document one more time, 
and we’ll see what the document looks like when it’s parsed with these 
three SoupStrainer objects:
html_doc = """  
The Dormouse's story
 
Once upon a time there were three little sisters; and their names were Elsie, Lacie and Tillie; and they lived at the bottom of a well.
 
.
 """ print(BeautifulSoup(html_doc, "html.parser", parse_only=only_a_tags).prettify()) # # Elsie # # # Lacie # # # Tillie # print(BeautifulSoup(html_doc, "html.parser", parse_only=only_tags_with_id_link2).prettify()) # # Lacie # print(BeautifulSoup(html_doc, "html.parser", parse_only=only_short_strings).prettify()) # Elsie # , # Lacie # and # Tillie # . #
You can also pass a SoupStrainer into any of the methods covered 
in Searching the tree. This probably isn’t terribly useful, but I 
thought I’d mention it:
soup = BeautifulSoup(html_doc) soup.find_all(only_short_strings) # [u'nn', u'nn', u'Elsie', u',n', u'Lacie', u' andn', u'Tillie', # u'nn', u'. ', u'n']
Troubleshooting
diagnose()
If you’re having trouble understanding what Beautiful Soup does to a 
document, pass the document into the diagnose() function. (New in 
Beautiful Soup 4.2.0.) Beautiful Soup will print out a report showing 
you how different parsers handle the document, and tell you if you’re 
missing a parser that Beautiful Soup could be using:
from bs4.diagnose import diagnose data = open("bad.html").read() diagnose(data) # Diagnostic running on Beautiful Soup 4.2.0 # Python version 2.7.3 (default, Aug 1 2012, 05:16:07) # I noticed that html5lib is not installed. Installing it may help. # Found lxml version 2.3.2.0 # # Trying to parse your data with html.parser # Here's what html.parser did with the document: # .
Just looking at the output of diagnose() may show you how to solve the 
problem. Even if not, you can paste the output of diagnose() when 
asking for help.
Errors when parsing a document
There are two different kinds of parse errors. There are crashes, 
where you feed a document to Beautiful Soup and it raises an 
exception, usually an HTMLParser.HTMLParseError . And there is 
unexpected behavior, where a Beautiful Soup parse tree looks a lot 
different than the document used to create it.
Almost none of these problems turn out to be problems with Beautiful 
Soup. This is not because Beautiful Soup is an amazingly well-written 
piece of software. It’s because Beautiful Soup doesn’t include any 
parsing code. Instead, it relies on external parsers. If one parser 
isn’t working on a certain document, the best solution is to try a 
different parser. See Installing a parser for details and a parser 
comparison.
The most common parse errors are HTMLParser.HTMLParseError: 
malformed start tag and HTMLParser.HTMLParseError: bad end 
tag . These are both generated by Python’s built-in HTML parser 
library, and the solution is to :ref:`install lxml or 
html5lib. `
The most common type of unexpected behavior is that you can’t find a 
tag that you know is in the document. You saw it going in, but 
find_all() returns [] or find() returns None . This is 
another common problem with Python’s built-in HTML parser, which 
sometimes skips tags it doesn’t understand. Again, the solution is to 
:ref:`install lxml or html5lib. `
Version mismatch problems

SyntaxError: Invalid syntax (on the line ROOT_TAG_NAME = 
u'[document]' ): Caused by running the Python 2 version of 
Beautiful Soup under Python 3, without converting the code.
ImportError: No module named HTMLParser — Caused by running the 
Python 2 version of Beautiful Soup under Python 3.
ImportError: No module named html.parser — Caused by running the 
Python 3 version of Beautiful Soup under Python 2.
ImportError: No module named BeautifulSoup — Caused by running 
Beautiful Soup 3 code on a system that doesn’t have BS3 
installed. Or, by writing Beautiful Soup 4 code without knowing that 
the package name has changed to bs4 .
ImportError: No module named bs4 — Caused by running Beautiful 
Soup 4 code on a system that doesn’t have BS4 installed.

Parsing XML
By default, Beautiful Soup parses documents as HTML. To parse a 
document as XML, pass in «xml» as the second argument to the 
BeautifulSoup constructor:
soup = BeautifulSoup(markup, "xml")
You’ll need to :ref:`have lxml installed ` .
Other parser problems

If your script works on one computer but not another, it’s probably 
because the two computers have different parser libraries 
available. For example, you may have developed the script on a 
computer that has lxml installed, and then tried to run it on a 
computer that only has html5lib installed. See Differences between 
parsers for why this matters, and fix the problem by mentioning a 
specific parser library in the BeautifulSoup constructor.
Because HTML tags and attributes are case-insensitive, all three HTML 
parsers convert tag and attribute names to lowercase. That is, the 
markup is converted to . If you want to 
preserve mixed-case or uppercase tags and attributes, you’ll need to 
:ref:`parse the document as XML. `

Miscellaneous

UnicodeEncodeError: 'charmap' codec can't encode character 
u'xfoo' in position bar (or just about any other 
UnicodeEncodeError ) — This is not a problem with Beautiful Soup. 
This problem shows up in two main situations. First, when you try to 
print a Unicode character that your console doesn’t know how to 
display. (See this page on the Python wiki for help.) Second, when 
you’re writing to a file and you pass in a Unicode character that’s 
not supported by your default encoding. In this case, the simplest 
solution is to explicitly encode the Unicode string into UTF-8 with 
u.encode("utf8") .
KeyError: [attr] — Caused by accessing tag['attr'] when the 
tag in question doesn’t define the attr attribute. The most 
common errors are KeyError: 'href' and KeyError: 
'class' . Use tag.get('attr') if you’re not sure attr is 
defined, just as you would with a Python dictionary.
AttributeError: 'ResultSet' object has no attribute 'foo' — This 
usually happens because you expected find_all() to return a 
single tag or string. But find_all() returns a _list_ of tags 
and strings—a ResultSet object. You need to iterate over the 
list and look at the .foo of each one. Or, if you really only 
want one result, you need to use find() instead of 
find_all() .
AttributeError: 'NoneType' object has no attribute 'foo' — This 
usually happens because you called find() and then tried to 
access the .foo` attribute of the result. But in your case, 
find() didn’t find anything, so it returned None , instead of 
returning a tag or a string. You need to figure out why your 
find() call isn’t returning anything.

Improving Performance
Beautiful Soup will never be as fast as the parsers it sits on top 
of. If response time is critical, if you’re paying for computer time 
by the hour, or if there’s any other reason why computer time is more 
valuable than programmer time, you should forget about Beautiful Soup 
and work directly atop lxml.
That said, there are things you can do to speed up Beautiful Soup. If 
you’re not using lxml as the underlying parser, my advice is to 
:ref:`start ` . Beautiful Soup parses documents 
significantly faster using lxml than using html.parser or html5lib.
You can speed up encoding detection significantly by installing the 
cchardet library.
Parsing only part of a document won’t save you much time parsing 
the document, but it can save a lot of memory, and it’ll make 
searching the document much faster.
Beautiful Soup 3
Beautiful Soup 3 is the previous release series, and is no longer 
being actively developed. It’s currently packaged with all major Linux 
distributions:
$ apt-get install python-beautifulsoup
It’s also published through PyPi as BeautifulSoup .:
$ pip install BeautifulSoup
You can also download a tarball of Beautiful Soup 3.2.0.
If you ran easy_install beautifulsoup or easy_install 
BeautifulSoup , but your code doesn’t work, you installed Beautiful 
Soup 3 by mistake. You need to run easy_install beautifulsoup4 .
The documentation for Beautiful Soup 3 is archived online. If 
your first language is Chinese, it might be easier for you to read 
the Chinese translation of the Beautiful Soup 3 documentation, 
then read this document to find out about the changes made in 
Beautiful Soup 4.
Porting code to BS4
Most code written against Beautiful Soup 3 will work against Beautiful 
Soup 4 with one simple change. All you should have to do is change the 
package name from BeautifulSoup to bs4 . So this:
from BeautifulSoup import BeautifulSoup
from bs4 import BeautifulSoup

If you get the ImportError «No module named BeautifulSoup», your 
problem is that you’re trying to run Beautiful Soup 3 code, but you 
only have Beautiful Soup 4 installed.
If you get the ImportError «No module named bs4», your problem 
is that you’re trying to run Beautiful Soup 4 code, but you only 
have Beautiful Soup 3 installed.

Although BS4 is mostly backwards-compatible with BS3, most of its 
methods have been deprecated and given new names for PEP 8 compliance. There are numerous other 
renames and changes, and a few of them break backwards compatibility.
Here’s what you’ll need to know to convert your BS3 code and habits to BS4:
You need a parser
Beautiful Soup 3 used Python’s SGMLParser , a module that was 
deprecated and removed in Python 3.0. Beautiful Soup 4 uses 
html.parser by default, but you can plug in lxml or html5lib and 
use that instead. See Installing a parser for a comparison.
Since html.parser is not the same parser as SGMLParser , it 
will treat invalid markup differently. Usually the «difference» is 
that html.parser crashes. In that case, you’ll need to install 
another parser. But sometimes html.parser just creates a different 
parse tree than SGMLParser would. If this happens, you may need to 
update your BS3 scraping code to deal with the new tree.
Method names

renderContents -> encode_contents
replaceWith -> replace_with
replaceWithChildren -> unwrap
findAll -> find_all
findAllNext -> find_all_next
findAllPrevious -> find_all_previous
findNext -> find_next
findNextSibling -> find_next_sibling
findNextSiblings -> find_next_siblings
findParent -> find_parent
findParents -> find_parents
findPrevious -> find_previous
findPreviousSibling -> find_previous_sibling
findPreviousSiblings -> find_previous_siblings
nextSibling -> next_sibling
previousSibling -> previous_sibling

Some arguments to the Beautiful Soup constructor were renamed for the 
same reasons:

BeautifulSoup(parseOnlyThese=. ) -> BeautifulSoup(parse_only=. )
BeautifulSoup(fromEncoding=. ) -> BeautifulSoup(from_encoding=. )

I renamed one method for compatibility with Python 3:

Tag.has_key() -> Tag.has_attr()

I renamed one attribute to use more accurate terminology:

Tag.isSelfClosing -> Tag.is_empty_element

I renamed three attributes to avoid using words that have special 
meaning to Python. Unlike the others, these changes are not backwards 
compatible. If you used these attributes in BS3, your code will break 
on BS4 until you change them.

UnicodeDammit.unicode -> UnicodeDammit.unicode_markup
Tag.next -> Tag.next_element
Tag.previous -> Tag.previous_element

Generators
I gave the generators PEP 8-compliant names, and transformed them into 
properties:

childGenerator() -> children
nextGenerator() -> next_elements
nextSiblingGenerator() -> next_siblings
previousGenerator() -> previous_elements
previousSiblingGenerator() -> previous_siblings
recursiveChildGenerator() -> descendants
parentGenerator() -> parents

So instead of this:
for parent in tag.parentGenerator(): .
You can write this:
for parent in tag.parents: .
(But the old code will still work.)
Some of the generators used to yield None after they were done, and 
then stop. That was a bug. Now the generators just stop.
There are two new generators, :ref:`.strings and 
.stripped_strings ` . .strings yields 
NavigableString objects, and .stripped_strings yields Python 
strings that have had whitespace stripped.
XML
There is no longer a BeautifulStoneSoup class for parsing XML. To 
parse XML you pass in «xml» as the second argument to the 
BeautifulSoup constructor. For the same reason, the 
BeautifulSoup constructor no longer recognizes the isHTML 
argument.
Beautiful Soup’s handling of empty-element XML tags has been 
improved. Previously when you parsed XML you had to explicitly say 
which tags were considered empty-element tags. The selfClosingTags 
argument to the constructor is no longer recognized. Instead, 
Beautiful Soup considers any empty tag to be an empty-element tag. If 
you add a child to an empty-element tag, it stops being an 
empty-element tag.
Entities
An incoming HTML or XML entity is always converted into the 
corresponding Unicode character. Beautiful Soup 3 had a number of 
overlapping ways of dealing with entities, which have been 
removed. The BeautifulSoup constructor no longer recognizes the 
smartQuotesTo or convertEntities arguments. (Unicode, 
Dammit still has smart_quotes_to , but its default is now to turn 
smart quotes into Unicode.) The constants HTML_ENTITIES , 
XML_ENTITIES , and XHTML_ENTITIES have been removed, since they 
configure a feature (transforming some but not all entities into 
Unicode characters) that no longer exists.
If you want to turn Unicode characters back into HTML entities on 
output, rather than turning them into UTF-8 characters, you need to 
use an :ref:`output formatter ` .
Miscellaneous
:ref:`Tag.string ` now operates recursively. If tag A 
contains a single tag B and nothing else, then A.string is the same as 
B.string. (Previously, it was None.)
Multi-valued attributes like class have lists of strings as 
their values, not strings. This may affect the way you search by CSS 
class.
If you pass one of the find* methods both :ref:`text ` and 
a tag-specific argument like :ref:`name ` , Beautiful Soup will 
search for tags that match your tag-specific criteria and whose 
:ref:`Tag.string ` matches your value for :ref:`text 
` . It will not find the strings themselves. Previously, 
Beautiful Soup ignored the tag-specific arguments and looked for 
strings.
The BeautifulSoup constructor no longer recognizes the 
markupMassage argument. It’s now the parser’s responsibility to 
handle markup correctly.
The rarely-used alternate parser classes like 
ICantBelieveItsBeautifulSoup and BeautifulSOAP have been 
removed. It’s now the parser’s decision how to handle ambiguous 
markup.
The prettify() method now returns a Unicode string, not a bytestring.
If you’re using a recent version of Debian or Ubuntu Linux, you can 
install Beautiful Soup with the system package manager:
$ apt-get install python-bs4
Beautiful Soup 4 is published through PyPi, so if you can’t install it 
with the system packager, you can install it with easy_install or 
pip . The package name is beautifulsoup4 , and the same package 
works on Python 2 and Python 3.
$ pip install beautifulsoup4
(The BeautifulSoup package is probably not what you want. That’s 
the previous major release, Beautiful Soup 3. 
Lots of software uses 
BS3, so it’s still available, but if you’re writing new code you 
should install beautifulsoup4 .)
If you don’t have easy_install or pip installed, you can 
download the Beautiful Soup 4 source tarball and 
install it with setup.py .
$ python setup.py install
If all else fails, the license for Beautiful Soup allows you to 
package the entire library with your application. You can download the 
tarball, copy its bs4 directory into your application’s codebase, 
and use Beautiful Soup without installing it at all.
I use Python 2.7 and Python 3.2 to develop Beautiful Soup, but it 
should work with other recent versions.
2.1. Problems after installation¶
Beautiful Soup is packaged as Python 2 code. When you install it for 
use with Python 3, it’s automatically converted to Python 3 code. If 
you don’t install the package, the code won’t be converted. There have 
also been reports on Windows machines of the wrong version being 
installed.
If you get the ImportError “No module named HTMLParser”, your 
problem is that you’re running the Python 2 version of the code under 
Python 3.
If you get the ImportError “No module named html.parser”, your 
problem is that you’re running the Python 3 version of the code under 
Python 2.
In both cases, your best bet is to completely remove the Beautiful 
Soup installation from your system (including any directory created 
when you unzipped the tarball) and try the installation again.
If you get the SyntaxError “Invalid syntax” on the line 
ROOT_TAG_NAME = u'[document]' , you need to convert the Python 2 
code to Python 3. You can do this either by installing the package:
$ python3 setup.py install
or by manually running Python’s 2to3 conversion script on the 
bs4 directory:
2.2. Installing a parser¶
Beautiful Soup supports the HTML parser included in Python’s standard 
library, but it also supports a number of third-party Python parsers. 
One is the lxml parser. Depending on your setup, 
you might install lxml with one of these commands:
$ apt-get install python-lxml
$ pip install lxml
Another alternative is the pure-Python html5lib parser, which parses HTML the way a 
web browser does. Depending on your setup, you might install html5lib 
with one of these commands:
$ apt-get install python-html5lib
$ pip install html5lib
This table summarizes the advantages and disadvantages of each parser library:

Batteries included
Decent speed
Lenient (as of Python 2.7.3 
and 3.2.)


Not very lenient 
(before Python 2.7.3 
or 3.2.2)


Very fast
Lenient


External C dependency


Very fast
The only currently supported 
XML parser


External C dependency


Extremely lenient
Parses pages the same way a 
web browser does
Creates valid HTML5


Very slow
External Python 
dependency

If you can, I recommend you install and use lxml for speed. If you’re 
using a version of Python 2 earlier than 2.7.3, or a version of Python 
3 earlier than 3.2.2, it’s essential that you install lxml or 
html5lib–Python’s built-in HTML parser is just not very good in older 
versions.
Note that if a document is invalid, different parsers will generate 
different Beautiful Soup trees for it. See Differences between parsers for details.
Summary: To install BeautifulSoup in WIndows use the command: pip install beautifulsoup4 . To install it in Linux use the command: sudo apt-get install python3-bs4 .
 Aim : In this tutorial we will discuss how to to install BeautifulSoup?
Since BeautifulSoup is not a Python standard library we need to install it before we can use it to scrape websites. Hence, we will have a look at the steps to install the BeautifulSoup 4 package (also called bs4) and also discuss some of the problems that come up after the installation.
Note: The current release is Beautiful Soup 4.9.3 (October 3, 2020).
❂ Installing BeautifulSoup In Linux Machine
❖ Platform: Debian or Ubuntu
If you are using Python in a Debian or Ubuntu-based OS, you need to install Beautiful Soup with the help of the system package manager using the following command:
✻ For Python 2.x
$sudo apt-get install python-bs4
✻ For Python 3.x
$sudo apt-get install python3-bs4
In case you do not want to use the system package manager, you can use easy_install or pip to install bs4.
✻ Command to install bs4 using easy_install:
$easy_install beautifulsoup4
✻ Command to install bs4 using pip:
$pip install beautifulsoup4
# Note: If you are using Python3, you might need to install easy_install3 or pip3 respectively before you can use them.
❖ Platform: Windows
Installing bs4 in Windows is a one step process and is very easy. Use the following command to install it using your command line interface.
pip install beautifulsoup4
❂ Some Common Problems After Installation
You might encounter an error if there’s a wrong version being installed. Let us have a look at the reason behind the errors!
❖ Error:  ImportError “No module named HTMLParser” 
Reason: The error occurs because you are using Python 2 version of the code in Python 3.
❖ Error:  ImportError “No module named html.parser” 
Reason: The error occurs because you are using Python 3 version of the code in Python 2.
Solution: Remove existing installation and re-install BeautifulSoup.
❖ Error: SyntaxError “Invalid syntax” on the line ROOT_TAG_NAME = u’[document]’
Solution: Convert Python 2 version of the code to Python 3 by either:

Installing the package −

python3 setup.py install


2to3-3.2 -w bs4

❂ Installing The Parser
Beautiful Soup supports the HTML parser by defaultwhich is included in the Python’s standard library. However it also supports other external or third party Python parsers as shown in the table below:



Parser
Typical usage
Advantages
Disadvantages


Python’s html.parser
BeautifulSoup(markup, "html.parser")
Batteries includedDecent speedLenient (As of Python 2.7.3 and 3.2.)
Not as fast as lxml, less lenient than html5lib.


lxml’s HTML parser
BeautifulSoup(markup, "lxml")
Very fastLenient
External C dependency


lxml’s XML parser
BeautifulSoup(markup, "lxml-xml") BeautifulSoup(markup, "xml")
Very fastThe only currently supported XML parser
External C dependency


html5lib
BeautifulSoup(markup, "html5lib")
Extremely lenientParses pages the same way a web browser doesCreates valid HTML5
Very slowExternal Python dependency



Use the following commands to install the lxml or the html5lib parser,
Linux:
$apt-get install python-lxml $apt-get insall python-html5lib
Windows:
$pip install lxml $pip install html5lib
Conclusion
With that, we come to the end of this crisp tutorial on how to install the BeautifulSoup library. Please feel free to follow the steps and install it in your system. If you want to learn how to use the BeautifulSoup library and scrape a webpage, please follow this TUTORIAL and have a look at the step by step guide to scrape your webpage.
Please subscribe and stay tuned for more interesting articles!
Where to Go From Here?
Enough theory. Let’s get some practice!
Coders get paid six figures and more because they can solve problems more effectively using machine intelligence and automation.
To become more successful in coding, solve more real problems for real people. That’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?
You build high-value coding skills by working on practical coding projects!
Do you want to stop learning with toy projects and focus on practical code projects that earn you money and solve real problems for people?
🚀 If your answer is YES!, consider becoming a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.
If you just want to learn about the freelancing opportunity, feel free to watch my free webinar “How to Build Your High-Income Skill Python” and learn how I grew my coding business online and how you can, too—from the comfort of your own home.
Join the free webinar now!
I am a professional Python Blogger and Content creator. I have published numerous articles and created courses over a period of time. Presently I am working as a full-time freelancer and I have experience in domains like Python, AWS, DevOps, and Networking.
You can contact me @:
Автор оригинала: Shubham Sayon.
Резюме: Чтобы установить BeautifulSoup в Windows Используйте команду: PIP Установите BeautifulSoup4 Отказ Чтобы установить его в Linux, используйте команду: sudo apt-get install python3-bs4 Отказ
Цель : В этом руководстве мы обсудим, как установить BeautifulSoup ?
Поскольку BeautifulSoup не является стандартной библиотекой Python, нам нужно установить его, прежде чем мы сможем использовать его, чтобы соскрести сайты. Следовательно, мы посмотрим на шаги для установки пакета Boysuous 4 (также называемого BS4), а также обсуждают некоторые из проблем, которые придумывают после установки.
Примечание: Текущий релиз это Красивый суп 4.9.3 (3 октября 2020 года).
❂ Установка красивыхsoup В машине Linux
❖ Платформа: Debian или Ubuntu
Если вы используете Python в ОС на основе Debian или Ubuntu, вам необходимо установить красивый суп с помощью менеджера системного пакета, используя следующую команду:
✻ для Python 2.x.
$sudo apt-get install python-bs4
✻ для Python 3.x.
$sudo apt-get install python3-bs4
Если вы не хотите использовать диспетчер пакетов системы, вы можете использовать easy_install или пипс установить BS4.
✻ Команда для установки BS4 Использование easy_install :
$easy_install beautifulsoup4
✻ Команда для установки BS4 Использование Пип :
$pip install beautifulsoup4
# Примечание: Если вы используете Python3, вам может потребоваться установить easy_install3. или PIP3 соответственно, прежде чем вы сможете их использовать.
❖ Платформа: Windows
Установка BS4 В Windows является одним из шагов и очень прост. Используйте следующую команду, чтобы установить его, используя интерфейс командной строки.
pip install beautifulsoup4
❂ Некоторые распространенные проблемы после установки
Вы можете столкнуться с ошибкой, если установлена неправильная версия. Давайте посмотрим на причину ошибок!
❖ Ошибка : ImportError «Нет модуля по имени HTMLParser»
Причина: Ошибка возникает, потому что вы используете версию Python 2 в Python 3.
❖ Ошибка : ImportError «Нет модуля по имени HTML.Parser»
Причина: Ошибка возникает, потому что вы используете версию Python 3 в Python 2.
Решение: Удалите существующую установку и переустановите BeautifulSoup.
❖ Ошибка .: SyntaxError “Неверный синтаксис” на линии «[документ]»
Решение: Конвертировать Python 2 версии кода в Python 3 с помощью:

Установка пакета –

python3 setup.py. установить


2To3-3,2 -W BS4

❂ Установка парсера
Красивый суп поддерживает Parser HTML по умолчанию, которое включено в стандартную библиотеку Python. Однако он также поддерживает другую внешнюю или стороннюю парсеров Python, как показано в таблице ниже:



Анализатор
Типичное использование
Преимущества
Недостатки


Python HTML.Parser.
BeautifulSoupsup (Markup, «HTML.Parser»)
Аккумуляторы в комплекте удерживают Speedlenient (как на Python 2.7.3 и 3.2.)
Не так быстро, как lxml, менее снисходительно, чем html5lib.


HTML Parser LXML
CountrySoup (разметки, “lxml”)
Очень Fastlenient
Внешняя зависимость


XML Parser LXML
Beautifulsoup (Markup, «LXML-XML») BeautifulSoup (Markup, «XML»)
Очень быстро Только в настоящее время поддерживается XML Parser
Внешняя зависимость


HTML5LIB
BeautifulSoup (Markup, «HTML5LIB»)
Чрезвычайно lenientparses страниц так же, как веб-браузер делает Valid HTML5
Очень замедленная зависимость Python



Используйте следующие команды для установки lxml или HTML5LIB парсер,
Linux:
$apt-get install python-lxml $apt-get insall python-html5lib
Windows:
$pip install lxml $pip install html5lib
Заключение
С этим мы дойдем до конца этого хрустящего урока о том, как установить Beautifulsoup библиотека. Пожалуйста, не стесняйтесь следить за шагами и установить его в свою систему. Если вы хотите узнать, как использовать библиотеку BeautifulSoup и Scrape веб-страницу, пожалуйста, следуйте за Это руководство И посмотрите на шаг за шагом руководство, чтобы соскрести свою веб-страницу.
Пожалуйста, подпишитесь и оставайтесь настроенными для более интересных статей!
Куда пойти отсюда?
Достаточно теории, давайте познакомимся!
Чтобы стать успешным в кодировке, вам нужно выйти туда и решать реальные проблемы для реальных людей. Вот как вы можете легко стать шестифункциональным тренером. И вот как вы польские навыки, которые вам действительно нужны на практике. В конце концов, что такое использование теории обучения, что никто никогда не нуждается?
Практические проекты – это то, как вы обостряете вашу пилу в кодировке!
Вы хотите стать мастером кода, сосредоточившись на практических кодовых проектах, которые фактически зарабатывают вам деньги и решают проблемы для людей?
Затем станьте питоном независимым разработчиком! Это лучший способ приближения к задаче улучшения ваших навыков Python – даже если вы являетесь полным новичком.
Присоединяйтесь к моему бесплатным вебинаре «Как создать свой навык высокого дохода Python» и посмотреть, как я вырос на моем кодированном бизнесе в Интернете и как вы можете, слишком от комфорта вашего собственного дома.
Присоединяйтесь к свободному вебинару сейчас!
Я профессиональный Python Blogger и Content Creator. Я опубликовал многочисленные статьи и создал курсы в течение определенного периода времени. В настоящее время я работаю полный рабочий день, и у меня есть опыт в областях, таких как Python, AWS, DevOps и Networking.
Вы можете связаться со мной @: