Web Scrapping

Introduction

Web Scrapping is data scrapping used to extract data from websites. It’s an automated process of extracting data from different sites or from single sites. Web Scrapping involves 2 main things:

1) Fetching:

· It is downloading a page and use when needed to extract data from it. It’s just like when we use a browser it downloads web pages.

2) Extracting:

· Extracting is like fetching or getting data from web pages.

Web Scrapping is used for:

1) Scrapping data from the site

2) Web and Data mining and many more

Let’s get our hands dirty….

First things first, let’s import libraries we need

re: for regular expression
requests: for sending HTTP requests
Counter: for counting
BeautifulSoup: main library for extracting data.

Now, let’s aim at our target website. And fetch data from that website.

URL: stores link for IMDB site
response: fetches a page from site
soup: Contains all contents of that page. Which can later use to extract data.

Now, let’s extract data.

crew: Here, we used comprehensive list to iterate over <table> <td> attribute with class name titleColumn to extract crew.

Title: Here, we used comprehensive list to iterate over <table> <td> attribute with class name posterColumn to extract title.

Ratings: Here, we used comprehensive list to iterate over <table> <td> attribute to extract ratings.

Year: To store year attribute stored in <span> with class name secondaryInfo.

This code block, zip all our variables and bind them into a dictionary and then append it into a list. This makes our list filled with our data.

This function helps to identify the year which most number of release in order to identify the year with the great movie releases and display it.

This code block helps to see which director has the most number of releases or more successful films.

So this was all about Web Scrapping and a small project of web scrapping.

Related posts:

This step-by-step guide/tutorial will show you how to create custom buttons that replace the default variant options on your Shopify product page and link to different product pages for each variant…

Database modeling is the process of making a data model for data that will be stored inside the database. Data modeling helps with enforcing business rules, government policies, and regulatory…

La venta de inmuebles en la ciudad de Buenos Aires había comenzado el año con un envión por el dólar barato y la posibilidad de sacar un crédito hipotecario. Pero este empuje cambió de rumbo y se…