person holding silver bitcoin coin
Photo by Crypto Crow on Pexels.com – Cryptocurrency Prices Without an API

The Task – Cryptocurrency Web Scraping with Python

Cryptocurrency Web Scraping with Python is more complicated than using an API but has several advantages. Most cryptocurrency APIs have very limited free versions and expensive paid versions. Paid APIs such as CoinGecko, Binance, and CoinAPI are great for full-fledged web applications. However, what if we need to get prices for a personal project or small-scale solution? A good example is building a neural network of historical price data of multiple coins. How could we accomplish this?

An easy solution is using web scraping through the Requests and BeautifulSoup Python libraries. This requires a bit more code than hitting an endpoint. Although, this will enable us to obtain cryptocurrency prices with Python web scraping. Let’s get started.

Prerequisites

Before we begin, make sure you have at least some experience with the Requests and BeautifulSoup python libraries. These are fairly easy libraries to learn and require at least basic knowledge of HTML, HTTP requests, and request headers. Now let’s get into the code.

The Code

The cryptocurrency data we are going to scrape comes from Yahoo Finance. I selected this data source because the site includes easy-to-read tables containing price data. In order to accomplish this, we can request the webpage and loop through the table element with BeautifulSoup. First, let’s start with the request code by using the example of Bitcoin:

import requests
from bs4 import BeautifulSoup

table_dict = {'date': [], 'open': [], 'high': [], 'low': [], 'close': [], 'adj close': [], 'volume': []}

keys = list(table_dict.keys())

headers = headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'}
web = requests.get('https://finance.yahoo.com/quote/BTC-USD/history', headers=headers).text

The above defines our libraries, sets up our data structure, includes our headers (for bypassing a robot check), and our request object. Next, let’s implement our beautiful soup code:

soup = BeautifulSoup(web, 'html.parser')

table = soup.find('tbody')

rows = table.find_all('tr')

print(rows)

Here, we are simply creating our soup object, finding the data table, and its rows. This prepares us for the below loop code:

for row in rows:
    
    for c in range(0, len(row.find_all('td'))):
        if row.find_all('td')[c].text == '-':
            table_dict[keys[c]].append('0')
        else:
            table_dict[keys[c]].append(row.find_all('td')[c].text)
        
        
print(table_dict)

This is a two-dimensional loop that accesses each individual cell within each row for the table. This code automatically adds the column data to the mapped dictionary keys. You could additionally go the extra step and make sure the dates and numbers are in the proper format. Important note: all HTML data is returned as string data. Use Python’s native object conversion methods to get numbers as well as dates from the strings. Here’s the bitcoin historical data in JSON format:

Cryptocurrency Web Scraping with Python – Result

Now you know how to accomplish cryptocurrency web scraping with Python. Need even more help? Contact me here -> https://codedogtutoring.com/connect/