The Power of the Cloud and Unsupervised Learning

Background

This report is generated of what cryptocurrencies are available on the trading market and how they can be grouped using classification. I have used unsupervivsed machine learning and Amazon SageMaker by clustering cryptocurrencies and creating plots to present the results.

The following tasks have been completed:

1. Data Preprocessing: Prepare data for dimension reduction with PCA and clustering using K-Means.
2. Reducing Data Dimensions Using PCA: Reduce data dimension using the PCA algorithm from sklearn.
3. Clustering Cryptocurrencies Using K-Means: Predict clusters using the cryptocurrencies data using the KMeans algorithm from sklearn.
4. Visualizing Results: Create some plots and data tables to present the results.
5. Challenge: Deploy notebook to Amazon SageMaker.

---

Instructions

Data Preprocessing

In this section, you will load the information about cryptocurrencies and perform data preprocessing tasks. You can choose one of the following methods to load the data:

1. Using the provided CSV file, create a Path object and read the file data directly into a DataFrame named crypto_df using pd.read_csv().

2. Using the following requests library, retreive the necessary data from the following API endpoint from CryptoCompare - https://min-api.cryptocompare.com/data/all/coinlist. You will need to use the 'Data' key from the json response, then transpose the DataFrame. Name your DataFrame crypto_df.

With the data loaded into a Pandas DataFrame, continue with the following data preprocessing tasks.

3. Keep only the necessary columns: 'CoinName','Algorithm','IsTrading','ProofType','TotalCoinsMined','TotalCoinSupply'

4. Keep only the cryptocurrencies that are trading.

5. Keep only the cryptocurrencies with a working algorithm.

6. Remove the IsTrading column.

7. Remove all cryptocurrencies with at least one null value.

8. Remove all cryptocurrencies that have no coins mined.

9. Drop all rows where there are 'N/A' text values.

10. Store the names of all cryptocurrencies in a DataFrame named coins_name, use the crypto_df.index as the index for this new DataFrame.

11. Remove the CoinName column.

12. Create dummy variables for all the text features, and store the resulting data in a DataFrame named X.

13. Use the StandardScaler from sklearn to standardize all the data of the X DataFrame. Remember, this is important prior to using PCA and K-Means algorithms.

Reducing Data Dimensions Using PCA

Use the PCA algorithm from sklearn to reduce the dimensions of the X DataFrame down to three principal components.

Once you have reduced the data dimensions, create a DataFrame named pcs_df using as columns names "PC 1", "PC 2" and "PC 3"; use the crypto_df.index as the index for this new DataFrame.

You should have a DataFrame like the following:

Clustering Cryptocurrencies Using K-Means

In this section, you will use the KMeans algorithm from sklearn to cluster the cryptocurrencies using the PCA data.

Perform the following tasks:

1. Create an Elbow Curve to find the best value for k using the pcs_df DataFrame.

2. Once you define the best value for k, run the Kmeans algorithm to predict the k clusters for the cryptocurrencies data. Use the pcs_df to run the KMeans algorithm.

3. Create a new DataFrame named clustered_df, that includes the following columns "Algorithm", "ProofType", "TotalCoinsMined", "TotalCoinSupply", "PC 1", "PC 2", "PC 3", "CoinName", "Class". You should maintain the index of the crypto_df DataFrames as is shown bellow.

   

Visualizing Results

In this section, you will create some data visualization to present the final results. Perform the following tasks:

1. Create a 3D-Scatter using Plotly Express to plot the clusters using the clustered_df DataFrame. You should include the following parameters on the plot: hover_name="CoinName" and hover_data=["Algorithm"] to show this additional info on each data point.

2. Use hvplot.table to create a data table with all the current tradable cryptocurrencies. The table should have the following columns: "CoinName", "Algorithm", "ProofType", "TotalCoinSupply", "TotalCoinsMined", "Class"

3. Create a scatter plot using hvplot.scatter, to present the clustered data about cryptocurrencies having x="TotalCoinsMined" and y="TotalCoinSupply" to contrast the number of available coins versus the total number of mined coins. Use the hover_cols=["CoinName"] parameter to include the cryptocurrency name on each data point.

Optional Challenge

For the challenge section, you have to upload your Jupyter notebook to Amazon SageMaker and deploy it.

The hvplot and Plotly Express libraries are not included in the built-in anaconda environments, so for this challenge section, you should use the altair library instead.

Perform the following tasks:

1. Upload your Jupyter notebook and rename it as crypto_clustering_sm.ipynb

2. Select the conda_python3 environment.

3. Import the altair library by running the following code before the initial imports.

!pip install -U altair

4. Use the altair scatter plot to create the Elbow Curve.

5. Use the altair scatter plot, instead of the 3D-Scatter from Plotly Express, to visualize the clusters. Since this is a 2D-Scatter, use x="PC 1" and y="PC 2" for the axes, and add the following columns as tool tips: "CoinName", "Algorithm", "TotalCoinsMined", "TotalCoinSupply".

6. Use the altair scatter plot to visualize the tradable cryptocurrencies using x="TotalCoinsMined" and y="TotalCoinSupply" for the axes.

7. Show the table of current tradable cryptocurrencies using the display() command.

8. Remove all hvplot and Plotly Express references from your code.

Complementary Resources

• Altair visualization library website.

• Simple line chart using Altair.

• Simple Scatter Plot with Tooltips using Altair

• Color customization on Altair

• Printing all rows from a DataFrame

• Install External Libraries and Kernels in Amazon SageMaker Notebook Instances