Open-source datasets are essential for data science students, researchers, and professionals to experiment with various artificial intelligence (AI) and machine learning (ML) algorithms. Tasks such as computer vision, regression, semantic analysis, and data analysis require substantial datasets to practice and apply these techniques effectively.
In recent research papers, proprietary datasets are often used and not made available to the public. This challenges those looking to apply and test their new skills. If you’ve encountered this issue, we’ve got you covered. Here’s a curated list of 20 open datasets, categorized by their application domains, along with use cases to get you started.
Image Classification
The MNIST database is a large database of handwritten digits commonly used for training various image processing systems. The database is also widely used for training and testing in the machine learning field. It has over 70,000 images for digit recognition.
Size & Number of Records: ~50 MB, Number of Records: 70,000 images in 10 classes
Use Cases: Great for beginners to practice image classification techniques and algorithms like convolutional neural networks (CNNs).
The CIFAR-10 dataset, created by the Canadian Institute For Advanced Research, consists of 60,000 32x32 color images in 10 classes, with 6,000 images per class. It is commonly used to train machine learning and computer vision algorithms.
Sample of CIFAR-10
Size & Number of Records: 170 MB, 60,000 images in 10 classes
Use Cases: Ideal for testing and comparing different image classification models.
ImageNet is a large-scale visual database designed for use in visual object recognition research, containing over 14 million images and more than 20,000 categories. More than 14 million images have been hand-annotated by the project to indicate what objects are pictured.
Size & Number of Records: 150GB, 1,500,000; each with multiple bounding boxes and respective class labels
Use Cases: Used in large-scale visual recognition challenges and for training deep learning models.
Fashion-MNIST, developed by Zalando, is a dataset of Zalando’s article images, consisting of 70,000 grayscale images in 10 categories, intended as a drop-in replacement for the original MNIST.
Size & Number of Records: 30 MB, 70,000 images in 10 classes
Use Cases: Useful for image classification tasks related to fashion and e-commerce.
The CelebA dataset, provided by the Chinese University of Hong Kong, contains over 200,000 celebrity images with 40 attribute labels per image.
Number of Records: more than 200K celebrity images, each with 40 attribute annotations
Use Cases: Face recognition and attribute prediction tasks.
Natural Language Processing (NLP)
The IMDB dataset, created by Stanford University, consists of 50,000 highly polar movie reviews for binary sentiment classification.
Size & Number of Records: 80 MB, 25,000 highly polar movie reviews for training, and 25,000 for testing
Use Cases: Sentiment analysis and natural language processing tasks.
The 20 Newsgroups dataset, compiled by Ken Lang, is a collection of approximately 20,000 newsgroup documents, partitioned across 20 different newsgroups.
Size & Number of Records: 20 MB, 20,000 messages taken from 20 newsgroups
Use Cases: Text classification and clustering tasks.
Image credits to Rajpurkar et al.
The Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset consisting of questions posed by crowd workers on a set of Wikipedia articles, where the answer to every question is a segment of text from the corresponding passage. Stanford Question Answering Dataset with 100,000+ questions for machine comprehension.
Use Cases: Training and evaluating question-answering systems and language models.
Image Source: https://medium.com/nlplanet/two-minutes-nlp-glue-tasks-and-2022-leaderboard-517baedfa597
The General Language Understanding Evaluation (GLUE) benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems.
Use Cases: Comprehensive benchmarking of NLP models on a variety of tasks.
The COCO dataset, provided by Microsoft, is a large-scale object detection, segmentation, and captioning dataset, containing over 330,000 images with five captions per image.
Use Cases: Image captioning and visual question-answering tasks.
6. Sentiment140
Sentiment140 is a popular dataset that can be used for sentiment analysis. Emotions have been removed from the data. The final dataset has 6 features: polarity of the tweet, id of the tweet, date of the tweet, the query, username of the tweeter, text of the tweet
Size & Number of Records: 80 mb, 1,60,000 tweets
This is an open dataset released by Yelp for learning purposes. It consists of millions of user reviews, business attributes, and over 200,000 pictures from multiple metropolitan areas.
Size & Number of Records: 2.66 GB JSON, 2.9 GB SQL and 7.5 GB Photos (all compressed), 5,200,000 reviews, 174,000 business attributes, 200,000 pictures and 11 metropolitan areas
Time Series
Samples of activity data from UCI HAR
The UCI Human Activity Recognition (HAR) dataset contains data collected from the accelerometers of smartphones worn by participants while performing six different activities.
Use Cases: Human activity recognition and time series analysis.
Speech Recognition
The LibriSpeech dataset is a corpus of approximately 1,000 hours of 16kHz read English speech, derived from audiobooks available on the LibriVox project.
Size & Number of Records: ~60 GB 1000 hours of speech
Use Cases: Training and evaluating speech recognition systems.
The VoxCeleb dataset contains large-scale speaker identification data, including more than 1 million utterances by 6,000 speakers, collected from YouTube videos.
Size & Number of Records: 150 MB, 100,000 instances by 1,251 celebrities
Use Cases: Speaker identification and verification tasks.
Healthcare
The Medical Information Mart for Intensive Care (MIMIC-III) is a comprehensive clinical database developed by MIT that includes detailed information about patients admitted to critical care units.
Use Cases: Clinical decision support and predictive modeling in healthcare.
2. Breast Cancer Wisconsin Dataset
The Breast Cancer Wisconsin Dataset contains features computed from digitized images of fine needle aspirate (FNA) of breast mass tissue samples.
Use Cases: Cancer detection and classification tasks.
Recommender Systems
The MovieLens dataset, created by GroupLens Research, is a collection of movie ratings data used for research in recommender systems. Some of the datasets present are given below:
MovieLens 25M Dataset
MovieLens Tag Genome Dataset 2021
MovieLens Latest Datasets
MovieLens 1B Synthetic Dataset
Use Cases: Building and evaluating recommender systems.
The Amazon Product Reviews dataset, compiled by Jianmo Ni from UCSD, contains millions of customer reviews and ratings for a variety of products on Amazon. Datasets contain the following features:
user/item interactions
star ratings
timestamps
product reviews
social networks
item-to-item relationships (e.g. purchases, compatibility)
product images
price, brand, and category information
GPS data
heart-rate sequences
other metadata
Use Cases: Sentiment analysis and product recommendation tasks.
Miscellaneous
Partial visualization of Bremen (Germany)
OpenStreetMap (OSM) is a collaborative project to create a free editable map of the world, containing geographic data contributed by a global community.
Use Cases: Geographic information system (GIS) applications, route planning, and location-based services.
Conclusion
Open-source datasets play a crucial role in advancing the skills of data science students, researchers, and professionals. By providing ample opportunities to practice and apply various AI and ML algorithms, these datasets help bridge the gap between theoretical knowledge and practical application.
This curated list of 20 datasets across different domains acts as a valuable resource for anyone looking to deepen their understanding and expertise in machine learning and deep learning. Dive into these datasets, experiment with your models, and take your data science journey to the next level.
Happy learning and innovating!
Resources
1. https://paperswithcode.com/datasets
2. https://www.kaggle.com/datasets