Twitter Sentiment Analysis

Getting data from twitter

1. Using Tweepy (uses official twitter API)

• easy
• several limitations
• check https://developer.twitter.com/en/docs/rate-limits

2. Using Twint (unofficial)

• a little difficult
• less limitation
• amazing for large volume

Analysis

• Vader sentiment using python package https://towardsdatascience.com/sentimental-analysis-using-vader-a3415fef7664
• Naive bayes for sentiment analysis (a little of DIY) https://www.datacamp.com/community/tutorials/simplifying-sentiment-analysis-python
• Spacy package
• Gensim package
• etc.

Installation

Clone the repo

git clone https://github.com/cloudfactory/sentiment_analysis_twitter_starter_code

CD into the cloned directory and create a virtualenv

python -m venv env

Enable virtualenv

source env/bin/activate

Install dependency packages from requirements.txt

pip install -r requirements.txt

Oper jupyter lab session

jupyter-lab

A simple twitter sentiment analysis poc

import tweepy
import json
from tweepy import OAuthHandler
import pandas as pd

Full documentation here: https://docs.tweepy.org/en/stable/client.html#tweets

Access keys

Apply at twitter developer and receive these:

API_SECRET_KEY = "fill this in"
API_KEY = "fill this in"
ACCESS_TOKEN = "fill this in"
ACCESS_TOKEN_SECRET = "fill this in"

class TwitterClient(object):
    '''
    Twitter Client
    '''

    def __init__(self):
        '''
        Class constructor or initialization method.
        '''
        # read keys from the secret credentials file
        api_key = API_KEY
        api_secret =API_SECRET_KEY
        access_token = ACCESS_TOKEN
        access_token_secret = ACCESS_TOKEN_SECRET

        try:
            self.auth = OAuthHandler(api_key, api_secret)
            self.auth.set_access_token(access_token,
                                       access_token_secret)
            self.api = tweepy.API(self.auth)
        except:
            print('Error: Authentication error')


    def get_tweets(self):
        tweet = self.api.user_timeline(screen_name ='kathmandupost', count=20)
        return tweet

raw = TwitterClient().get_tweets()

df = pd.json_normalize([r._json for r in raw])

df.head()

df.text

0     EDITORIAL: Protect paddy farmers\n\nEase of ac...
1     Despite some opposition, Oli appears to have p...
2     Over 100,000 doses of Pfizer-BioNtech vaccine ...
3     Congress may appoint deputy Speaker, leaving s...
4     Everything you need to know about the Covid-19...
5     United States to provide 100,620 doses of Pfiz...
6     Paddy damage by freak rains estimated at Rs8.2...
7     Dalit representatives complain of social discr...
8     Consult Delhi for census in Kalapani, census b...
9     Supreme Court justices to boycott full court m...
10    London expands vehicle levy to improve air qua...
11    ‘Children are going to die’, UN agency warns a...
12    EDITORIAL: Railblock ahead\n\nDelay in operati...
13    Nepal reports 673 new Covid-19 cases, 13 death...
14    Whether it’s supply or demand, oil era heads f...
15    Justices to decide their further step after me...
16    Climate change: what are the economic stakes?\...
17    Oslo opens museum to “The Scream” painter Munc...
18    Everything you need to know about the Covid-19...
19    Fauci says vaccines for kids between 5-11 like...
Name: text, dtype: object

Sentiment analysis

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

def sentiment_scores(sentence):
    sid_obj = SentimentIntensityAnalyzer()
    return sid_obj.polarity_scores(sentence)

df['sentiment_scores'] = df.text.apply(sentiment_scores)

df[['text', 'sentiment_scores']]

sentiments = pd.concat([df, df.sentiment_scores.apply(pd.Series)], axis=1)[['text', 'neg', 'neu', 'pos']]

sentiments

Depending on the threshold you create, you can also label something as positive or negative sentiment now