I am working on a project as I bring to land a position in big data. I wanted to fetch tweets in order to perform sentiment analysis, and I am too using Selenium and Safari as browser.

I can share my code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from time import sleep
import getpass

my_user = "publics_peaking"
my_pass = getpass.getpass()
#you will be asked to input your password

# Initialize the Safari WebDriver (allow for automation on Safari) 
driver = webdriver.Safari()
driver.get("https://twitter.com/i/flow/login")
# driver.maximize_window()
sleep(3)

#Feeding the webpage with user_id
user_id = driver.find_element(By.XPATH,"//input[@type='text']")
user_id.send_keys(my_user)
user_id.send_keys(Keys.ENTER)

#Feeding the webpage with user_id
user_id = driver.find_element(By.XPATH,"//input[@type='text']")
user_id.send_keys(my_user)
user_id.send_keys(Keys.ENTER)

#Feeding the webpage with password
password = driver.find_element(By.XPATH,"//input[@type='password']")
password.send_keys(my_pass)
password.send_keys(Keys.ENTER)

# Wait for the cookie consent button to appear and click 'Refuse non-essential cookies'
refuse_cookies_button = WebDriverWait(driver, 10).until(
  EC.element_to_be_clickable((By.XPATH, "//button[contains(., 'Refuse non-essential cookies')]"))
)
refuse_cookies_button.click()

So far it works just fine – I am instead having troubles with feeding the search button with a specific topic in order to get tweets, which I would later fetch in order to create a dataframe.

Can anybody help with the below? I don’t understand what am I doing wrong:

search_item = "your_topic"
#Accessing the search box, inputting the topic and execute 'enter' button
search_box = driver.find_element(By.XPATH,"//input[@data-testid='SearchBox_Search_Input']")
search_box.send_keys(search_item)
search_box.send_keys(Keys.ENTER)

#Fetching tweets

tweets = driver.find_elements(By.XPATH,"//div[@data-testid='tweetText']")
while True:
    for tweet in tweets:
        all_tweets.add(tweet.text)
    driver.execute_script('window.scrollTo(0, document.body.scrollHeight);')
    sleep(3)
    tweets = driver.find_elements(By.XPATH,"//div[@data-testid='tweetText']")
    if len(all_tweets)>50: #adjust figure to fetch more than 50 tweets
        break

all_tweets = list(all_tweets)
all_tweets[0]

Thank you in advance!

N

I was able to access the Twitter login webpage, input username and password, and refuse cookies using a Selenium script with a Safari web driver through Python. I cannot get the script to access the search box, paste the topic I would like to find tweets for and hence enter to get the required tweets.