I have been trying to run a Python script that goes into a website using Selenium. Scrape some data and check with a text file if there has been anychanges. If there is a difference send a mail to desired mail adresses.
Now, this code works on my Macbook Pro M3Pro (Silicon).
At first I created this code with FireFox but after running into issues with using the code on Koyeb, I have decided that chrome would be a better option.
(Both Firefox and Chrome versions were succesfully running on my pc.)
But in the end I still couldn’t get it working on Koyeb.
Here is the error I am getting:
Traceback (most recent call last):
File "/workspace/main.py", line 84, in <module>
latestNews=getLatestNews()
^^^^^^^^^^^^^^^
File "/workspace/main.py", line 55, in getLatestNews
driver = webdriver.Chrome(options=chrome_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.heroku/python/lib/python3.12/site-packages/selenium/webdriver/chrome/webdriver.py", line 45, in __init__
super().__init__(
File "/app/.heroku/python/lib/python3.12/site-packages/selenium/webdriver/chromium/webdriver.py", line 66, in __init__
super().__init__(command_executor=executor, options=options)
File "/app/.heroku/python/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 212, in __init__
self.start_session(capabilities)
File "/app/.heroku/python/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 299, in start_session
response = self.execute(Command.NEW_SESSION, caps)["value"]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.heroku/python/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 354, in execute
self.error_handler.check_response(response)
File "/app/.heroku/python/lib/python3.12/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: Chrome failed to start: exited normally.
(session not created: DevToolsActivePort file doesn't exist)
(The process started from chrome location /app/.cache/selenium/chrome/linux64/127.0.6533.119/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Stacktrace:
#0 0x562e0a5f96ca <unknown>
#1 0x562e0a2ca600 <unknown>
#2 0x562e0a302485 <unknown>
#3 0x562e0a2fe2f8 <unknown>
#4 0x562e0a348ce8 <unknown>
#5 0x562e0a33c643 <unknown>
#6 0x562e0a30cd31 <unknown>
#7 0x562e0a30d79e <unknown>
#8 0x562e0a5c125b <unknown>
#9 0x562e0a5c51f2 <unknown>
#10 0x562e0a5ae615 <unknown>
#11 0x562e0a5c5d82 <unknown>
#12 0x562e0a59325f <unknown>
#13 0x562e0a5e8e68 <unknown>
#14 0x562e0a5e9040 <unknown>
#15 0x562e0a5f849c <unknown>
#16 0x7fdcb0bd5609 start_thread
And here is my codes:
main.py:
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import expected_conditions
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.chrome.options import Options
import os
os.environ['MOZ_HEADLESS'] = '1'
def send_email(subject, body, to_email):
smtp_server = "smtp.zoho.eu"
smtp_port = 587
from_email = "<mail>"
password = "<password>"
# Create the email
msg = MIMEMultipart()
msg['From'] = from_email
msg['To'] = to_email
msg['Subject'] = subject
# Attach the email body
msg.attach(MIMEText(body, 'plain'))
try:
# Set up the server
server = smtplib.SMTP(smtp_server, smtp_port)
server.starttls() # Upgrade the connection to a secure encrypted SSL/TLS connection
server.login(from_email, password)
# Send the email
server.sendmail(from_email, to_email, msg.as_string())
print("Email sent successfully to: "+to_email+" !")
except Exception as e:
print(f"Failed to send email: {e}")
finally:
server.quit() # Close the connection to the server
def getLatestNews():
# Start Selenium
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-dev-shm-usage")
driver = webdriver.Chrome(options=chrome_options)
driver.implicitly_wait(6)
driver.get("<LINK>")
if((driver.find_element(By.XPATH, "//li[4]/a").text)=="Select Language TRnTR"):
driver.find_element(By.XPATH, "//li[4]/a").click()
driver.find_element(By.XPATH, "//div[2]/div[8]/a").click()
# Get the data from Web
latestNews = (driver.find_element(By.XPATH, "//div[3]/div[2]/div/a").text) + " " + (driver.find_element(By.XPATH, "//span[2]").text)+ " " + (driver.find_element(By.XPATH, "//div[2]/div/span").text) + " " + (driver.find_element(By.XPATH, "//div[2]/span").text)
print(latestNews)
driver.close()
return latestNews
def is_changed(latestNews):
f = open("latestNews.txt", "r")
if(f.read()!=latestNews):
f = open("latestNews.txt", "w")
f.write(latestNews)
f.close()
return True
f.close()
return False
def mailDelivery(latestNews):
mailList= ["[email protected]","[email protected]","[email protected]","[email protected]","[email protected]"]
for i in mailList:
send_email("Topic", "Some textnn"+latestNews, i)
while(True):
latestNews=getLatestNews()
if (is_changed(latestNews)):
mailDelivery(latestNews)
else:
print("No difference")
time.sleep(3600)
requirements.txt:
selenium
Procfile:
web: python main.py
I have checked a lots of Stackoverflow questions and have applied lots of them. But the current version still does not work. And couldn’t find anyone trying to run selenium on Koyeb.
Edit: I also want to mention that I am using the free plan on Koyeb. Which gives like really slow CPU and 512MB RAM
Tiwo is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.