I’m having an issue with a bash script. Specifically, I’m trying to extract data from a webpage using grep and sed, but it never returns anything. I’ve also tried with awk, but no luck. Does anyone have any ideas?
#!/bin/bash
# Define variables for URL and browser
sGCitta="fucecchio"
sGTypo="vendita-case"
sGDomain="immobiliare"
url="https://www.$sGDomain.it/$sGTypo/$sGCitta"
# Get the HTML content of the page
html_content=$(curl -s -L "$url")
# Find all listings on the page using grep
annunci=$(echo "$html_content" | grep -o '<div class="listing-item">.*</div>')
# Print the listings found after the grep operation for debugging
echo "Listings found:"
echo "$annunci"
# Connection to the SQLite database
db_file="immo.db"
# Loop through the listings
while read -r annunci; do
# Extract information from the listing using sed
prezzo=$(echo "$annunci" | sed -n 's/.*<span class="in-formattedPrice__text">([^<]*)</span>.*/1/p')
link=$(echo "$annunci" | sed -n 's/.*<a class="in-listingCardTitle" href="([^"]*)"[^>]*>.*/1/p')
descrizione=$(echo "$annunci" | sed -n 's/.*<a class="in-listingCardTitle" title="([^"]*)"[^>]*>.*/1/p')
metratura=$(echo "$annunci" | sed -n 's/.*<span>([^<]*)</span>.*/1/p')
# Check if the description contains the word "asta"
if [[ "$descrizione" =~ "asta" ]]; then
astaOnOff=1
else
astaOnOff=0
fi
# Insert data into the SQLite database
sqlite3 "$db_file" "INSERT INTO $sGDomain (prezzo, link, descrizione, metratura, asta) VALUES ('$prezzo', '$link', '$descrizione', '$metratura', '$astaOnOff')"
done <<< "$annunci"
I tried with awk but I couldn’t succeed. I think it’s a problem with the regular expression. I expect it to return the price, square meters, link, and description.