With this code I can go to this webpage and capture the first 10 events. The problem is that there are 30+ events and I need to capture them all. When you are scrolling manually with the mouse something triggers the retrieval of the next 10 events. How do I do this programmatically in VBA until I get to the bottom?
Sub DownloadFile()
Dim WinHttpReq As Object
Dim oStream As Object
Dim myURL As String
Dim LocalFilePath As String
myURL = "https://www.meetup.com/cflfreethought/events/"
LocalFilePath = "C:UsersBrooksDesktopHTML_download.txt"
Set WinHttpReq = CreateObject("Microsoft.XMLHTTP")
WinHttpReq.Open "GET", myURL, False, "", "" '("username", "password")
WinHttpReq.send
If WinHttpReq.status = 200 Then
Set oStream = CreateObject("ADODB.Stream")
oStream.Open
oStream.Type = 1
oStream.Write WinHttpReq.responseBody
oStream.SaveToFile LocalFilePath, 2 ' 1 = no overwrite, 2 = overwrite
oStream.Close
End If
End Sub
Brooks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
The thing with Microsoft.XMLHTTP is that it’s not a browser, it’s mainly a library that allows you to send and receive HTTP requests, but it doesn’t render the HTML page and doesn’t execute any JavaScript code on the client-side.
If you need to scroll the page and trigger Javascript code to load more results, you’ll need to use a browser. There are a few ways to do that with VBA, but the most common approach is to use a VBA wrapper around the popular Selenium Library.
Regarding which wrapper to choose, the most popular is remains SeleniumBasic. There is even a video tutorial to help you get started using Chrome.
However, SeleniumBasic hasn’t received an update in a while and it no longer works with FireFox. If you need to use FireFox or want to use a more recent project, you can also have a look at SeleniumVBA or TinySeleniumVBA.