I want to retrieve at least 40 unique web-link using the API call from the DuckDuckGo search engine by searching the query . I managed to retrieve 19 items using the link, but the number is low.
this is my code that using:
public void search(String searchQuery) {
Set<Link> result = new HashSet<>();
try {
Connection connection = Jsoup
.connect("https://duckduckgo.com/html/?q=" + searchQuery)
.timeout(ConstVal.TIME_OUT);
Document doc = null;
doc = connection.get();
Elements links = doc.select(".result__a[href]");
Set<Element> collect = links.parallelStream().collect(Collectors.toSet());
for (Element link : collect) {
if (result.size() >= 40)
break;
String temp = link.attr("href");
if (temp.startsWith("")) {
Link build = Link.builder()
.domainName(getDomainName(temp))
.build();
result.add(build);
}
}
} catch (IOException e) {
throw new RuntimeException(e);
}
}
String getDomainName(String url) {
String domainName = "";
Matcher matcher = Pattern.compile(DOMAIN_NAME_PATTERN).matcher(url);
if (matcher.find()) {
domainName = matcher.group(0).toLowerCase().trim();
}
return domainName;
}
please give the way to solve this task.