JAVA代码爬虫获取网站信息

￡神魔★判官ぃ 2022-12-03 02:23 252阅读 0赞

目前想获得一些计算机及CPU型号对应的性能分数.在网站上有

[https://www.cpubenchmark.net/cpu\_list.php][https_www.cpubenchmark.net_cpu_list.php]

哈哈哈...(你们明白要干嘛的...)

毕竟型号还是有限的,所以类似于字典表在数据库中,方便自己后期查询维护.

直接上代码,一知半解的,还不怎么懂...

public void getComputScore(List<ZnComputScore> result){
    		log.info("开始:爬取电脑性能得分数据");
    		CloseableHttpClient httpClient = HttpClients.createDefault();
    		CloseableHttpResponse response = null;
    		//2.创建get请求，相当于在浏览器地址栏输入 网址
    		HttpGet request = new HttpGet("https://www.cpubenchmark.net/cpu_list.php");
    		//设置请求头，将爬虫伪装成浏览器
    		request.setHeader("User-Agent","Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36");
    		try {
    			log.info("开始执行爬取数据的http请求.");
    			//3.执行get请求，相当于在输入地址栏后敲回车键
    			response = httpClient.execute(request);
    			log.info("结束执行http的execute请求");
    			//4.判断响应状态为200，进行处理
    			if(response.getStatusLine().getStatusCode() == 200) {
    				//5.获取响应内容
    				HttpEntity httpEntity = response.getEntity();
    				String html = EntityUtils.toString(httpEntity, "utf-8");
    				//6.Jsoup解析html
    				Document document = Jsoup.parse(html);
    				//像js一样，通过标签获取title
    				//像js一样，通过id 获取文章列表元素对象
    				//像js一样，通过class 获取列表下的所有博客
    				Elements trs =document.getElementsByTag("tr");
    				log.info("开始封装[{}]条Element数据",trs.size());
    				for (Element a_s:trs) {
    					Elements tds = a_s.getElementsByTag("tr");
    					ZnComputScore comput = new ZnComputScore();
    					if (a_s.children().size() < 2 || a_s.child(0)== null || a_s.child(1)==null){
    						continue;
    					}
    					comput.setCpuText(a_s.child(0).text());
    					comput.setCpuScore(a_s.child(1).text());
    					result.add(comput);
    				}
    			} else {
    				//如果返回状态不是200，比如404（页面不存在）等，根据情况做处理，这里略
    				System.out.println("返回状态不是200");
    				System.out.println(EntityUtils.toString(response.getEntity(), "utf-8"));
    			}
    		} catch (ClientProtocolException e) {
    			e.printStackTrace();
    		} catch (IOException e) {
    			e.printStackTrace();
    		} finally {
    			//6.关闭
    			HttpClientUtils.closeQuietly(response);
    			HttpClientUtils.closeQuietly(httpClient);
    			log.info("结束:爬取[{}]条电脑得分数据",result.size());
    		}
    	}

<dependency>
    			<groupId>org.apache.httpcomponents</groupId>
    			<artifactId>httpclient</artifactId>
    			<version>4.4.1</version>
    		</dependency>

<dependency>
    			<groupId>org.jsoup</groupId>
    			<artifactId>jsoup</artifactId>
    			<version>1.12.1</version>
    			<scope>compile</scope>
    		</dependency>

httpclient用于请求数据

jsoup用于解析请求返回的html文件

至此就直接用吧....

其他页面啥的,可以通过输出html文件进行业务解析,拿到自己想要的数据.

[https_www.cpubenchmark.net_cpu_list.php]: https://www.cpubenchmark.net/cpu_list.php