java爬取https网站的数据实现

深碍√TFBOYSˉ_ 2023-05-31 16:22 616阅读 0赞

**java爬取http类型的网站比较容易实现，因为不需要建立证书的通道，直接通过httpclient访问链接获取相应源码就可以获取相关数据，现在我们可以通过证书的方式，实现java爬取https网站的相关数据。**

# **1.下载网站的证书 ** #

**以CSDN博客为例 谷歌浏览器右键网页——点击检查——点击security出现如下界面**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70][]

**点击view certificate**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 1][]

**点击证书路径 我们可以发现证书如下：**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 2][]

**想要导出证书  需要IE浏览器得支持 我们使用IE浏览器重新访问该网站**

**点击工具——Internet选项**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 3][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 4][]

**可以找到相关证书：**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 5][]

**接下来点击导出**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 6][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 7][]

**选择保存的目录以及文件名**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 8][]

**导出即可**

# **2.java命令行将证书生成秘钥库** #

**进入JDK 安装bin目录，命令行运行:keytool -import -alias Root -file d:/test.cer -keystore “d:/test.keystore” -storepass 123456。转换证书**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 9][]

# **3.编写代码** #

**项目目录如下：红色框框圈住的为需要的包，下载地址为：**[https://download.csdn.net/download/qq\_36706878/11855431][https_download.csdn.net_download_qq_36706878_11855431]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 10][]

**代码如下：**

public static Document sendHttp(String url) {
            String html = "";
            CloseableHttpClient httpclient = null;
            CloseableHttpResponse response = null;
            try {
                SSLConnectionSocketFactory sslsf = createSSLConnSocketFactory();
                httpclient = HttpClients.custom()
                    .setSSLSocketFactory(sslsf).build();
                HttpGet httpget = new HttpGet(url);
                httpget.addHeader(HttpHeaders.USER_AGENT,
                        "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0");
                RequestConfig requestConfig = RequestConfig.custom()
                        .setSocketTimeout(10000).setConnectTimeout(10000).build();// 设置请求和传输超时时间
                httpget.setConfig(requestConfig);
                System.out.println("Executing request " + httpget.getRequestLine());
                response = httpclient.execute(httpget);
                HttpEntity entity = response.getEntity();
                System.out.println("----------------------------------------");
                System.out.println(response.getStatusLine());
                int resStatu = response.getStatusLine().getStatusCode();// 返回码
                if (resStatu == HttpStatus.SC_OK) {// 200正常 其他就不对
                    // 获得相应实体
                    if (entity != null) {
                        html = EntityUtils.toString(entity, "UTF-8");
                        html = html.replace(" ", " ");
                    }
                }
                EntityUtils.consume(entity);
            } catch(Exception e){
                e.printStackTrace();
            }finally{
                if(response!=null){
                    try {
                        response.close();
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                }
                if(httpclient!=null){
                    try {
                        httpclient.close();
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                }
            }
            Document document = Jsoup.parse(html);
    		return document;
        }
    	
    	// ssl通道证书的创建
        private static SSLConnectionSocketFactory createSSLConnSocketFactory()
                throws Exception {
            SSLContext sslcontext = SSLContexts
                    .custom()
                    .loadTrustMaterial(
                            new File(
                                    "E:/key.keystore"),
                            "123456".toCharArray(), new TrustSelfSignedStrategy())   //文件和密码要对应
                    .build();
            SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(
                    sslcontext, new String[] { "TLSv1" }, null,
                    SSLConnectionSocketFactory.getDefaultHostnameVerifier());
            return sslsf;
        }

**测试结果：**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 11][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70]: /images/20230601/2456afa90bae435aaa1d42b0ee51ae62.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 1]: /images/20230601/d5942302ddc94ad886e4c5ca10980ea8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 2]: /images/20230601/52500bb22ae44385967ef44281581b1c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 3]: /images/20230601/6c3725bc547f4e89a8ee921ce2ec3ecb.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 4]: /images/20230601/5e75292304874d9c9fd28bbb4dd835d2.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 5]: /images/20230601/1b0195f9819e437d96f1ee61779761e5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 6]: /images/20230601/698727eb94474e7dab0862a558fea725.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 7]: /images/20230601/e64fd7cbd3ee4570bce6e0a1d0b77f8b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 8]: /images/20230601/49adb1fd98e2430b863cd5c38cf0738d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 9]: /images/20230601/5fc6196c38f0455fa12d410c34e2f8b8.png
[https_download.csdn.net_download_qq_36706878_11855431]: https://download.csdn.net/download/qq_36706878/11855431
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 10]: /images/20230601/1c30f07a2b9941a4bf198cb1f9a132ba.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NzA2ODc4_size_16_color_FFFFFF_t_70 11]: /images/20230601/19a15a598a2145ae9eae254bbb7423f7.png