【httpclient编写爬虫】post提交json数据和普通键值

ゝ一世哀愁。 2022-07-13 08:09 177阅读 0赞

# 写在开头 #

在开发爬虫的过程中，难免碰到post提交的问题。  
本文比较了两种数据提交方式，并且使用httpclient模拟网站post提交两种数据。

我见过的post提交方式有两种：

1.  普通的键值对提交方式；
2.  提交json数据。

我所使用的httpclient版本

<dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpclient</artifactId>
        <version>4.5.2</version>
    </dependency>

# 普通键值对的提交方式 #

CloseableHttpClient httpclient = HttpClients.createDefault();
    
    HttpPost httpPost = new HttpPost("http://targethost/login");
    List<NameValuePair> nvps = new ArrayList<NameValuePair>();
    nvps.add(new BasicNameValuePair("username", "vip"));
    nvps.add(new BasicNameValuePair("password", "secret"));
    httpPost.setEntity(new UrlEncodedFormEntity(nvps));
    CloseableHttpResponse response2 = httpclient.execute(httpPost);
    
    try {
        System.out.println(response2.getStatusLine());
        HttpEntity entity2 = response2.getEntity();
        // do something useful with the response body
        // and ensure it is fully consumed
        EntityUtils.consume(entity2);
    } finally {
        response2.close();
    }

# JSON数据提交方式 #

要提交的数据

{
        "username" : "vip",
        "password" : "secret" }

代码

import org.apache.http.Consts;
    import org.apache.http.HttpEntity;
    import org.apache.http.client.methods.CloseableHttpResponse;
    import org.apache.http.client.methods.HttpPost;
    import org.apache.http.entity.StringEntity;
    import org.apache.http.impl.client.CloseableHttpClient;
    import org.apache.http.impl.client.HttpClients;
    import org.apache.http.util.EntityUtils;
    
    import java.io.IOException;
    
    /** * Created by CarlZhang on 2017/1/1. */
    public class PostJsonTest { 
        public static void main(String[] args) {
            CloseableHttpClient httpclient = HttpClients.createDefault();
            try {
    
                HttpPost httpPost = new HttpPost("http://targethost/login");
    
                //json数据{"username":"vip","password":"secret"}
                String jsonStr  = "{\"username\":\"vip\",\"password\":\"secret\"}";
    
                StringEntity se = new StringEntity(jsonStr, Consts.UTF_8);
                se.setContentEncoding("UTF-8");
                se.setContentType("application/json");
    
                httpPost.setEntity(se);
                CloseableHttpResponse response2 = httpclient.execute(httpPost);
    
                try {
                    System.out.println(response2.getStatusLine());
                    HttpEntity entity2 = response2.getEntity();
                    // do something useful with the response body
                    // and ensure it is fully consumed
                    //EntityUtils.consume(entity2);
                    String res = EntityUtils.toString(entity2);
                    System.out.println(res);
                } finally {
                    response2.close();
                }
    
            } catch (IOException e) {
                e.printStackTrace();
            }finally {
                try {
                    httpclient.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }
    }

# 实例-JSON提交 #

如下是我在某个网站点击发帖，然后在chrome （按住F12键）打开的dubug工具，可以看到我提交的post请求。其中Form Data就是使用post提交的json数据。  
![这里写图片描述][SouthEast]

后端怎么拿到这些数据的呢？  
该网站使用的开源库latke中的[Requests类][Requests]  
可以看到，它是通过Reader对象去读入流数据的。

/** * Gets the request json object with the specified request. * * @param request the specified request * @param response the specified response, sets its content type with "application/json" * @return a json object * @throws ServletException servlet exception * @throws IOException io exception */
        public static JSONObject parseRequestJSONObject(final HttpServletRequest request, final HttpServletResponse response)
            throws ServletException, IOException {
            response.setContentType("application/json");
    
            final StringBuilder sb = new StringBuilder();
            BufferedReader reader;
    
            final String errMsg = "Can not parse request[requestURI=" + request.getRequestURI() + ", method=" + request.getMethod()
                + "], returns an empty json object";
    
            try {
                try {
                    reader = request.getReader();
                } catch (final IllegalStateException illegalStateException) {
                    reader = new BufferedReader(new InputStreamReader(request.getInputStream()));
                }
    
                String line = reader.readLine();
    
                while (null != line) {
                    sb.append(line);
                    line = reader.readLine();
                }
                reader.close();
    
                String tmp = sb.toString();
    
                if (Strings.isEmptyOrNull(tmp)) {
                    tmp = "{}";
                }
    
                return new JSONObject(tmp);
            } catch (final Exception ex) {
                LOGGER.log(Level.ERROR, errMsg, ex);
    
                return new JSONObject();
            }
        }

另外，前端js代码是通过jquery的类库去提交的json数据  
![这里写图片描述][SouthEast 1]

# 实例-普通post提交 #

![这里写图片描述][SouthEast 2]

有兴趣可以查看我这篇文章[【js类库AngularJs】解决angular+springmvc的post提交问题][js_AngularJs_angular_springmvc_post]

[SouthEast]: /images/20220713/179db441c93a40c8b0249fa95df761c9.png
[Requests]: https://github.com/b3log/latke/blob/master/latke/src/main/java/org/b3log/latke/util/Requests.java
[SouthEast 1]: /images/20220713/ddba9900532a44e1ae4e2495904dfcb6.png
[SouthEast 2]: /images/20220713/2ffd5c557f744c9bb30fef29ba7f3822.png
[js_AngularJs_angular_springmvc_post]: http://blog.csdn.net/u012386311/article/details/53966083