如何获取全平台每日热搜——接口篇

本是古典何须时尚 2023-10-13 14:01 63阅读 0赞

今天来做一个每日热点页面，用来接收各个平台实时热搜榜单。

逻辑就是找到平台后台**热搜接口**，自己后台处理一下数据就能给咱们的前端使用了。

那么可能会有小伙伴会问，那么为什么不直接前端调用各个平台热搜接口呢？

因为跨域问题，只能配置允许跨域或者前端和后端用同一个域名，后端用反代一下接口解决。人家平台肯定不会在后端允许跨域，所以只能用第二种解决了。

和之前搞微信小程序差不多，小程序不允许调用未备案域名的接口，所以折中的办法就是找个已经备案的域名去请求数据，再由备案的域名返回数据给小程序。

### 找接口 ###

这里以某博作为例子，因为我经常上班等电梯的时候会刷一刷ԅ(¯﹃¯ԅ)。

进入到热搜页面，首先**F12**调出来页面控制台，选择**Fetch/XHR**过滤请求，再根据名称来推断函数的作用。通常情况下函数名称和它的作用紧密相关，某博程序员素养非常高，点赞( o｀ω′)。

![format_png][]

这里这个**HotSearch**就是获取热搜的函数名称了，看看调用的是哪个接口吧。

![format_png 1][]

既然已经之后接口了，接下来就可以后台处理数据，只把我们需要的数据处理出来就可以了。

### 处理数据 ###

可以用**postman**请求一下接口，看看数据结构。

{
        
        "ok": 1,
        "data": {
        
            "hotgovs": [
                {
        
                    "flag": 2,
                    "name": "#中美鼓岭情缘为何延续至今#",
                    "topic_flag": 1,
                    "note": "#中美鼓岭情缘为何延续至今#",
                    "url": "http://weibo.com/1663072851/N8SFibFSv",
                    "pos": 1,
                    "small_icon_desc_color": "#ff9406",
                    "icon_desc": "热",
                    "word": "#中美鼓岭情缘为何延续至今#",
                    "mid": "4921167602782591",
                    "small_icon_desc": "热",
                    "icon_desc_color": "#ff9406",
                    "is_gov": 1,
                    "is_hot": 1
                }
            ],
            "realtime": [
                {
        
                    "flag": 2,
                    "icon_desc": "热",
                    "star_name": {
        },
                    "icon_desc_color": "#ff9406",
                    "small_icon_desc": "热",
                    "small_icon_desc_color": "#ff9406",
                    "is_hot": 1,
                    "note": "荷兰政府宣布将集体辞职",
                    "extension": 2,
                    "onboard_time": 1688813492,
                    "star_word": 0,
                    "fun_word": 0,
                    "channel_type": "",
                    "ad_info": "",
                    "topic_flag": 1,
                    "label_name": "热",
                    "word_scheme": "#荷兰政府宣布将集体辞职#",
                    "realpos": 1,
                    "num": 1318435,
                    "subject_label": "",
                    "word": "荷兰政府宣布将集体辞职",
                    "subject_querys": "",
                    "mid": "4921233814326440",
                    "expand": 0,
                    "category": "国外要闻,社会新闻",
                    "emoticon": "",
                    "raw_hot": 1318435,
                    "rank": 0
                },
           "hotgov": {
        
                "flag": 2,
                "name": "#中美鼓岭情缘为何延续至今#",
                "topic_flag": 1,
                "note": "#中美鼓岭情缘为何延续至今#",
                "url": "http://weibo.com/1663072851/N8SFibFSv",
                "pos": 1,
                "small_icon_desc_color": "#ff9406",
                "icon_desc": "热",
                "word": "#中美鼓岭情缘为何延续至今#",
                "mid": "4921167602782591",
                "small_icon_desc": "热",
                "icon_desc_color": "#ff9406",
                "is_gov": 1,
                "is_hot": 1
            }
    }

稍微找了一条来做演示，这里热搜是在**realtime**字段里。

而我们需要找里面的

1.  热搜名字 ：note
2.  热搜类型 ：category
3.  热搜热度 ：num
4.  热搜链接 ：url

前面三个字段可以直接从接口返回的数据中获得，但是跳转热搜详情的**url**并没有传进来。

这里我们可以随便找一个热搜，点进去看它的**url**有没有规律，下面是几个热搜详情的链接。

https://s.weibo.com/weibo?q=%23荷兰政府宣布将集体辞职%23
    
    https://s.weibo.com/weibo?q=%23第一次上交工资就被警察查了%23
    
    https://s.weibo.com/weibo?q=%23小浪底又见黄河之水天上来%23

不难看出除了热搜名变了之外，其他都一样，那么就可以根据这个规律生成热搜详情的链接。

以下是对某博热搜数据处理的java代码，抛砖引玉。

public List<WeiBoHot> getWeiBoHotList(){
        
        String data = restTemplate.getForObject("https://weibo.com/ajax/side/hotSearch",String.class);
        Map dataMap = JSON.parseObject(data,new TypeReference<Map>(){
        });
        Map data1 = (Map) dataMap.get("data");
        List<Map<String,String>> hotList = (List)data1.get("realtime");
        List<WeiBoHot> weiBoHotList = new ArrayList<>();
        hotList.forEach(hot->{
        
            WeiBoHot weiBoHot = WeiBoHot.builder()
                                    .category(hot.get("category"))
                                    .title(hot.get("note"))
                                    .hot(String.valueOf(hot.get("num")))
                                    .url("https://s.weibo.com/weibo?q=%23"+hot.get("note")+"%23")
                                    .build();
            weiBoHotList.add(weiBoHot);
        });
        return weiBoHotList;
    }

下面是处理后的结果

{
        
        "code": 200,
        "message": "请求成功",
        "data": [
            {
        
                "category": "幽默",
                "title": "关于自愿离职公司一次性发放十万",
                "hot": "1712800",
                "url": "https://s.weibo.com/weibo?q=%23关于自愿离职公司一次性发放十万%23"
            },
            {
        
                "category": "国外要闻,社会新闻",
                "title": "荷兰政府宣布将集体辞职",
                "hot": "1127967",
                "url": "https://s.weibo.com/weibo?q=%23荷兰政府宣布将集体辞职%23"
            },
            {
        
                "category": "社会新闻",
                "title": "小浪底又见黄河之水天上来",
                "hot": "1036609",
                "url": "https://s.weibo.com/weibo?q=%23小浪底又见黄河之水天上来%23"
            },
            {
        
                "category": "艺人",
                "title": "澳门偶遇",
                "hot": "1036577",
                "url": "https://s.weibo.com/weibo?q=%23澳门偶遇%23"
            },
            {
        
                "category": "艺人",
                "title": "李玟老公称一直对李玟很好",
                "hot": "1035354",
                "url": "https://s.weibo.com/weibo?q=%23李玟老公称一直对李玟很好%23"
            },
            {
        
                "category": "影视",
                "title": "泰妃看肖战出场的反应",
                "hot": "1033850",
                "url": "https://s.weibo.com/weibo?q=%23泰妃看肖战出场的反应%23"
            },
            {
        
                "category": null,
                "title": "紫白金宝宝成长有惊喜",
                "hot": "989876",
                "url": "https://s.weibo.com/weibo?q=%23紫白金宝宝成长有惊喜%23"
            },
            {
        
                "category": "幽默",
                "title": "90后终于熬成了挂钥匙的那群人",
                "hot": "789548",
                "url": "https://s.weibo.com/weibo?q=%2390后终于熬成了挂钥匙的那群人%23"
            },
            {
        
                "category": "综艺",
                "title": "Angelababy生孩子期间打上武则天高位",
                "hot": "779800",
                "url": "https://s.weibo.com/weibo?q=%23Angelababy生孩子期间打上武则天高位%23"
            },
            {
        
                "category": "影视",
                "title": "魏大勋马伯骞赢麻了",
                "hot": "715457",
                "url": "https://s.weibo.com/weibo?q=%23魏大勋马伯骞赢麻了%23"
            },
            {
        
                "category": "体育",
                "title": "陈梦",
                "hot": "708355",
                "url": "https://s.weibo.com/weibo?q=%23陈梦%23"
            },
            {
        
                "category": "影视",
                "title": "杨洋和张彬彬 魏大勋和王楚然",
                "hot": "708139",
                "url": "https://s.weibo.com/weibo?q=%23杨洋和张彬彬 魏大勋和王楚然%23"
            },
            {
        
                "category": "幽默",
                "title": "第一次上交工资就被警察查了",
                "hot": "707220",
                "url": "https://s.weibo.com/weibo?q=%23第一次上交工资就被警察查了%23"
            },
            {
        
                "category": "音乐",
                "title": "鹿晗好喜欢戴金手链",
                "hot": "657362",
                "url": "https://s.weibo.com/weibo?q=%23鹿晗好喜欢戴金手链%23"
            },
            {
        
                "category": "体育",
                "title": "德国女生坚持跑步20周成功蜕变",
                "hot": "559402",
                "url": "https://s.weibo.com/weibo?q=%23德国女生坚持跑步20周成功蜕变%23"
            },
            {
        
                "category": "社会新闻",
                "title": "全国校园音乐大赛总决赛",
                "hot": "430639",
                "url": "https://s.weibo.com/weibo?q=%23全国校园音乐大赛总决赛%23"
            },
            {
        
                "category": "音乐",
                "title": "TMEA致敬李玟",
                "hot": "397846",
                "url": "https://s.weibo.com/weibo?q=%23TMEA致敬李玟%23"
            },
            {
        
                "category": "社会正能量",
                "title": "唤醒植物人丈夫后她退还18万元善款",
                "hot": "397292",
                "url": "https://s.weibo.com/weibo?q=%23唤醒植物人丈夫后她退还18万元善款%23"
            },
            {
        
                "category": "社会新闻",
                "title": "9个孩子被骗缅甸遭电击开水烫",
                "hot": "341643",
                "url": "https://s.weibo.com/weibo?q=%239个孩子被骗缅甸遭电击开水烫%23"
            },
            {
        
                "category": "游戏",
                "title": "BW2023",
                "hot": "323029",
                "url": "https://s.weibo.com/weibo?q=%23BW2023%23"
            },
            {
        
                "category": "影视",
                "title": "消失的她票房",
                "hot": "317806",
                "url": "https://s.weibo.com/weibo?q=%23消失的她票房%23"
            },
            {
        
                "category": "综艺",
                "title": "何炅跟着杨紫一起跪",
                "hot": "315557",
                "url": "https://s.weibo.com/weibo?q=%23何炅跟着杨紫一起跪%23"
            },
            {
        
                "category": "影视",
                "title": "我的人间烟火 伪骨科",
                "hot": "313546",
                "url": "https://s.weibo.com/weibo?q=%23我的人间烟火 伪骨科%23"
            },
            {
        
                "category": "影视",
                "title": "长相思",
                "hot": "312093",
                "url": "https://s.weibo.com/weibo?q=%23长相思%23"
            },
            {
        
                "category": "社会新闻",
                "title": "女生放假回家帮2嫂子带5娃",
                "hot": "309125",
                "url": "https://s.weibo.com/weibo?q=%23女生放假回家帮2嫂子带5娃%23"
            },
            {
        
                "category": "社会新闻",
                "title": "女儿因蛋糕贵大哭求妈妈退款",
                "hot": "289159",
                "url": "https://s.weibo.com/weibo?q=%23女儿因蛋糕贵大哭求妈妈退款%23"
            },
            {
        
                "category": "社会新闻",
                "title": "老人游泳误入深海漂浮1小时获救",
                "hot": "279311",
                "url": "https://s.weibo.com/weibo?q=%23老人游泳误入深海漂浮1小时获救%23"
            },
            {
        
                "category": "音乐",
                "title": "鹿晗回复邓超喊话买单",
                "hot": "279278",
                "url": "https://s.weibo.com/weibo?q=%23鹿晗回复邓超喊话买单%23"
            },
            {
        
                "category": "艺人",
                "title": "宋轶真的穿白色衣服了",
                "hot": "279248",
                "url": "https://s.weibo.com/weibo?q=%23宋轶真的穿白色衣服了%23"
            },
            {
        
                "category": "艺人",
                "title": "五月天演唱会歌迷举发光蚊帐被疯狂吐槽",
                "hot": "279243",
                "url": "https://s.weibo.com/weibo?q=%23五月天演唱会歌迷举发光蚊帐被疯狂吐槽%23"
            },
            {
        
                "category": "社会新闻",
                "title": "小熊猫幼崽和妈妈亲亲",
                "hot": "279198",
                "url": "https://s.weibo.com/weibo?q=%23小熊猫幼崽和妈妈亲亲%23"
            },
            {
        
                "category": "社会新闻",
                "title": "女子找手机全飞机乘客干等3小时",
                "hot": "278191",
                "url": "https://s.weibo.com/weibo?q=%23女子找手机全飞机乘客干等3小时%23"
            },
            {
        
                "category": "音乐",
                "title": "汪苏泷演唱会",
                "hot": "273512",
                "url": "https://s.weibo.com/weibo?q=%23汪苏泷演唱会%23"
            },
            {
        
                "category": "游戏,艺人",
                "title": "花傲天上场",
                "hot": "266122",
                "url": "https://s.weibo.com/weibo?q=%23花傲天上场%23"
            },
            {
        
                "category": "影视",
                "title": "胡一统都当校长了",
                "hot": "265047",
                "url": "https://s.weibo.com/weibo?q=%23胡一统都当校长了%23"
            },
            {
        
                "category": "艺人",
                "title": "李玟二姐喊话让知情人站出来",
                "hot": "259244",
                "url": "https://s.weibo.com/weibo?q=%23李玟二姐喊话让知情人站出来%23"
            },
            {
        
                "category": "音乐",
                "title": "TMEA音乐盛典",
                "hot": "255619",
                "url": "https://s.weibo.com/weibo?q=%23TMEA音乐盛典%23"
            },
            {
        
                "category": "体育",
                "title": "王楚钦vs莫雷加德",
                "hot": "255031",
                "url": "https://s.weibo.com/weibo?q=%23王楚钦vs莫雷加德%23"
            },
            {
        
                "category": "社会新闻",
                "title": "猴痘病毒是怎么感染的",
                "hot": "248859",
                "url": "https://s.weibo.com/weibo?q=%23猴痘病毒是怎么感染的%23"
            },
            {
        
                "category": "体育",
                "title": "亚运会门票",
                "hot": "242707",
                "url": "https://s.weibo.com/weibo?q=%23亚运会门票%23"
            },
            {
        
                "category": "音乐",
                "title": "威神V全开麦",
                "hot": "223627",
                "url": "https://s.weibo.com/weibo?q=%23威神V全开麦%23"
            },
            {
        
                "category": "音乐",
                "title": "LESSERAFIM用中文打招呼",
                "hot": "216596",
                "url": "https://s.weibo.com/weibo?q=%23LESSERAFIM用中文打招呼%23"
            },
            {
        
                "category": "综艺",
                "title": "你好星期六",
                "hot": "206639",
                "url": "https://s.weibo.com/weibo?q=%23你好星期六%23"
            },
            {
        
                "category": "音乐",
                "title": "汪苏泷北京演唱会",
                "hot": "204911",
                "url": "https://s.weibo.com/weibo?q=%23汪苏泷北京演唱会%23"
            },
            {
        
                "category": "社会新闻",
                "title": "上海迪士尼回应禁止游客带凳子入园",
                "hot": "186105",
                "url": "https://s.weibo.com/weibo?q=%23上海迪士尼回应禁止游客带凳子入园%23"
            },
            {
        
                "category": "情感",
                "title": "会谈恋爱的女生是什么样的",
                "hot": "182973",
                "url": "https://s.weibo.com/weibo?q=%23会谈恋爱的女生是什么样的%23"
            },
            {
        
                "category": "游戏",
                "title": "DYG对战eStar",
                "hot": "181609",
                "url": "https://s.weibo.com/weibo?q=%23DYG对战eStar%23"
            },
            {
        
                "category": "游戏",
                "title": "TT战胜OMG",
                "hot": "176669",
                "url": "https://s.weibo.com/weibo?q=%23TT战胜OMG%23"
            },
            {
        
                "category": "作品衍生",
                "title": "群演谈肖战",
                "hot": "175875",
                "url": "https://s.weibo.com/weibo?q=%23群演谈肖战%23"
            },
            {
        
                "category": "幽默",
                "title": "谁能抵挡吐着粉嫩小舌头的小熊猫",
                "hot": "174833",
                "url": "https://s.weibo.com/weibo?q=%23谁能抵挡吐着粉嫩小舌头的小熊猫%23"
            },
            {
        
                "category": "音乐",
                "title": "TMEA直播",
                "hot": "173712",
                "url": "https://s.weibo.com/weibo?q=%23TMEA直播%23"
            }
        ]
    }

现在这种格式就能让前端调用了。

不过有些平台接口会限定**headers**，比如某站。如果后端请求时没有配置headers的话会请求失败。

下面是一个设置headers的例子。

HttpHeaders headers = new HttpHeaders();
    headers.set("Referer","https://www.bilibili.com/v/popular/rank/all/");
    HttpEntity httpEntity = new HttpEntity(headers);

通常情况下设置**referer**就可以了，更加复杂的需要具体分析。

### 总结 ###

目前只做了两个，某博和某站。

本来某乎也抓了，可惜只有登录后才能调那个接口，真恶心。

还有的平台没开放api，比如某瓣，现在的想法是爬虫抓，正则匹配抓下来当前热搜。不过那样感觉效率有点低，明天再瞅瞅有啥好方法吧。

最后贴一个今天刚学的前端css实现。

overflow:auto不显示滚动条

::-webkit-scrollbar {
        
            width: 0px;
    }

原文链接：[https://www.xiaoliu.life/p/20230708a][https_www.xiaoliu.life_p_20230708a]

[format_png]: https://img-blog.csdnimg.cn/img_convert/a2eb0d1511fef7a8f13b587e5787fc19.webp?x-oss-process=image/format,png
[format_png 1]: https://img-blog.csdnimg.cn/img_convert/69b3fb060a96caf488f1b2b55e2f388b.webp?x-oss-process=image/format,png
[https_www.xiaoliu.life_p_20230708a]: https://www.xiaoliu.life/p/20230708a