课程源码、课件及课后作业地址：

最近想爬一个aspx网站，发现aspx网站爬虫方法挺复杂的，网上介绍都挺笼统，请问老师有没有这方面比较详细的教程资料？

for title_href in soup.find_all('div', class_='news_type_block'): print([title.get('title') for title in title_href.find_all('a') if title.get('title')]) 最下面两行是啥语法？

请问老师，那个headers={}中的内容怎么获取呀？

老师我想问下headers是如何伪造的，是打开F12，在network中随便找个请求将其中的headers拷贝下来使用嘛

project interpreter里已经装了bs4，但总是报错ModuleNotFoundError: No module named 'bs4'是为什么呢？

网页改了，div用items__content，两个双下划线的： def craw2(url): response = requests.get(url, headers=headers) # print(response.text) soup = BeautifulSoup(response.text, 'lxml') for title_href in soup.find_all('div', class_='items__content'): for title in title_href.find_all('a'): if title.get('title'): print(title.get('title')) def my_thread(current_url, page_number): craw2(current_url) print('上面为第%s页的数据\n\n\n\n' % (page_number)) for i in range(0, 46, 15): if i == 0: url = 'http://www.infoq.com/news' else: url = 'http://www.infoq.com/news' + str(i) t1 = threading.Thread(target=my_thread, args=(url, i)) t1.start()

这个网页已经爬不了啦

老师，网站的数据如果是页面打开后，通过JS调用接口去生成的，是不是不能使用这种方式

现在运行没有打印数据，是不是别人做了反爬虫？萌新求解

下面是 infoq 改版后网页的获取内容的代码，但是得到的内容并不是浏览器查看html的内容，而是十分少的内容，还有乱麻。请问怎么回事呢？ --- from bs4 import BeautifulSoup import requests header_i = { "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3", "Accept-Encoding": "gzip, deflate, br", "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8", "Cache-Control": "no-cache", "Connection": "keep-alive", "Cookie": "_ga=GA1.2.236308595.1542204557; _itt=1; GCID=7d9c08f-e052716-92486ca-1ef06ad-cd; GCESS=BAQEAC8NAAMEbUWUXAIEbUWUXAEEmV4PAAoEAAAAAAYE1hDl2gcEjYGeiwkBAQgBAwsCBAAMAQEFBAAAAAA-; Hm_lvt_094d2af1d9a57fd9249b3fa259428445=1553224053; Hm_lpvt_094d2af1d9a57fd9249b3fa259428445=1553227368; SERVERID=1fa1f330efedec1559b3abbcb6e30f50|1553227540|1553224054", "DNT": "1", "Host": "www.infoq.cn", "Pragma": "no-cache", "Upgrade-Insecure-Requests": "1", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36", } url = 'https://www.infoq.cn' response = requests.get(url, headers=header_i) soup = BeautifulSoup(response.text, 'lxml') print(soup.prettify())