selenium模拟浏览器操作时 element is not attached to the page document 等异常的处理

selenium模拟浏览器操作时 element is not attached to the page document 等异常的处理

问题场景

利用selenium、chromedriver模拟浏览器操作网页。一套共280多个页面，循环获取页面元素内容，有输入、跳转等动作

问题现象
主要出现在以下这段代码处，打开页面后，获取页面上样式表为“.el-input__inner”的内容。

...
WebDriver w = new ChromeDriver()
w.get(baseUrl);
....//输入框、模拟输入搜索词
WebElement search = w.findElement(By.cssSelector(".el-input__inner"));
...

如果网络不正常、url访问很慢、页面访问排版错乱的情况发生就会出现以下错误

错误一

org.openqa.selenium.StaleElementReferenceException: stale element reference: element is not attached to the page document(Session info: chrome=67.0.3396.87)(Driver info: chromedriver=2.38.552518 (183d19265345f54ce39cbb94cf81ba5f15905011),platform=Mac OS X 10.13.6 x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds
For documentation on this error, please visit: .html
Build info: version: '3.9.1', revision: '63f7b50', time: '2018-02-07T22:25:02.294Z'
System info: host: 'chenhailongdeMacBook-Pro.local', ip: 'fe80:0:0:0:c1f:5ed9:ab23:ed6b%en0', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.13.6', java.version: '1.8.0_171'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities {acceptInsecureCerts: false, acceptSslCerts: false, applicationCacheEnabled: false, browserConnectionEnabled: false, browserName: chrome, chrome: {chromedriverVersion: 2.38.552518 (, userDataDir: /var/folders/}, cssSelectorsEnabled: true, databaseEnabled: false, handlesAlerts: true, hasTouchScreen: false, javascriptEnabled: true, locationContextEnabled: true, mobileEmulationEnabled: false, nativeEvents: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: MAC, platformName: MAC, rotatable: false, setWindowRect: true, takesHeapSnapshot: true, takesScreenshot: true, unexpectedAlertBehaviour: , unhandledPromptBehavior: , version: 67.0.3396.87, webStorageEnabled: true}
Session ID: e9c1ee0bf9fc2354b7c45f2ae1dc0ceb
*** Element info: {Using=xpath, value=div}wInstance0(Native Method)wInstance(NativeConstructorAccessorImpl.java:62)wInstance(DelegatingConstructorAccessorImpl.java:45)at wInstance(Constructor.java:423)at org.ateThrowable(ErrorHandler.java:214)at org.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166)at org.structValue(JsonHttpResponseCodec.java:40)at org.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:80)at org.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:44)at org.ute(HttpCommandExecutor.java:160)at org.ute(DriverCommandExecutor.java:83)at org.ute(RemoteWebDriver.java:601)at org.ute(RemoteWebElement.java:279)at org.RemoteWebElement.findElement(RemoteWebElement.java:179)at org.RemoteWebElement.findElementByXPath(RemoteWebElement.java:255)at org.openqa.selenium.By$ByXPath.findElement(By.java:361)at org.RemoteWebElement.findElement(RemoteWebElement.java:175)at com.chl.webmagic.processor.LawInnerProcessor.process(LawInnerProcessor.java:128)decraft.webmagic.Spider.processRequest(Spider.java:416)decraft.webmagic.Spider$1.run(Spider.java:321)decraft.webmagic.thread.CountableThreadPool$1.run(CountableThreadPool.java:74)at urrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at urrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)

错误二

org.openqa.selenium.WebDriverException: unknown error: Element <li class="number">...</li> is not clickable at point (365, 647). Other element would receive the click: <div class="el-loading-mask el-loading-fade-leave el-loading-fade-leave-active" style="">...</div>(Session info: chrome=67.0.3396.87)(Driver info: chromedriver=2.38.552518 (183d19265345f54ce39cbb94cf81ba5f15905011),platform=Mac OS X 10.13.6 x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds
Build info: version: '3.9.1', revision: '63f7b50', time: '2018-02-07T22:25:02.294Z'
System info: host: 'chenhailongdeMacBook-Pro.local', ip: 'fe80:0:0:0:c1f:5ed9:ab23:ed6b%en0', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.13.6', java.version: '1.8.0_171'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities {acceptInsecureCerts: false, acceptSslCerts: false, applicationCacheEnabled: false, browserConnectionEnabled: false, browserName: chrome, chrome: {chromedriverVersion: 2.38.552518 (, userDataDir: /var/folders/}, cssSelectorsEnabled: true, databaseEnabled: false, handlesAlerts: true, hasTouchScreen: false, javascriptEnabled: true, locationContextEnabled: true, mobileEmulationEnabled: false, nativeEvents: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: MAC, platformName: MAC, rotatable: false, setWindowRect: true, takesHeapSnapshot: true, takesScreenshot: true, unexpectedAlertBehaviour: , unhandledPromptBehavior: , version: 67.0.3396.87, webStorageEnabled: true}
Session ID: wInstance0(Native Method)wInstance(NativeConstructorAccessorImpl.java:62)wInstance(DelegatingConstructorAccessorImpl.java:45)at wInstance(Constructor.java:423)at org.ateThrowable(ErrorHandler.java:214)at org.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166)at org.structValue(JsonHttpResponseCodec.java:40)at org.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:80)at org.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:44)at org.ute(HttpCommandExecutor.java:160)at org.ute(DriverCommandExecutor.java:83)at org.ute(RemoteWebDriver.java:601)at org.ute(RemoteWebElement.java:279)at org.RemoteWebElement.click(RemoteWebElement.java:83)at com.chl.webmagic.processor.LawInnerProcessor.process(LawInnerProcessor.java:149)decraft.webmagic.Spider.processRequest(Spider.java:416)decraft.webmagic.Spider$1.run(Spider.java:321)decraft.webmagic.thread.CountableThreadPool$1.run(CountableThreadPool.java:74)at urrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at urrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)

问题三

org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"css selector","selector":".el-input__inner"}(Session info: chrome=67.0.3396.87)(Driver info: chromedriver=2.38.552518 (183d19265345f54ce39cbb94cf81ba5f15905011),platform=Mac OS X 10.13.6 x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds
For documentation on this error, please visit: .html
Build info: version: '3.9.1', revision: '63f7b50', time: '2018-02-07T22:25:02.294Z'
System info: host: 'chenhailongdeMacBook-Pro.local', ip: 'fe80:0:0:0:c1f:5ed9:ab23:ed6b%en0', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.13.6', java.version: '1.8.0_171'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities {acceptInsecureCerts: false, acceptSslCerts: false, applicationCacheEnabled: false, browserConnectionEnabled: false, browserName: chrome, chrome: {chromedriverVersion: 2.38.552518 (, userDataDir: /var/folders/}, cssSelectorsEnabled: true, databaseEnabled: false, handlesAlerts: true, hasTouchScreen: false, javascriptEnabled: true, locationContextEnabled: true, mobileEmulationEnabled: false, nativeEvents: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: MAC, platformName: MAC, rotatable: false, setWindowRect: true, takesHeapSnapshot: true, takesScreenshot: true, unexpectedAlertBehaviour: , unhandledPromptBehavior: , version: 67.0.3396.87, webStorageEnabled: true}
Session ID: e7ad3e18335ec9aa43f55bfdc83bcadf
*** Element info: {Using=css selector, value=.el-input__inner}wInstance(Unknown Source)wInstance(DelegatingConstructorAccessorImpl.java:45)at wInstance(Constructor.java:423)at org.ateThrowable(ErrorHandler.java:214)at org.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166)at org.structValue(JsonHttpResponseCodec.java:40)at org.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:80)at org.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:44)at org.ute(HttpCommandExecutor.java:160)at org.ute(DriverCommandExecutor.java:83)at org.ute(RemoteWebDriver.java:601)at org.RemoteWebDriver.findElement(RemoteWebDriver.java:371)at org.RemoteWebDriver.findElementByCssSelector(RemoteWebDriver.java:465)at org.openqa.selenium.By$ByCssSelector.findElement(By.java:430)at org.RemoteWebDriver.findElement(RemoteWebDriver.java:363)at com.chl.webmagic.processor.LawInnerProcessor.process(LawInnerProcessor.java:73)decraft.webmagic.Spider.processRequest(Spider.java:416)decraft.webmagic.Spider$1.run(Spider.java:321)decraft.webmagic.thread.CountableThreadPool$1.run(CountableThreadPool.java:74)at urrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at urrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)

问题处理

我这里出现问题的主要原因有两个, 1.页面元素渲染慢，还没有正常加载完数据内容，就开始执行**w.findElement(By.cssSelector(".el-input__inner"));**逻辑。 2.页面样式错乱,加载错乱。如一些动态js绑定的操作是在页面加载后立马执行的，但是一些原因下没有生效。导致selenium无法执行操作，无法获取元素等。

处理方式，首先是增加线程等待，
Thread.sleep(8 * 1000);保证在获取元素时页面已经加载完毕，但是这种处理会让整个处理周期变长。另一个的办法，就是观察页面特征元素，确保一个元素是页面加载完必有的。通过判断是否存在来确定是否加载完成，如果没有完成执行刷新页面操作(针对加载不完全的处理)。