#872 Added documentation about investigating #872 issue (#877)

* #872 Added documentation about investigating #872 issue * #872 added info about raising issue to TeamCity * #872 update status info
Flank · Jul 10, 2020 · 38e965e · 38e965e
1 parent 8ee1317
commit 38e965e
Showing 1 changed file with 185 additions and 0 deletions.
diff --git a/docs/bugs/872_error_when_parsing_Junit_raport_using_TeamCityCI.md b/docs/bugs/872_error_when_parsing_Junit_raport_using_TeamCityCI.md
@@ -0,0 +1,185 @@
+# Non-deterministic issue when parsing JUnitReport.xml generated by Flank 20.06.2 [#872](https://github.com/Flank/flank/issues/872)
+
+
+
+## Description
+
+After updating from Flank 8.0.1 to 20.06.2 and switching to server-side sharding sometimes on our CI (TeamCity) we see following error when parsing `JUnitReport.xml`:
+```
+[17:00:00]Ant JUnit report watcher
+[17:00:00][Ant JUnit report watcher] 1 report found for paths:
+[17:00:00][Ant JUnit report watcher] results/**/JUnitReport.xml
+[17:00:00][Ant JUnit report watcher] Parsing errors
+[17:00:00][Parsing errors] Failed to parse 1 report
+[17:00:00][Parsing errors] results/2020-06-25_16-47-33.675000_QnCW/JUnitReport.xml: Content is not allowed in prolog.
+[17:00:00][Parsing errors] jetbrains.buildServer.util.XmlXppAbstractParser$3: Content is not allowed in prolog.
+	at jetbrains.buildServer.util.XmlXppAbstractParser.parse(XmlXppAbstractParser.java:39)
+	at jetbrains.buildServer.util.XmlXppAbstractParser.parse(XmlXppAbstractParser.java:31)
+	at jetbrains.buildServer.xmlReportPlugin.parsers.antJUnit.AntJUnitReportParser.parse(AntJUnitReportParser.java:179)
+	at jetbrains.buildServer.xmlReportPlugin.ParseReportCommand.run(ParseReportCommand.java:62)
+	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
+	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
+	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
+	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
+	at java.lang.Thread.run(Thread.java:748)
+Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
+	at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
+	at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
+	at jetbrains.buildServer.util.XmlXppAbstractParser.parseWithSAX(XmlXppAbstractParser.java:240)
+	at jetbrains.buildServer.util.XmlXppAbstractParser.parse(XmlXppAbstractParser.java:37)
+	... 8 more
+```
+The issue is non-deterministic, in most cases the report gets parsed, but this never happened to us when using Flank 8. Did anyone already run into the problem? Any ideas what could be a cause? The XML files look correct for me, I compared two files from failed and successful runs, the difference is only in reported time of running tests, both files do not contain BOM marker (checked using vim).
+
+**To Reproduce**
+
+1. Run tests using Flank and then try to parse the report using TeamCity's report parsing feature.
+
+**Expected behavior**
+
+No error raised by TeamCity when parsing the `JUnitReport.xml`.
+
+**Details (please complete the following information):**
+
+Flank 20.06.2
+
+##### Additional info
+
+Asked user for providing raport which fails on TeamCity CI and he replied with failing report fail:
+
+```xml
+<?xml version='1.0' encoding='UTF-8' ?>
+<testsuites>
+  <testsuite name="Pixel2-27-en-portrait" tests="2" failures="0" flakes="0" errors="0" skipped="0" time="39.151" timestamp="2020-07-01T16:46:41" hostname="localhost">
+    <testcase name="loginSuccessBetaEnvTest" classname="com.miquido.play360.login.LoginBetaEnvTest" time="19.133"/>
+    <testcase name="testPaymentsCard" classname="com.miquido.play360.dashboard.PaymentCardBetaEnvTest" time="20.019"/>
+  </testsuite>
+</testsuites>
+
+```
+
+
+
+## Problem investigation techniques
+
+### Validate XML
+
+#### Description
+
+Attached XML report was checked using some online tools to validate XML files
+
+#### Execution
+
+Following sites were used to validate XML (https://www.xmlvalidation.com/, https://www.w3schools.com/xml/xml_validator.asp, https://www.freeformatter.com/xml-validator-xsd.html, https://codebeautify.org/xmlvalidator, https://www.liquid-technologies.com/online-xml-validator)
+
+#### Results
+
+All tools confirm that provided XML report is valid
+
+
+
+### Clear XML buffer unnecessary characters
+
+#### Description
+
+During research about stack trace provided in the task description, user suggest on [stack overflow ](https://stackoverflow.com/questions/3030903/content-is-not-allowed-in-prolog-when-parsing-perfectly-valid-xml-on-gae/3030913#3030913)that some XML files could contain bad whitespace characters which are bytes order marked
+
+#### Execution
+
+1. Change `JUnitXML.kt` file to trim characters suggested in post
+
+   ```kotlin
+   fun JUnitTestResult?.xmlToString(): String {
+       if (this == null) return ""
+       val prefix = "<?xml version='1.0' encoding='UTF-8'?>"
+
+       return (prefix + System.lineSeparator() + xmlPrettyWriter.writeValueAsString(this)).clearByteOrderMarkersFromBuffer()
+   }
+
+   // According to occasional problem with parsing byte order markers should be cleared from buffer
+   // https://stackoverflow.com/a/3030913
+   private fun String.clearByteOrderMarkersFromBuffer() = trim().replaceFirst("^([\\W]+)<","<")
+   ```
+
+2. Try following code when reading file and compare it without using option to clear buffer
+
+#### Results
+
+No characters were deleted, XML has the same length with and without suggested "fix"
+
+
+
+### Checking hex code of provided raport
+
+#### Description
+
+XML report hex code was checked to make sure that no unnecessary characters are added to the file.
+
+#### Execution
+
+1. The report was parsed to hex representation of chars and then in revert side. Used websites (https://tomeko.net/online_tools/file_to_hex.php, https://tomeko.net/online_tools/hex_to_file.php)
+2. The file was checked using [online hex viewer](https://www.onlinehexeditor.com/) 
+
+#### Results
+
+No unnecessary characters found on file.
+
+
+
+### Compare reading report with using Kotlin file loading
+
+#### Description
+
+By using `java.nio.File("JUnitReport.xml")` methods `readLines()`/ `length()`/`readText()` checking is provided file is different than generated similar one (also with clearing byte order markers)
+
+#### Results
+
+No issues found in the report
+
+
+
+### Checking git history
+
+#### Description
+
+Checked git history of XML reporting code.
+
+#### Results
+
+No breaking changes were introduced between Flank 8.0.1 and 20.06.2
+
+
+
+### Ask user about TeamCity configuration
+
+#### Description
+
+User was asked about agents count and configuration 
+
+#### Results
+
+All agents are docker's containers so they are always the same and the problem isn't here. 
+
+User have 2 types of tests
+
+1. Using MockWebServer and have ~140 tests executed on multiple shards
+2. Using real backend, not many tests executed on single shards - this type of tests sometime falling
+
+When user back to old configuration, he can still reproduce problem
+
+Between versions user changed config from:
+`num-uniform-shards: 1`
+to
+`num-test-runs: 1`
+
+But when user back to old configuration, he can still reproduce problem.
+
+## Current situation
+
+#### Issue reported to TeamCity https://youtrack.jetbrains.com/issue/TW-66753
+
+TeamCity team cannot reproduce the problem, thay asked about additional logs, but User could deliver them in next week (starting 13-07-2020), until that we must wait.
+
+
+
+#### User is testing TeamCity CI with Flank 8 to make sure that the problem is related to Flank