diff --git a/docs/bugs/872_error_when_parsing_Junit_raport_using_TeamCityCI.md b/docs/bugs/872_error_when_parsing_Junit_raport_using_TeamCityCI.md new file mode 100644 index 0000000000..2da3ff16a0 --- /dev/null +++ b/docs/bugs/872_error_when_parsing_Junit_raport_using_TeamCityCI.md @@ -0,0 +1,185 @@ +# Non-deterministic issue when parsing JUnitReport.xml generated by Flank 20.06.2 [#872](https://github.com/Flank/flank/issues/872) + + + +## Description + +After updating from Flank 8.0.1 to 20.06.2 and switching to server-side sharding sometimes on our CI (TeamCity) we see following error when parsing `JUnitReport.xml`: +``` +[17:00:00]Ant JUnit report watcher +[17:00:00][Ant JUnit report watcher] 1 report found for paths: +[17:00:00][Ant JUnit report watcher] results/**/JUnitReport.xml +[17:00:00][Ant JUnit report watcher] Parsing errors +[17:00:00][Parsing errors] Failed to parse 1 report +[17:00:00][Parsing errors] results/2020-06-25_16-47-33.675000_QnCW/JUnitReport.xml: Content is not allowed in prolog. +[17:00:00][Parsing errors] jetbrains.buildServer.util.XmlXppAbstractParser$3: Content is not allowed in prolog. + at jetbrains.buildServer.util.XmlXppAbstractParser.parse(XmlXppAbstractParser.java:39) + at jetbrains.buildServer.util.XmlXppAbstractParser.parse(XmlXppAbstractParser.java:31) + at jetbrains.buildServer.xmlReportPlugin.parsers.antJUnit.AntJUnitReportParser.parse(AntJUnitReportParser.java:179) + at jetbrains.buildServer.xmlReportPlugin.ParseReportCommand.run(ParseReportCommand.java:62) + at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) + at java.util.concurrent.FutureTask.run(FutureTask.java:266) + at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) + at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) + at java.lang.Thread.run(Thread.java:748) +Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog. + at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) + at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) + at jetbrains.buildServer.util.XmlXppAbstractParser.parseWithSAX(XmlXppAbstractParser.java:240) + at jetbrains.buildServer.util.XmlXppAbstractParser.parse(XmlXppAbstractParser.java:37) + ... 8 more +``` +The issue is non-deterministic, in most cases the report gets parsed, but this never happened to us when using Flank 8. Did anyone already run into the problem? Any ideas what could be a cause? The XML files look correct for me, I compared two files from failed and successful runs, the difference is only in reported time of running tests, both files do not contain BOM marker (checked using vim). + +**To Reproduce** + +1. Run tests using Flank and then try to parse the report using TeamCity's report parsing feature. + +**Expected behavior** + +No error raised by TeamCity when parsing the `JUnitReport.xml`. + +**Details (please complete the following information):** + +Flank 20.06.2 + +##### Additional info + +Asked user for providing raport which fails on TeamCity CI and he replied with failing report fail: + +```xml + + + + + + + + +``` + + + +## Problem investigation techniques + +### Validate XML + +#### Description + +Attached XML report was checked using some online tools to validate XML files + +#### Execution + +Following sites were used to validate XML (https://www.xmlvalidation.com/, https://www.w3schools.com/xml/xml_validator.asp, https://www.freeformatter.com/xml-validator-xsd.html, https://codebeautify.org/xmlvalidator, https://www.liquid-technologies.com/online-xml-validator) + +#### Results + +All tools confirm that provided XML report is valid + + + +### Clear XML buffer unnecessary characters + +#### Description + +During research about stack trace provided in the task description, user suggest on [stack overflow ](https://stackoverflow.com/questions/3030903/content-is-not-allowed-in-prolog-when-parsing-perfectly-valid-xml-on-gae/3030913#3030913)that some XML files could contain bad whitespace characters which are bytes order marked + +#### Execution + +1. Change `JUnitXML.kt` file to trim characters suggested in post + + ```kotlin + fun JUnitTestResult?.xmlToString(): String { + if (this == null) return "" + val prefix = "" + + return (prefix + System.lineSeparator() + xmlPrettyWriter.writeValueAsString(this)).clearByteOrderMarkersFromBuffer() + } + + // According to occasional problem with parsing byte order markers should be cleared from buffer + // https://stackoverflow.com/a/3030913 + private fun String.clearByteOrderMarkersFromBuffer() = trim().replaceFirst("^([\\W]+)<","<") + ``` + +2. Try following code when reading file and compare it without using option to clear buffer + +#### Results + +No characters were deleted, XML has the same length with and without suggested "fix" + + + +### Checking hex code of provided raport + +#### Description + +XML report hex code was checked to make sure that no unnecessary characters are added to the file. + +#### Execution + +1. The report was parsed to hex representation of chars and then in revert side. Used websites (https://tomeko.net/online_tools/file_to_hex.php, https://tomeko.net/online_tools/hex_to_file.php) +2. The file was checked using [online hex viewer](https://www.onlinehexeditor.com/) + +#### Results + +No unnecessary characters found on file. + + + +### Compare reading report with using Kotlin file loading + +#### Description + +By using `java.nio.File("JUnitReport.xml")` methods `readLines()`/ `length()`/`readText()` checking is provided file is different than generated similar one (also with clearing byte order markers) + +#### Results + +No issues found in the report + + + +### Checking git history + +#### Description + +Checked git history of XML reporting code. + +#### Results + +No breaking changes were introduced between Flank 8.0.1 and 20.06.2 + + + +### Ask user about TeamCity configuration + +#### Description + +User was asked about agents count and configuration + +#### Results + +All agents are docker's containers so they are always the same and the problem isn't here. + +User have 2 types of tests + +1. Using MockWebServer and have ~140 tests executed on multiple shards +2. Using real backend, not many tests executed on single shards - this type of tests sometime falling + +When user back to old configuration, he can still reproduce problem + +Between versions user changed config from: +`num-uniform-shards: 1` +to +`num-test-runs: 1` + +But when user back to old configuration, he can still reproduce problem. + +## Current situation + +#### Issue reported to TeamCity https://youtrack.jetbrains.com/issue/TW-66753 + +TeamCity team cannot reproduce the problem, thay asked about additional logs, but User could deliver them in next week (starting 13-07-2020), until that we must wait. + + + +#### User is testing TeamCity CI with Flank 8 to make sure that the problem is related to Flank