CLDR-17014 Illustrate/fix extra-path bug

[btangmu]

-Writing a CLDRFile to XML failed for extra paths such as this metazone path as shown by the new test in TestSTFactory.xml

-Using cldrFile.fullIterable.forEach instead of xmlSource.forEach in CldrXmlWriter fixes the bug

CLDR-17014

• This PR completes the ticket.

ALLOW_MANY_COMMITS=true

[btangmu]

The same test failure occurs here that I saw locally:

Error: TestSTFactory.TestVettingDataDriven:469->runDataDrivenTest:492 93f Expected from XML aksalA at mul://ldml/dates/timeZoneNames/metazone[@type="Alaska"]/long/generic got null

The failure occurs in this code in DataDrivenSTTestHandler.java:

        // Verify from XML
        File outFile = new File(targDir, locale.getBaseName() + ".xml");
        if (outFile.exists()) outFile.delete();
        PrintWriter pw;
        pw = FileUtilities.openUTF8Writer(targDir.getAbsolutePath(), locale.getBaseName() + ".xml");
        cf.write(pw, TestSTFactory.noDtdPlease);
        pw.close();

        // System.out.println("Read back..");
        CLDRFile readBack = null;
        readBack = CLDRFile.loadFromFile(outFile, locale.getBaseName(), DraftStatus.unconfirmed);
        String reRead = readBack.getStringValue(xpath);
        String fullXpathBack = readBack.getFullXPath(xpath);
        Status xpathStatusBack;
        if (fullXpathBack == null || itMoved) {
            xpathStatusBack = Status.missing;
        } else {
            XPathParts xpp2 = XPathParts.getFrozenInstance(fullXpathBack);
            String statusFromXpathBack = xpp2.getAttributeValue(-1, "draft");

            if (statusFromXpathBack == null) {
                statusFromXpathBack = "approved"; // no draft =
                // approved
            }
            xpathStatusBack = Status.fromString(statusFromXpathBack);
        }

        if (value == null && reRead != null) {
            fail(
                    pathCount
                            + "e Expected null value from XML at "
                            + locale
                            + ":"
                            + xpath
                            + " got "
                            + reRead);
        } else if (value != null && !value.equals(reRead)) {
            fail(
                    pathCount
                            + "f Expected from XML "
                            + value
                            + " at "
                            + locale
                            + ":"
                            + xpath
                            + " got "
                            + reRead);

The error message includes "f Expected from XML ...".

The failure happens where it calls cf.write(pw, TestSTFactory.noDtdPlease) which invokes:

    public CldrXmlWriter(CLDRFile cldrFile, PrintWriter pw, Map<String, ?> options) {
        this.options = options;
        this.pw = pw;
        this.cldrFile = cldrFile;

        xmlSource = cldrFile.dataSource;
        orderedSet = new TreeSet<>(cldrFile.getComparator());
        xmlSource.forEach(orderedSet::add);

I think orderedSet is the set of paths to be written, and it does not include extra paths since it only derives from cldrFile.dataSource, skipping cldrFile.extraPaths. A solution might involve calling cldrFile.fullIterable instead.

Notice this old comment in CLDRFile:

XMLSource dataSource; // TODO(jchye): make private

In general it may be problematic that dataSource isn't private, and also that CLDRFile implements Iterable<String> but this method of iteration also doesn't include extra paths, for which cldrFile.fullIterable needs to be called explicitly. It might be better if the distinction between extra paths and regular paths were more encapsulated, and most code would get the full set of paths by default.

[btangmu]

The second commit to this PR fixes the test failure introduced by the first commit

[btangmu]

Testing in ST, without the changes in this PR, suggests that if I vote (as Admin) for a value for an extra path that had no value, and then I generate VXML, that path fails to be written. However, if I restart ST between voting and generating VXML, the path is written. Seemingly loadVoteValues (which calls internalSetVoteForValue, which calls makeSureInPathsForFile) in STFactory.java accounts for loading the votes when a locale is first opened after starting ST. It's maybe not surprising then that the bug hasn't been evident before when generating VXML.

[btangmu]

@macchiati shall I switch the ticket from Design to Accepted, and merge this? Seems like the design -- to handle language paths similarly to timezone paths, as extra paths -- has turned out to be viable, this bug fix being the last hang-up I'm aware of. The other PR, with the lion's share of changes, could then be merged after a little more testing: #4254

[macchiati]

Yes, let's do that.

…

On Fri, Jan 10, 2025 at 3:39 PM Tom Bishop ***@***.***> wrote: @macchiati <https://github.com/macchiati> shall I switch the ticket from Design to Accepted, and merge this? Seems like the design -- to handle language paths similarly to timezone paths, as extra paths -- has turned out to be viable, this bug fix being the last hang-up I'm aware of. The other PR, with the lion's share of changes, could then be merged after a little more testing: #4254 <#4254> — Reply to this email directly, view it on GitHub <#4256 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMCLN5BCBEPFSHAIFFT2KBK3VAVCNFSM6AAAAABUFBAQ7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOBUHA3DQOBWG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>