CLDR-18155 Order languageData's scripts by number of users #4237

conradarcturus · 2024-12-10T07:58:09Z

This sorts the scripts in the SupplementalData tags such that the first script has the highest population. This helps resolve some of the ambiguities interpreting the data.

This PR completes the ticket.

ALLOW_MANY_COMMITS=true

jira-pull-request-webhook · 2024-12-10T07:58:52Z

Notice: the branch changed across the force-push!

tools/cldr-code/src/main/java/org/unicode/cldr/tool/ConvertLanguageData.java is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

conradarcturus · 2024-12-10T08:00:13Z

common/supplemental/supplementalData.xml

@@ -1346,7 +1346,7 @@ XXX Code for transations where no currency is involved
 		<language type="awa" scripts="Deva"/>
 		<language type="awa" territories="IN" alt="secondary"/>
 		<language type="ay" scripts="Latn" territories="BO"/>
-		<language type="az" scripts="Arab Cyrl Latn" territories="AZ"/>
+		<language type="az" scripts="Arab Latn Cyrl" territories="AZ"/>


likely subtags for az is az_Latn_AZ but I will note there are more Azerbaijani speakers in Iran (az_Arab_IR). It's just that they haven't transitioned to the internet like Azerbaijan has.

While more speakers, it appears that there is more written in Latin (AZ) that Arabic (IR). So probably better Latn Arab Cyrl.

This language points out a good flaw in population estimates alone -- and it depends on the usage.

Do people want likely subtags based on the largest corpora, for active internet users, for official government mandates, for any potential user (regardless of online status), for spoken content, for local content v global content (eg. AE's largest internet usage is en_AE not ar_AE).

An interesting idea (for LikelySubtags) could be to add alt="official" and potentially other alt tags when it's different. But yes, let's just stew on this thought and return to it if/when we have a good evidence it would be useful downstream.

Strong agreement with what Mark said: despite spoken Azerbaijani being quite common in Iran with more speakers than in Azerbaijan, literacy in the writing system is pretty low in Iran, due to it not being formally taught in schools. There are books and magazines published in az-Arab in Iran, but most Iranian Azerbaijani speakers have difficulty reading them, even if they are literate in Persian (which is is theoretically a close writing system). So if anyone runs into written Azerbaijani in the wild these days, there's a much higher chance it's in Latin than in Arabic.

I also believe there's much more spoken "content" (read "media") produced in the language in Azerbaijan than in Iran (there's limited Azerbaijani radio and TV stations in Iran, and I believe less podcasts and songs produced in Iran). But either way, I don't think we want to tag such content as az-Arab or az-Latn. That's just az (or azb/azj if you insist on using ISO 639-3), and not az-Latn or az-Arab.

Thanks for the feedbacks -- in the meetings last week Mark recommended putting the script from Likely Subtags first, regardless of population. So this will now read as Latn Arab Cyrl.

common/supplemental/supplementalData.xml

conradarcturus

Added in-line comments for some languages with interesting changes. Otherwise they all match likely subtags and public records.

tools/cldr-code/src/main/java/org/unicode/cldr/util/SupplementalDataInfo.java

macchiati · 2024-12-10T15:24:02Z

tools/cldr-code/src/main/java/org/unicode/cldr/util/SupplementalDataInfo.java

+            for (String script : scripts) {
+                scriptsByPopulation.put(script, 0);
+            }
+            return setScripts(scriptsByPopulation);


Better to build once, make immutable.

I know generally what you mean but I'm not sure how you want me to do this. Do you want me to avoid calling setScripts and include the code here as well? Personally I prefer DRY code over WET.

I looked this over more, sorry for being so terse.

All the data from SupplementalDataInfo needs to be immutable; that is, you must not be able to change an object you fetch from it and have that modify what SupplementalDataInfo does.

Now, the BasicLanguageData data is collected and accessed via languageToBasicLanguageData, which is protected via

1,206: CldrUtility.protectCollection(languageToBasicLanguageData);

So as long as BasicLanguageData is Freezable, and Freeze is correctly implemented, we are in good shape.

However, you add a new field scriptsByPopulation, which is not protected in the freeze() method. If any process can get access to scriptsByPopulation, then it needs to be.

That is, if scriptsByPopulation can be accessed by any public call to a BasicLanguageData call OR it can be modified by any public call, then it needs to be protected in the freeze call.

It appears that it can be, by at least one call, addScript.

Cf. @srl295 @btangmu

jira-pull-request-webhook · 2024-12-16T10:09:08Z

Notice: the branch changed across the force-push!

common/supplemental/likelySubtags.xml is now changed in the branch
common/supplemental/supplementalData.xml is different
common/testData/localeIdentifiers/likelySubtags.txt is now changed in the branch
tools/cldr-code/src/main/java/org/unicode/cldr/tool/ConvertLanguageData.java is different
tools/cldr-code/src/main/java/org/unicode/cldr/util/DiffLanguageGroups.java is now changed in the branch
tools/cldr-code/src/main/java/org/unicode/cldr/util/SupplementalDataInfo.java is different
tools/cldr-code/src/main/resources/org/unicode/cldr/util/data/country_language_population.tsv is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

macchiati

Ok to merge. Can do fixups later if necessary

jira-pull-request-webhook · 2025-01-14T16:07:25Z

Notice: the branch changed across the force-push!

common/supplemental/likelySubtags.xml is different
common/supplemental/supplementalData.xml is different
common/testData/localeIdentifiers/likelySubtags.txt is different
tools/cldr-code/src/main/java/org/unicode/cldr/tool/ConvertLanguageData.java is different
tools/cldr-code/src/main/java/org/unicode/cldr/tool/GenerateLikelySubtags.java is different
tools/cldr-code/src/main/java/org/unicode/cldr/util/DiffLanguageGroups.java is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

jira-pull-request-webhook · 2025-01-14T16:43:40Z

Notice: the branch changed across the force-push!

tools/cldr-code/src/main/java/org/unicode/cldr/tool/ConvertLanguageData.java is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

In the latest review Mark mentioned that he linked the idea of regardless of population, the current likely subtag should be first in this list. This change makes that happen. This mostly affects historic languages but also a few with known overrides to their scripts. Changed languages: * Azerbaijani [az] gets Latin first. There are more Azerbaijani speakers in Iran (thereby using the Arabic script) but the online precense of Azeris is centered in Azerbaijan (thereby Latin script) * Lingua Franca Nova [lfn], a constructed language, gets Latin first * Panjabi [pa] (currently likely subtag to Gurmukhī script in India) no longer has Perso-Arabic first, because of online usage is different than populations * Pali [pi] gets Sinhala first in the secondary tags -- in fact though I need to fix this with my other script changes * Samaritan Aramaic [sam], an extinct language gets the Samaritan alphabet first * Old Irish [sga] gets Ogham writing first * Umbrian [xum] swaps to get Latin first but I'm also not sure this is correct -- but it is the current value in Likely subtags so I'll leave it be (for now).

jira-pull-request-webhook · 2025-01-14T16:43:53Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

macchiati · 2025-01-14T17:32:12Z

common/supplemental/supplementalData.xml

@@ -2198,9 +2198,9 @@ XXX Code for transations where no currency is involved
 		<language type="sel" scripts="Cyrl"/>
 		<language type="ses" scripts="Latn"/>
 		<language type="sg" scripts="Latn" territories="CF"/>
-		<language type="sga" scripts="Latn Ogam" alt="secondary"/>
+		<language type="sga" scripts="Ogam Latn" alt="secondary"/>


Does this take into account the volume of Old Irish text that is in Latn? (It may, just asking.)
The primary goal is to reflect current usage.

Thanks for taking a look. I believe you are correct, the likely subtag for Old Irish [sga] is wrong, it should be Latn not Ogam -- do you want me to fix that now or as part of the task to audit primary scripts (and thereby likely subtags) https://unicode-org.atlassian.net/browse/CLDR-18121

If we are sure the fixes can be applied before release, it's fine to do them in a separate ticket.

Er, I'll fix Old Irish RIGHT now since we are close to the v47 cut.

Ah, even if Nko is a minority script it still has to be regarded as the highest with other data we have.

jira-pull-request-webhook · 2025-01-15T00:13:05Z

Notice: the branch changed across the force-push!

common/supplemental/likelySubtags.xml is different
common/supplemental/supplementalData.xml is different
common/testData/localeIdentifiers/likelySubtags.txt is no longer changed in the branch
tools/cldr-code/src/main/java/org/unicode/cldr/tool/GenerateLikelySubtags.java is different
tools/cldr-code/src/main/resources/org/unicode/cldr/util/data/country_language_population.tsv is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

macchiati · 2025-01-15T02:16:13Z

Note: I filed a related ticket https://unicode-org.atlassian.net/browse/CLDR-18224

github-actions bot assigned conradarcturus Dec 10, 2024

conradarcturus force-pushed the CLDR-18155-Order-languageData-scripts-by-usage branch from 757c108 to cbce758 Compare December 10, 2024 07:58

conradarcturus commented Dec 10, 2024

View reviewed changes

common/supplemental/supplementalData.xml Outdated Show resolved Hide resolved

conradarcturus commented Dec 10, 2024

View reviewed changes

common/supplemental/supplementalData.xml Show resolved Hide resolved

conradarcturus commented Dec 10, 2024

View reviewed changes

macchiati reviewed Dec 10, 2024

View reviewed changes

tools/cldr-code/src/main/java/org/unicode/cldr/util/SupplementalDataInfo.java Outdated Show resolved Hide resolved

macchiati reviewed Dec 10, 2024

View reviewed changes

macchiati previously approved these changes Dec 10, 2024

View reviewed changes

conradarcturus marked this pull request as ready for review December 16, 2024 10:08

conradarcturus dismissed macchiati’s stale review via f03fd8c December 16, 2024 10:09

conradarcturus force-pushed the CLDR-18155-Order-languageData-scripts-by-usage branch from cbce758 to f03fd8c Compare December 16, 2024 10:09

macchiati previously approved these changes Dec 16, 2024

View reviewed changes

conradarcturus dismissed macchiati’s stale review via 341ff39 January 14, 2025 16:07

conradarcturus force-pushed the CLDR-18155-Order-languageData-scripts-by-usage branch from f03fd8c to 341ff39 Compare January 14, 2025 16:07

conradarcturus force-pushed the CLDR-18155-Order-languageData-scripts-by-usage branch from 341ff39 to 0a85ccb Compare January 14, 2025 16:43

conradarcturus added 3 commits January 14, 2025 08:43

CLDR-18155 Order languageData's scripts by number of users

a4930df

CLDR-18155 Recommended edits

3b576cd

conradarcturus force-pushed the CLDR-18155-Order-languageData-scripts-by-usage branch from 0a85ccb to 7e09402 Compare January 14, 2025 16:43

macchiati requested changes Jan 14, 2025

View reviewed changes

conradarcturus requested a review from macchiati January 14, 2025 19:53

conradarcturus added 2 commits January 14, 2025 13:19

CLDR-18155 Set Latin as the default script for Old Irish

f58182d

CLDR-18155 Fix Mandingo / Nkoo population

a1ea853

Ah, even if Nko is a minority script it still has to be regarded as the highest with other data we have.

conradarcturus force-pushed the CLDR-18155-Order-languageData-scripts-by-usage branch from 5189e1d to a1ea853 Compare January 15, 2025 00:12

macchiati approved these changes Jan 15, 2025

View reviewed changes

conradarcturus merged commit a0858ea into main Jan 15, 2025
16 checks passed

conradarcturus deleted the CLDR-18155-Order-languageData-scripts-by-usage branch January 15, 2025 18:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLDR-18155 Order languageData's scripts by number of users #4237

CLDR-18155 Order languageData's scripts by number of users #4237

conradarcturus commented Dec 10, 2024

jira-pull-request-webhook bot commented Dec 10, 2024

conradarcturus Dec 10, 2024

macchiati Dec 16, 2024

conradarcturus Dec 16, 2024

roozbehp Dec 16, 2024 •

edited

Loading

conradarcturus Jan 14, 2025

conradarcturus left a comment

macchiati Dec 10, 2024

conradarcturus Dec 16, 2024

macchiati Jan 14, 2025

jira-pull-request-webhook bot commented Dec 16, 2024

macchiati left a comment

jira-pull-request-webhook bot commented Jan 14, 2025

jira-pull-request-webhook bot commented Jan 14, 2025

jira-pull-request-webhook bot commented Jan 14, 2025

macchiati Jan 14, 2025

conradarcturus Jan 14, 2025

macchiati Jan 14, 2025

conradarcturus Jan 14, 2025

jira-pull-request-webhook bot commented Jan 15, 2025

macchiati commented Jan 15, 2025

CLDR-18155 Order languageData's scripts by number of users #4237

CLDR-18155 Order languageData's scripts by number of users #4237

Conversation

conradarcturus commented Dec 10, 2024

jira-pull-request-webhook bot commented Dec 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roozbehp Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

conradarcturus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jira-pull-request-webhook bot commented Dec 16, 2024

macchiati left a comment

Choose a reason for hiding this comment

jira-pull-request-webhook bot commented Jan 14, 2025

jira-pull-request-webhook bot commented Jan 14, 2025

jira-pull-request-webhook bot commented Jan 14, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jira-pull-request-webhook bot commented Jan 15, 2025

macchiati commented Jan 15, 2025

roozbehp Dec 16, 2024 •

edited

Loading