Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ICU-21480 update pom.xml to point to CLDR maven #1

Open
wants to merge 100 commits into
base: ICU-21480-brs69rc-integrateCLDR-39-alpha0
Choose a base branch
from

Conversation

srl295
Copy link

@srl295 srl295 commented Feb 2, 2021

  • updated README as well
  • deleted install-cldr-jars.sh

@srl295
Copy link
Author

srl295 commented Feb 2, 2021

@pedberg-icu following the instructions in icu/tools/cldr/cldr-to-icu/README.txt I get this error:

$ ant -f build-icu-data.xml 
Buildfile: /Users/srl295/src/icu/tools/cldr/cldr-to-icu/build-icu-data.xml

init-args:

prepare-jar:
     [exec] [INFO] Scanning for projects...
     [exec] [INFO] 
     [exec] [INFO] --------------------< org.unicode.icu:cldr-to-icu >---------------------
     [exec] [INFO] Building cldr-to-icu 1.0-SNAPSHOT
     [exec] [INFO] --------------------------------[ jar ]---------------------------------
     [exec] [INFO] 
     [exec] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ cldr-to-icu ---
     [exec] [INFO] Using 'UTF-8' encoding to copy filtered resources.
     [exec] [INFO] Copying 4 resources
     [exec] [INFO] 
     [exec] [INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @ cldr-to-icu ---
     [exec] [INFO] Nothing to compile - all classes are up to date
     [exec] [INFO] 
     [exec] [INFO] --- maven-assembly-plugin:3.1.1:single (default) @ cldr-to-icu ---
     [exec] [INFO] Building jar: /Users/srl295/src/icu/tools/cldr/cldr-to-icu/target/cldr-to-icu-1.0-SNAPSHOT-jar-with-dependencies.jar
     [exec] [INFO] ------------------------------------------------------------------------
     [exec] [INFO] BUILD SUCCESS
     [exec] [INFO] ------------------------------------------------------------------------
     [exec] [INFO] Total time:  6.565 s
     [exec] [INFO] Finished at: 2021-02-02T10:16:16-06:00
     [exec] [INFO] ------------------------------------------------------------------------

clean:
[outputDirectories] Deleting 3,934 auto-generated files...

convert:

     [exec] Buildfile: /Users/srl295/src/icu/tools/cldr/cldr-to-icu/build-icu-data.xml
     [exec] 
     [exec] convert-impl:
     [exec]   [convert] Configuration lists redundant locale IDs
     [exec]   [convert] The following IDs should be removed from the configuration:
     [exec]   [convert] nb_NO, nb_SJ
     [exec]   [convert] Canonical ID list is:
     [exec]   [convert]     // A
     [exec] BUILD FAILED
     [exec] /Users/srl295/src/icu/tools/cldr/cldr-to-icu/build-icu-data.xml:142: java.lang.IllegalStateException: Non-canonical configuration
     [exec]     at org.unicode.icu.tool.cldrtoicu.ant.LocaleIdResolver.resolve(LocaleIdResolver.java:78)
     [exec]     at org.unicode.icu.tool.cldrtoicu.ant.LocaleIdResolver.expandTargetIds(LocaleIdResolver.java:30)
     [exec] 
     [exec]     at org.unicode.icu.tool.cldrtoicu.ant.ConvertIcuDataTask.execute(ConvertIcuDataTask.java:326)  [convert]     af, agq, ak, am, ar, ars, as, asa, ast, az, az_AZ, az_Cyrl
     [exec]     at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
     [exec]     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     [exec] 
     [exec]   [convert]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
     [exec]   [convert]     // B
     [exec] 
     [exec]   [convert]     bas, be, bem, bez, bg, bm, bn, bo, br, brx, bs, bs_BA, bs_Cyrl      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     [exec]   [convert] 
     [exec]     at java.base/java.lang.reflect.Method.invoke(Method.java:564)
     [exec]     at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:99)
     [exec]     at org.apache.tools.ant.Task.perform(Task.java:350)
     [exec]     at org.apache.tools.ant.Target.execute(Target.java:449)
     [exec]     at org.apache.tools.ant.Target.performTasks(Target.java:470)
     [exec]     at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1401)
     [exec]     at org.apache.tools.ant.Project.executeTarget(Project.java:1374)
     [exec] 
     [exec]   [convert]     // C
     [exec]   [convert]     ca, ccp, ce, ceb, cgg, chr, ckb, cs, cy
     [exec]   [convert] 
     [exec]   [convert]     // D
     [exec]   [convert]     da, dav, de, dje, doi, dsb, dua, dyo, dz
     [exec]   [convert] 
     [exec]   [convert]     // E
     [exec]   [convert]     ebu, ee, el, en, en_NH, en_RH, eo, es, et, eu, ewo
     [exec]   [convert] 
     [exec]   [convert]     // F
     [exec]   [convert]     fa, ff, ff_Adlm, ff_CM, ff_GN, ff_MR, ff_SN, fi, fil, fo, fr, fur, fy
     [exec]   [convert] 
     [exec]   [convert]     // G
     [exec]   [convert]     ga, gd, gl, gsw, gu, guz, gv
     [exec]   [convert] 
     [exec]     at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
     [exec]     at org.apache.tools.ant.Project.executeTargets(Project.java:1264)
     [exec]   [convert]     // H
     [exec]   [convert]     ha, haw, he, hi, hr, hsb, hu, hy
     [exec]   [convert] 
     [exec]   [convert]     // I
     [exec]   [convert]     ia, id, ig, ii, in, in_ID, is, it, iw, iw_IL
     [exec]   [convert] 
     [exec]   [convert]     // J
     [exec]     at org.apache.tools.ant.Main.runBuild(Main.java:827)
     [exec]     at org.apache.tools.ant.Main.startAnt(Main.java:223)
     [exec]   [convert]     ja, jgo, jmc, jv    at org.apache.tools.ant.launch.Launcher.run(Launcher.java:284)
     [exec]   [convert] 
     [exec] 
     [exec]     at org.apache.tools.ant.launch.Launcher.main(Launcher.java:101)  [convert]     // K
     [exec]   [convert]     ka, kab, kam, kde, kea, khq, ki, kk, kkj, kl, kln, km, kn, ko, kok, ks
     [exec] 
     [exec] 
     [exec] Total time: 3 seconds
     [exec]   [convert]     ks_IN, ksb, ksf, ksh, ku, kw, ky
     [exec]   [convert] 
     [exec]   [convert]     // L
     [exec]   [convert]     lag, lb, lg, lkt, ln, lo, lrc, lt, lu, luo, luy, lv
     [exec]   [convert] 
     [exec]   [convert]     // M
     [exec]   [convert]     mai, mas, mer, mfe, mg, mgh, mgo, mi, mk, ml, mn, mni, mni_IN, mo, mr, ms
     [exec]   [convert]     mt, mua, my, mzn
     [exec]   [convert] 
     [exec]   [convert]     // N
     [exec]   [convert]     naq, nb, nd, ne, nl, nmg, nn, nnh, no, no_NO_NY, nus, nyn
     [exec]   [convert] 
     [exec]   [convert]     // O
     [exec]   [convert]     om, or, os
     [exec]   [convert] 
     [exec]   [convert]     // P
     [exec]   [convert]     pa, pa_Arab, pa_IN, pa_PK, pcm, pl, ps, pt
     [exec]   [convert] 
     [exec]   [convert]     // Q
     [exec]   [convert]     qu
     [exec]   [convert] 
     [exec]   [convert]     // R
     [exec]   [convert]     rm, rn, ro, rof, ru, rw, rwk
     [exec]   [convert] 
     [exec]   [convert]     // S
     [exec]   [convert]     sa, sah, saq, sat, sat_IN, sbp, sd, sd_Deva, sd_PK, se, seh, ses, sg, sh, sh_BA, sh_CS
     [exec]   [convert]     sh_YU, shi, shi_Latn, shi_MA, si, sk, sl, smn, sn, so, sq, sr, sr_BA, sr_CS, sr_Cyrl_CS, sr_Cyrl_YU
     [exec]   [convert]     sr_Latn, sr_Latn_CS, sr_Latn_YU, sr_ME, sr_RS, sr_XK, sr_YU, su, su_ID, sv, sw
     [exec]   [convert] 
     [exec]   [convert]     // T
     [exec]   [convert]     ta, te, teo, tg, th, ti, tk, tl, tl_PH, to, tr, tt, twq, tzm
     [exec]   [convert] 
     [exec]   [convert]     // U
     [exec]   [convert]     ug, uk, ur, uz, uz_AF, uz_Arab, uz_Cyrl, uz_UZ
     [exec]   [convert] 
     [exec]   [convert]     // V
     [exec]   [convert]     vai, vai_LR, vai_Latn, vi, vun
     [exec]   [convert] 
     [exec]   [convert]     // W
     [exec]   [convert]     wae, wo
     [exec]   [convert] 
     [exec]   [convert]     // X
     [exec]   [convert]     xh, xog
     [exec]   [convert] 
     [exec]   [convert]     // Y
     [exec]   [convert]     yav, yi, yo, yue, yue_CN, yue_HK, yue_Hans
     [exec]   [convert] 
     [exec]   [convert]     // Z
     [exec]   [convert]     zgh, zh, zh_CN, zh_HK, zh_Hant, zh_MO, zh_SG, zh_TW, zu
     [exec]   [convert] 

BUILD FAILED
/Users/srl295/src/icu/tools/cldr/cldr-to-icu/build-icu-data.xml:109: exec returned: 1

Total time: 12 seconds

@srl295
Copy link
Author

srl295 commented Feb 2, 2021

I got further with this change, but i haven't put it into the PR:

diff --git a/tools/cldr/cldr-to-icu/build-icu-data.xml b/tools/cldr/cldr-to-icu/build-icu-data.xml
index ea2f6e3f87..f1b9af507b 100644
--- a/tools/cldr/cldr-to-icu/build-icu-data.xml
+++ b/tools/cldr/cldr-to-icu/build-icu-data.xml
@@ -197,7 +197,7 @@
                 mt, mua, my, mzn
 
                 // N
-                naq, nb, nb_NO, nb_SJ, nd, ne, nl, nmg, nn, nnh, no, no_NO_NY, nus, nyn
+                naq, nb, nd, ne, nl, nmg, nn, nnh, no, no_NO_NY, nus, nyn
 
                 // O
                 om, or, os
@@ -290,7 +290,7 @@
 
                     // K-P
                     ka, kk, kl, km, kn, kok, ko, ku, ky, lb, lkt, ln, lo, lt, lv,
-                    mk, ml, mn, mo, mr, ms, mt, my, nb, nb_NO, ne, nl, nn, no, no_NO,
+                    mk, ml, mn, mo, mr, ms, mt, my, nb, ne, nl, nn, no, no_NO,
                     om, or, pa_IN, pa, pa_Guru, pl, ps, pt,
 
                     // R-T

@srl295
Copy link
Author

srl295 commented Feb 2, 2021

@hagbard it seems icu4c/source/data/build.xml is completely obsolete, and I delete it here. It seems the "generate production data" part that was all that was left is not needed because cldr-to-icu already consumes production data from cldr-staging.

@hagbard
Copy link

hagbard commented Feb 2, 2021

@hagbard it seems icu4c/source/data/build.xml is completely obsolete, and I delete it here. It seems the "generate production data" part that was all that was left is not needed because cldr-to-icu already consumes production data from cldr-staging.

No issues from me! (sorry for not being more attentive to this recently, I just started ramping up on a new project that's taking all my time).

@pedberg-icu
Copy link
Owner

Thanks for this. I have the integration working with the current process in my main PR (most of the work not pushed yet). Let me fix some test failures on that, then I will take a closer look at this.

@pedberg-icu pedberg-icu force-pushed the ICU-21480-brs69rc-integrateCLDR-39-alpha0 branch 6 times, most recently from 1279a61 to b0370b7 Compare February 10, 2021 18:42
pedberg-icu and others added 18 commits February 10, 2021 11:58
and fix a bug in Java UnicodeSet.retain(String) which added the string even if the set did not contain it before,
and some drive-by API doc fixes/clarifications
A regex pattern containing nested look-behind blocks could trigger an assertion
failure during pattern compilation. The problem was caused by an off-by-one
error in the code that computes an upper bound on the match length, needed
because look-behind expressions are constrained to not have unbounded match
length.

Nested look-behind blocks come into play because, when computing the maximum
match length of an outer block, any inner look-behind blocks are skipped over -
they do not directly contribute to the length matched by the outer block. The
problem was in the code that skips over these nested look-behind blocks.
to the icu repository. Added Unicode copyright notice to those files
that only have the IBM copyright notice.
A complimentary PR will remove data from the icu-data repository.

ICU-21487 Adds pure test data files which have no copyright notice to the
exemption list.
test runs with ICU stub data only.

ICU-21502 Review comments worked in.
sven-oly and others added 23 commits March 12, 2021 10:32
- Supplied missing API comment to com.ibm.icu.nubmer.Precision#trailingZeroDisplay(TrailingZeroDisplay)
- Generated API signature file for ICU4J 69.1 and added to the repository
- Delete ICU4J 59 API signature file
- Deleted unused icu4j/eclipse-build/build.properties. The folder eclipse-build was deleted, but someone worked on ICU version seemed to add it back again. Delete it again.
Also dropped ICU4J 64 test data.
evaluates to the empty string and the result files are missing the
perf test name as a result. E.g. '_Mar_16_155820.html' instead of
'unisetperf_Mar_16_155820.html'.
Not obvious why the regex was even created since it doesn't seem
necessary because the name of the caller doesn't require additional
parsing. The rpaticular code is from 2002, maybe changes in Perl
account for the issue(?)
Do not free baseName if it is pointing to fullNameBuffer.

Better Fix
- improve checks for CLDR resolutions
- update batch fetch from 50 to 100
- do not show no-commit tickets under illegal-state
- add navigation
- use numeric IDs for status, resolution, types with constants
 (S_, R_, I_)

fix bugs from CLDR-14224
- do not show not-in-query issues as open
- cache single-issue queries
- give information on each problem as to what it is
- put 'review' in a separate section (not counted in Problems)
- more detailed fix info under each issue

- fix section headings, always show component, add nocopyright option
- sort open issues by componnent
- sort the 'review' list

ICU-20444:
- exclude (via git-cherry) any commits already cherry picked to the branch.
- used a workaround to gitpython-developers/GitPython#846
that seemed to work well,
- warn about issues with excluded commits

CLDR-14524:
- Exclude "Obsolete" from requiring commits
clang 11 throws an error when using undefined variables starting with
TARGET_OS_, and TARGET_OS_SIMULATOR is missing in earlier versions
of macOS, so the code would not compile without an additional guard.

See also ICU-13440.
@srl295 srl295 force-pushed the update-cldr-tools branch from 0ee846b to afbf646 Compare April 1, 2021 17:13
- updated README as well
- deleted install-cldr-jars.sh
@srl295 srl295 force-pushed the update-cldr-tools branch from afbf646 to 5ded363 Compare April 1, 2021 17:18
pedberg-icu added a commit that referenced this pull request Sep 7, 2023
pedberg-icu added a commit that referenced this pull request Sep 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.