ICU-22798 Avoid stack overflow by return error.

[FrankYFTang]

Change the nested level limit from 0x7fff to 0x0fff since the fuzzer found in some machine 6400 level deep already cause stack overflow.

Checklist

• Required: Issue filed: https://unicode-org.atlassian.net/browse/ICU-22798
• Required: The PR title must be prefixed with a JIRA Issue number.
• Required: The PR description must include the link to the Jira Issue, for example by completing the URL in the first checklist item
• Required: Each commit message must be prefixed with a JIRA Issue number.
• Issue accepted (done by Technical Committee after discussion)
• Tests included, if applicable
• API docs and/or User Guide docs changed or added, if applicable

[richgillam]

LGTM

[FrankYFTang]

MSVC build all fail on the new test I add. Maybe the stack is even shallower, trying to reduce the limit to see would that make it pass

[FrankYFTang]

so I test to use different limit on MSVC in another testing PR
it turn out limit to 0x0fff (4095) or 0x07ff 2047 still cause very deep stack for MSVC and crash, but limit to 0x1ff (511) work. Trying the limit of 0x3ff (2047) on another PR now.

[jira-pull-request-webhook]

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

[FrankYFTang]

PTAL

[richgillam]

Belated ship-it. The additional drop in the level is fine with me.