Duplication between CARDINAL and PERCENT entities

[atwright147]

Given the following code:

import winkNLP, { CustomEntityExample, ItemEntity } from 'wink-nlp';
const model = require('wink-eng-lite-model');
const nlp = winkNLP(model);

const { its, as } = nlp;

const ents: CustomEntityExample[] = [
    {
        name: 'maths',
        patterns: [
            'CARDINAL',
            'CCONJ',
            'PERCENT',
            'NUM',
            'SYM',
            'MONEY',
            'plus',
            'add',
            '+',
            'subtract',
            'minus',
            '-',
            'multiply',
            'times',
            'x',
            'divide',
            '/',
        ],
    },
];

nlp.learnCustomEntities(ents, { matchValue: false, usePOS: true, useEntity: true });

const doc1 = nlp.readDoc('I want to know what 5 percent of 5 plus 10 minus 2 is');

console.table(doc1.customEntities().out());

I get the following:

┌─────────┬─────────────┐
│ (index) │   Values    │
├─────────┼─────────────┤
│    0    │ '5 percent' │
│    1    │     '5'     │
│    2    │   'plus'    │
│    3    │    '10'     │
│    4    │   'minus'   │
│    5    │     '2'     │
└─────────┴─────────────┘

The issue that I have is that the 5 is duplicated between index 0 and 1. Is there a way to determine that the 5 is from 5 percent?

Is there a way to stop that 5 from appearing in the list if it was already captured by the 'PERCENT' entity?

[sanjayaksaxena]

Hello @atwright147

The current code is detecting the following custom entities:

Given the custom entity examples & the configuration — { matchValue: false, usePOS: true, useEntity: true }, this is the correct output.

Now if you want to detect 5 and percent separately then use { matchValue: false, usePOS: true, useEntity: false } configuration; escape 'PERCENT' as '^PERCENT' (escaping is required as PERCENT is a named entity within winkNLP). Here is how the revised code would look like:

const winkNLP = require( 'wink-nlp' );
const model = require( 'wink-eng-lite-web-model' );
const nlp = winkNLP( model );
const its = nlp.its;
const as = nlp.as;

const examples = [
    {
        name: 'maths',
        patterns: [
            'CARDINAL',
            'CCONJ',
            'NUM',
            'SYM',
            'MONEY',
            'plus',
            'add',
            '+',
            'subtract',
            'minus',
            '-',
            'multiply',
            'times',
            'x',
            'divide',
            '/',
            '^PERCENT'
        ]
    }
];

nlp.learnCustomEntities(examples, { matchValue: false, usePOS: true, useEntity: false });

const text = 'I want to know what 5 percent of 5 plus 10 minus 2 is';
const doc = nlp.readDoc( text );

doc.customEntities().each( ( e ) => {
  e.markup( '<mark>', `<sub style="font-weight:900; font-size:0.6em"> ${e.out(its.type)}</sub></mark>` );
} );

doc.out( its.markedUpText );

The above code will produce the following output:

While traversing tokens you can always use the parentEntity() method to know the parent entity to which a particular token belongs; you can also obtain span of the entity using its.span helper.

For more details, please refer to the documentation.

Best,
Sanjaya