You can define new settings to configure NLU when you create or edit your project. Settings are passed in the text form within a JSON object.
\nYou must specify your parameters according to the project’s classifier algorithm.
\n\nCommon settings include parameters that do not depend on the algorithm of the classifier in the project:
\n{\n \"patternsEnabled\": true,\n \"tokenizerEngine\": \"udpipe\",\n \"dictionaryAutogeneration\": true\n}patternsEnabled — if the parameter is active, it is possible to use patterns in training phrases.tokenizerEngine — tokenizer.dictionaryAutogeneration — when the parameter is active, the filling of the user dictionary by entity content is executed.tokenizerEngineDepending on the language of the NLU you need to use different types of tokenizers.
\n| NLU language | \ntokenizerEngine | \nAdditional information | \n
|---|---|---|
| Russian | \nudpipe mystem morphsrus | \nThe default is udpipe. Tokenizers mystem and morphsrus are used for migration of projects from patterns to CAILA. | \n
| English | \nspacy udpipe | \n\n |
| Chinese | \npinyin | \n\n |
| Portuguese | \nudpipe | \n\n |
STS classifier settings:
\n{\n \"patternsEnabled\": true,\n \"namedEntitiesRequired\": true,\n \"tokenizerEngine\": \"udpipe\",\n\n \"stsSettings\": {\n \"exactMatch\": 1.0,\n \"lemmaMatch\": 0.95,\n \"jaccardMatch\": 0.5,\n \"jaccardMatchThreshold\": 0.82,\n \"acronymMatch\": 1.0,\n \"synonymMatch\": 0,\n \"synonymContextWeight\": 0.0,\n \"patternMatch\": 1,\n \"throughPatternMatch\": 0,\n \"wordSequence1\": 0.8,\n \"wordSequence2\": 0.9,\n \"wordSequence3\": 1.0,\n \"intermediateAlternativesLimit\": 5,\n \"finalAlternativesLimit\": 5,\n \"idfShift\": 0.0,\n \"idfMultiplier\": 1.0,\n \"namedEntitiesRequired\": true,\n }\n}exactMatch — weight of an exact word match in phrases.lemmaMatch — weight of a lemma word match.jaccardMatchThreshold — weight of a Jaccard-based symbol-by-symbol word match.acronymMatch — weight of an acronym-based word match.synonymMatch — weight for synonyms.synonymContextWeight — weight applied during ranking to the weight value from the synonym reference book.patternMatch — weight of a pattern-based match.throughPatternMatch — weight of a match by entities found in the example and the input text.wordSequence1 — weight of lookalike sequences with the length equal to 1.wordSequence2 — weight of lookalike sequences with the length equal to 2.wordSequence3 — weight of lookalike sequences with the length greater than 2.intermediateAlternativesLimit — cutoff threshold for intermediate alternatives processed by the algorithm.finalAlternativesLimit — maximum number of final results required for the algorithm to finish its work.namedEntitiesRequired — in order to get into the intent, the phrase must have a system entity.namedEntitiesRequiredLet's consider the parameter \"namedEntitiesRequired\": true. If a phrase with a system entity was added to the intent, for example:
I need @duckling.number of applesThat at request of the client I need apples — the phrase will not get in an intent as the system essence has not been found.
Override namedEntitiesRequired in the advanced NLU settings so that phrases without system entities activate the intent.
Deep Learning classifier settings
\n{\n \"patternsEnabled\": true,\n \"tokenizerEngine\": \"udpipe\",\n\n \"cnnSettings\": {\n \"lang\": \"ru\",\n \"kernel_sizes\": [\n 1,\n 2\n ],\n \"n_filters\": 1024,\n \"emb_drp\": 0.25,\n \"cnn_drp\": 0.25,\n \"bs\": 64,\n \"n_epochs\": 15,\n \"lr\": 0.001,\n \"pooling_name\": \"max\"\n },\n}kernel_sizes – list of convolution kernel sizes. A convolution kernel is the size of the context window to be taken into account by the classifier. For instance, “kernel_sizes”: [3] means that the model will use all the triplets of adjacent words to find features in the text. Multiple convolution kernels can be defined for a single model.
n_filters – number of filters. A filter is a specific pattern learned by the model. A model has a unique set of patterns for each kernel. For instance, if we specify “kernel_sizes”: [2,3] and “n_filters”: 512, the total number of filters will be 1024 (512 per kernel).
emb_drp – probability of a drop-out on the embedding layer. Drop-out is the mechanism that forcibly disables a part of weights in the network in the course of training. Drop-out prevents the network from overtraining (i.e. it helps to summarize the information instead of merely memorizing the entire dataset). emb_drp can take any value from 0 to 1.
сnn_drp – probability of a drop-out on the convolution layers of the network.
bs – size of the input batch for trainings. This value defines the number of training examples per step to be fed to the network in the course of training. If the dataset has less than 3000 examples, a value from 16 to 32 is recommended. For larger datasets, this value can be from 32 to 128.
n_epochs – number of learning epochs (i.e. the number of times the model will see all the training data).
lr – learning rate. The factor the model will use to update its weights in the course of training.
pooling_name – aggregation strategy. The model has to aggregate the patterns found in the input string (before the final classification layer). The following aggregation strategies are possible: max, mean, concat.
Deep Learning classifier settings for the dataset size of:
\n \"kernel_sizes\": [\n 2,\n 3,\n 4\n ],\n \"n_filters\": 1024-2048,\n \"emb_drp\": 0.3-0.4,\n \"cnn_drp\": 0.3-0.4,\n \"bs\": 64-128,\n \"n_epochs\": 3,\n \"lr\": 0.001,\n \"pooling_name\": \"max, concat\" \"kernel_sizes\": [\n 2,\n 3,\n 4\n ],\n \"n_filters\": 1024-2048,\n \"emb_drp\": 0.3-0.4,\n \"cnn_drp\": 0.3-0.4,\n \"bs\": 32-128,\n \"n_epochs\": 3,\n \"lr\": 0.001,\n \"pooling_name\": \"max, concat\" \"kernel_sizes\": [\n 2,\n 3,\n 4 // or [2,3]\n ],\n \"n_filters\": 1024,\n \"emb_drp\": 0.3-0.5,\n \"cnn_drp\": 0.3-0.5,\n \"bs\": 32-64,\n \"n_epochs\": 3-5,\n \"lr\": 0.001,\n \"pooling_name\": \"max\" \"kernel_sizes\": [\n 2,\n 3,\n 4 // or [2,3]\n ],\n \"n_filters\": 1024,\n \"emb_drp\": 0.4-0.5,\n \"cnn_drp\": 0.4-0.5,\n \"bs\": 32,\n \"n_epochs\": 4-7,\n \"lr\": 0.001,\n \"pooling_name\": \"max\" \"kernel_sizes\": [\n 2,\n 3\n ],\n \"n_filters\": 512,\n \"emb_drp\": 0.5,\n \"cnn_drp\": 0.5,\n \"bs\": 16-32,\n \"n_epochs\": 7-15,\n \"lr\": 0.001,\n \"pooling_name\": \"max\"{\n \"patternsEnabled\": true,\n \"tokenizerEngine\": \"udpipe\",\n\n \"classicMLSettings\": {\n \"C\": 1,\n \"lang\": \"ru\",\n \"word_ngrams\": [\n 1,\n 2\n ],\n \"lemma_ngrams\": [\n 0\n ],\n \"stemma_ngrams\": [\n 1,\n 2\n ],\n \"char_ngrams\": [\n 3,\n 4\n ],\n \"lower\": true\n }\n}С – regularization coefficient that can be used to control model overtraining. Use it to control larger values of the target function coefficients and to penalize them by the value of the parameter. It can take values in the following range: [0.01, 0.1, 1, 10].
word_ngrams – number of words to be combined into word combinations. For instance, for “word_ngrams”: [2, 3] combinations of two and three words will be used.
For instance, the following word combinations will be generated for the I like green apples phrase:
[\n \"I like\",\n \"like green\",\n \"green apples\",\n \"I like green\",\n \"like green apples\"\n]Values greater than 3 are not recommended for this parameter.
\nlemma_n_grams – number of words to be normalized and combined into word combinations. For instance, for “lemma_n_grams”: [2] combinations of two words will be used.For instance, the following word combinations will be generated for the I like green apples phrase:
[\n \"I like\",\n \"like green\",\n \"green apple\"\n]Values greater than 3 are not recommended for this parameter.
\nstemma_ngrams – number of stemmas to be combined into word combinations. A stemma is the stem of a source word which is not necessarily equal to the morphologic root of that word. For instance, for “stemma_ngrams”: [2] combinations of two stemmas will be used.For instance, the following word combinations will be generated for the I like green apples phrase:
[\n \"I like\",\n \"like green\",\n \"green apple\"\n]Using both lemma_n_grams and stemma_ngrams parameters is not recommended due to possible model overtraining. The recommended value of stemma_ngrams is also greater than 3.
char_n_grams – number of symbols to be combined and treated as a single unit of a phrase.For instance, for “char_n_grams”: [5] the phrase of green apples is converted to the following set:
[\n \"gre\",\n \"ree\",\n \"een\",\n ...\n]lower – if set to true, all the phrases are converted to lowercase.The JAICP platform supports the ability to connect an external NLU service that complies with the Model API specification. You will also be able to create and configure intents, entities, patterns.
\nModel API allows you to use third-party tokenizers, external named entity and intent recognition NLU services in JAICP projects.
\nTo use an external NLU service in a project, use externalNluSettings in the Advanced NLU settings field:
...\n\"externalNluSettings\": {\n \"nluProviderSettings\": {\n \"markup\": {\n \"nluType\": \"external\",\n \"url\": \"http://example.com\"\n },\n \"ner\": {\n \"nluType\": \"external\",\n \"url\": \"http://example.com\"\n },\n \"classification\": {\n \"nluType\": \"external\",\n \"url\": \"http://example.com\"\n }\n },\n \"language\": \"ja\",\n \"nluActionAdditionalProperties\": {\n \"markup\": null,\n \"ner\": null,\n \"classification\": {\n \"modelId\": \"123\",\n \"classifierName\": \"example\",\n \"properties\": null\n }\n }\n}\n...| Parameter | \nDescription | \n
|---|---|
classifierName | \nClassifier name. | \n
classification | \nMap of parameters for classification requests. | \n
language | \nExternal NLU language parameter. If not set, language from the project settings will be used. | \n
markup | \nMap of parameters for markup requests. | \n
modelID | \nClassifier model ID. | \n
ner | \nNamed entity recognition. Map of parameters for named entity recognition requests. | \n
nluActionAdditionalProperties | \nAdditional NLU service properties. | \n
nluProviderSettings | \nObject that determines where the NLU action is going to be performed. | \n
nluType | \nNLU type. You can set either external or caila NLU type. | \n
Please note that you cannot use both CAILA and external NLU service intents or entities at the same time in a project.
\nIn the JAICP project you can:
\nUse entities and intents from the external NLU service.
\n\nUse CAILA intents and entities from the external NLU service.
\nnluType to external for the ner parameter and to caila for the markup and classification parameters.q tag.Use CAILA entities and intents from the external NLU service.
\nnluType to external for the classification parameter and to caila for the markup and ner parameters.q tag.Use external NLU service markup with CAILA intents and entities.
\nnluType: to external for the markup parameter and to caila for the classification and ner.You can look through an example of an external NLU service in the GitHub repository
","frontmatter":{"title":"","description":null},"headings":[{"value":"Extended NLU settings"}]}},"pageContext":{"slug":"/docs/en/NLU_core/advanced_classifer_settings/","previous":{"fields":{"slug":"/docs/en/NLU_core/data/"},"frontmatter":{"title":"","description":null}},"next":{"fields":{"slug":"/docs/en/JS_API/session_lifetime_control/"},"frontmatter":{"title":"","description":null}}}}, "staticQueryHashes": ["1209419333"]}