English grammar/Temporary entries - Revision history

imported>Martins at 16:52, 16 October 2012

2012-10-16T16:52:27Z

← Older revision		Revision as of 16:52, 16 October 2012
Line 12:		Line 12:
	The OUTPUT for "asfsdfdfdsf" should be simply "asfsdfdfdsf". No process is actually required. However, the English dictionary contains the entries:<br />		The OUTPUT for "asfsdfdfdsf" should be simply "asfsdfdfdsf". No process is actually required. However, the English dictionary contains the entries:<br />
	*[as]{}"" (LEX=C,POS=SCJ,POS=AVR,att=@as,rel=man)<eng,255,0>; and		*[as]{}"" (LEX=C,POS=SCJ,POS=AVR,att=@as,rel=man)<eng,255,0>; and
	*[s]{}"" (LEX=I,POS=SFX,LST=SBW)<eng,255,0>;		*[s]{}"" (LEX=F,POS=SFX,LST=SBW)<eng,255,0>;
	If we do not control the [[tokenization]] process, "asfsdfdfdsf" will be tokenized as [as][f][s][dfdfd][s][f], which is not correct, since we expect the whole string to be treated as a single temporary entry. In order to avoid that, we have to inform the machine that words such as [as] may only happen after a blank space or the beginning of the sentence, and before a blank space. Similary, we have to tell the machine that the suffix [s] must be followed by a blank space. This kind of information is provided in the [[disambiguation grammar]] (d-grammar), and it is used to prevent the system from hyper-analyzing temporary entries. In the English d-grammar, this is done by the rules:		If we do not control the [[tokenization]] process, "asfsdfdfdsf" will be tokenized as [as][f][s][dfdfd][s][f], which is not correct, since we expect the whole string to be treated as a single temporary entry. In order to avoid that, we have to inform the machine that words such as [as] may only happen after a blank space or the beginning of the sentence, and before a blank space. Similary, we have to tell the machine that the suffix [s] must be followed by a blank space. This kind of information is provided in the [[disambiguation grammar]] (d-grammar), and it is used to prevent the system from hyper-analyzing temporary entries. In the English d-grammar, this is done by the rules:
	;(TEMP)(^BLK,^PUT,^STAIL)=0;		;(TEMP)(^BLK,^PUT,^STAIL)=0;

imported>Martins at 05:51, 1 August 2012

2012-08-01T05:51:44Z

← Older revision		Revision as of 05:51, 1 August 2012
Line 1:		Line 1:
	Temporary entries are entries that are not expected to be translated, such as URL's, e-mail addresses, phone numbers, etc. They are not included in the dictionary, because the UNL is the same as the source~~. For instance: the UNL for "asfsdfdfdsf" is "asfsdfdfdsf". For the time being, no transliteration is required either~~.		Temporary entries are entries that are not expected to be translated, such as URL's, e-mail addresses, phone numbers, etc. They are not included in the dictionary, because the UNL is the same as the source.

	== UNLization ==		== UNLization ==
	Temporary entries are not actually translated to UNL. They are represented, in UNL, between quotes, as natural language entries. The quotes are provided automatically by the system: all the words having the feature TEMP will be displayed, in the final output, between quotes. The feature TEMP is assigned automatically by the system to all the entries that were not found in the dictionary.		Temporary entries are not actually translated to UNL. They are represented, in UNL, between quotes, as natural language entries. For instance: the UNL for "asfsdfdfdsf" is "asfsdfdfdsf"; the UNL for "www.undlfoundation.org" is "www.undlfoundation.org"; etc. Temporary entries in non-Latin script have to be transliterated into Latin script. For instance: the UNL for "班禅额尔德尼" is "Panchen Erdeni". The quotes are provided automatically by the system: all the words having the feature TEMP will be displayed, in the final output, between quotes. The feature TEMP is assigned automatically by the system to all the entries that were not found in the dictionary.

	=== asfsdfdfdsf ===		=== asfsdfdfdsf ===

imported>Martins at 05:41, 1 August 2012

2012-08-01T05:41:47Z

← Older revision		Revision as of 05:41, 1 August 2012
Line 2:		Line 2:

	== UNLization ==		== UNLization ==
	Temporary entries are not actually translated to UNL. They are represented, in UNL, between quotes, ~~but these~~ quotes are provided automatically by the system: all the words having the feature TEMP will be displayed, in the final output, between quotes. The feature TEMP is assigned automatically by the system to all the entries that were not found in the dictionary.		Temporary entries are not actually translated to UNL. They are represented, in UNL, between quotes, as natural language entries. The quotes are provided automatically by the system: all the words having the feature TEMP will be displayed, in the final output, between quotes. The feature TEMP is assigned automatically by the system to all the entries that were not found in the dictionary.

	=== asfsdfdfdsf ===		=== asfsdfdfdsf ===

imported>Martins at 05:41, 1 August 2012

2012-08-01T05:41:10Z

← Older revision		Revision as of 05:41, 1 August 2012
Line 1:		Line 1:
	Temporary entries are entries that are not expected to be translated, such as URL's, e-mail addresses, phone numbers, etc. ~~Because of that, they~~ are not included in the dictionary, because the UNL is the same as the source. For instance: the UNL for "asfsdfdfdsf" is "asfsdfdfdsf". For the time being, no transliteration is required either.		Temporary entries are entries that are not expected to be translated, such as URL's, e-mail addresses, phone numbers, etc. They are not included in the dictionary, because the UNL is the same as the source. For instance: the UNL for "asfsdfdfdsf" is "asfsdfdfdsf". For the time being, no transliteration is required either.

	== ~~Issues~~ ==		== UNLization ==
	;Temporary entries are ~~expressed between "quotes"~~		Temporary entries are not actually translated to UNL. They are represented, in UNL, between quotes, but these quotes are provided automatically by the system: all the words having the feature TEMP will be displayed, in the final output, between quotes. The feature TEMP is assigned automatically by the system to all the entries that were not found in the dictionary.
	~~:Note that temporary entries~~ are represented, in UNL, between quotes, but these quotes are provided automatically by the system: all the words having the feature TEMP will be displayed, in the final output, between quotes. The feature TEMP is assigned automatically by the system to all the entries that were not found in the dictionary.

	~~== UNLization ==~~
	=== asfsdfdfdsf ===		=== asfsdfdfdsf ===
	*INPUT (ENG): asfsdfdfdsf		*INPUT (ENG): asfsdfdfdsf

imported>Martins at 20:01, 31 July 2012

2012-07-31T20:01:44Z

← Older revision		Revision as of 20:01, 31 July 2012
Line 67:		Line 67:
	;<nowiki>(TEMP,^DIGIT,^W)(^BLK,^PUT,^STAIL)=0;</nowiki>		;<nowiki>(TEMP,^DIGIT,^W)(^BLK,^PUT,^STAIL)=0;</nowiki>
	;<nowiki>(^BLK,^PUT,^SHEAD)(TEMP,^W)=0;</nowiki>		;<nowiki>(^BLK,^PUT,^SHEAD)(TEMP,^W)=0;</nowiki>
	And assign the feature "W" to all words that have been already tokenized in a general rule such as:		And assign the feature "W" to all words that have been already tokenized. This can be done by a general rule such as:
	;<nowiki>(^W,^SBW,^BLK,^SHEAD,^STAIL,^CHEAD,^CTAIL):=(+W);</nowiki>		;<nowiki>(^W,^SBW,^BLK,^SHEAD,^STAIL,^CHEAD,^CTAIL):=(+W);</nowiki>
	:assigns the feature W to the words that are not subword (SBW), blank (BLK), SHEAD, STAIL, CHEAD or CTAIL		:assigns the feature W to the words that are not subword (SBW), blank (BLK), SHEAD, STAIL, CHEAD or CTAIL

imported>Martins at 20:01, 31 July 2012

2012-07-31T20:01:09Z

← Older revision		Revision as of 20:01, 31 July 2012
Line 63:		Line 63:
	;(TEMP,^DIGIT)(^BLK,^PUT,^STAIL)=0;		;(TEMP,^DIGIT)(^BLK,^PUT,^STAIL)=0;
	However, this is not enough. The rule above will block the deletion of blank spaces after temporary words. Note that disambiguation rules are applied not only during tokenization, but during transformation as well. No transformation rule can generate a result that is prohibited by disambiguation rules. If we have an input like [www.undlfoundation.org][ ][is][ ][a][ ][website], and the rule:		However, this is not enough. The rule above will block the deletion of blank spaces after temporary words. Note that disambiguation rules are applied not only during tokenization, but during transformation as well. No transformation rule can generate a result that is prohibited by disambiguation rules. If we have an input like [www.undlfoundation.org][ ][is][ ][a][ ][website], and the rule:
	;(BLK):=;		;<nowiki>(BLK):=;</nowiki>
	this rule will never apply to the first blank space, because the state [www.undlfoundation.org][is] is prohibited by the d-rule (there must be a BLK, a PUT or STAIL after TEMP). In order to prevent this from happening, we can further specify the d-rules as:		this rule will never apply to the first blank space, because the state [www.undlfoundation.org][is] is prohibited by the d-rule (there must be a BLK, a PUT or STAIL after TEMP). In order to prevent this from happening, we can further specify the d-rules as:
	;(TEMP,^DIGIT,^W)(^BLK,^PUT,^STAIL)=0;		;<nowiki>(TEMP,^DIGIT,^W)(^BLK,^PUT,^STAIL)=0;</nowiki>
	;(^BLK,^PUT,^SHEAD)(TEMP,^W)=0;		;<nowiki>(^BLK,^PUT,^SHEAD)(TEMP,^W)=0;</nowiki>
	And assign the feature "W" to all words that have been already tokenized in a general rule such as:		And assign the feature "W" to all words that have been already tokenized in a general rule such as:
	;(^W,^SBW,^BLK,^SHEAD,^STAIL,^CHEAD,^CTAIL):=(+W);		;<nowiki>(^W,^SBW,^BLK,^SHEAD,^STAIL,^CHEAD,^CTAIL):=(+W);</nowiki>
	:assigns the feature W to the words that are not subword (SBW), blank (BLK), SHEAD, STAIL, CHEAD or CTAIL		:assigns the feature W to the words that are not subword (SBW), blank (BLK), SHEAD, STAIL, CHEAD or CTAIL
	This will prevent the temporary string from being hyper-segmented, but will allow us to delete the blank spaces before and after it after the tokenization.		This will prevent the temporary string from being hyper-segmented, but will allow us to delete the blank spaces before and after it after the tokenization.

imported>Martins at 20:00, 31 July 2012

2012-07-31T20:00:14Z

← Older revision		Revision as of 20:00, 31 July 2012
Line 25:		Line 25:
	*OUTPUT (UNL): "asfsdfdfdsf asfsdfdfdsf"		*OUTPUT (UNL): "asfsdfdfdsf asfsdfdfdsf"
	*DICTIONARY: no entry is necessary		*DICTIONARY: no entry is necessary
	*T-GRAMMAR: (TEMP,%x)(BLK,%y)(TEMP,%z):=(%x&%y&%z~~,-BLK~~); (merges two temporary nodes with a space in-between)		*T-GRAMMAR: (TEMP,%x)(BLK,%y)(TEMP,%z):=(%x&%y&%z); (merges two temporary nodes with a space in-between)
	*D-GRAMMAR: the same as above, to prevent hyper-segmentation.		*D-GRAMMAR: the same as above, to prevent hyper-segmentation.
	From the application of the d-rule above, the tokenization result will be [asfsdfdfdsf][ ][asfsdfdfdsf], which is not actually good, because we would like the whole string to be one single token [asfsdfdfdsf asfsdfdfdsf]. We need then a rule to merge the two temporary words, which is the following:		From the application of the d-rule above, the tokenization result will be [asfsdfdfdsf][ ][asfsdfdfdsf], which is not actually good, because we would like the whole string to be one single token [asfsdfdfdsf asfsdfdfdsf]. We need then a rule to merge the two temporary words, which is the following:
Line 56:		Line 56:
	#*[/(?i)([^ ]+@[^ ]+)/]{}""(TEMP,LEX=N,POS=PPN,EMAIL)<eng,0,0>;		#*[/(?i)([^ ]+@[^ ]+)/]{}""(TEMP,LEX=N,POS=PPN,EMAIL)<eng,0,0>;
	In the English grammar, we have adopted the second solution. These entries are available in the default dictionary, at the end of the English dictionary.		In the English grammar, we have adopted the second solution. These entries are available in the default dictionary, at the end of the English dictionary.

			=== Extending the rules ===
			A major problem with temporary nodes is that they may be part of sentences containing non-temporary words. This requires some additional measures. For instance, the rule
			;(TEMP)(^BLK,^PUT,^STAIL)=0;
			is too generic to be actually useful. Consider for instance, the case of "1st" or "1980's". In both cases we have temporary words ("1" and "1980") that are not followed by BLK, PUT or STAIL. We should then inform the system that digits may come before words:
			;(TEMP,^DIGIT)(^BLK,^PUT,^STAIL)=0;
			However, this is not enough. The rule above will block the deletion of blank spaces after temporary words. Note that disambiguation rules are applied not only during tokenization, but during transformation as well. No transformation rule can generate a result that is prohibited by disambiguation rules. If we have an input like [www.undlfoundation.org][ ][is][ ][a][ ][website], and the rule:
			;(BLK):=;
			this rule will never apply to the first blank space, because the state [www.undlfoundation.org][is] is prohibited by the d-rule (there must be a BLK, a PUT or STAIL after TEMP). In order to prevent this from happening, we can further specify the d-rules as:
			;(TEMP,^DIGIT,^W)(^BLK,^PUT,^STAIL)=0;
			;(^BLK,^PUT,^SHEAD)(TEMP,^W)=0;
			And assign the feature "W" to all words that have been already tokenized in a general rule such as:
			;(^W,^SBW,^BLK,^SHEAD,^STAIL,^CHEAD,^CTAIL):=(+W);
			:assigns the feature W to the words that are not subword (SBW), blank (BLK), SHEAD, STAIL, CHEAD or CTAIL
			This will prevent the temporary string from being hyper-segmented, but will allow us to delete the blank spaces before and after it after the tokenization.

	== NLization ==		== NLization ==
	The NLization of temporary UWs do not require any action from the user. The UW "asfsdfdfdsf" will be automatically generated as "asfsdfdfdsf" if this UW is not in the dictionary.		The NLization of temporary UWs do not require any action from the user. The UW "asfsdfdfdsf" will be automatically generated as "asfsdfdfdsf" if this UW is not in the dictionary.

imported>Martins at 19:48, 31 July 2012

2012-07-31T19:48:08Z

← Older revision		Revision as of 19:48, 31 July 2012
Line 28:		Line 28:
	*D-GRAMMAR: the same as above, to prevent hyper-segmentation.		*D-GRAMMAR: the same as above, to prevent hyper-segmentation.
	From the application of the d-rule above, the tokenization result will be [asfsdfdfdsf][ ][asfsdfdfdsf], which is not actually good, because we would like the whole string to be one single token [asfsdfdfdsf asfsdfdfdsf]. We need then a rule to merge the two temporary words, which is the following:		From the application of the d-rule above, the tokenization result will be [asfsdfdfdsf][ ][asfsdfdfdsf], which is not actually good, because we would like the whole string to be one single token [asfsdfdfdsf asfsdfdfdsf]. We need then a rule to merge the two temporary words, which is the following:
	;<nowiki>(TEMP,%x)(BLK,%y)(TEMP,%z):=(%x&%y&%z~~,-BLK~~);</nowiki>		;<nowiki>(TEMP,%x)(BLK,%y)(TEMP,%z):=(%x&%y&%z);</nowiki>
	The command & merges the nodes, as described in the [[UNL Grammar Specs]].		The command & merges the nodes, as described in the [[UNL Grammar Specs]].

imported>Martins at 19:47, 31 July 2012

2012-07-31T19:47:45Z

← Older revision		Revision as of 19:47, 31 July 2012
Line 28:		Line 28:
	*D-GRAMMAR: the same as above, to prevent hyper-segmentation.		*D-GRAMMAR: the same as above, to prevent hyper-segmentation.
	From the application of the d-rule above, the tokenization result will be [asfsdfdfdsf][ ][asfsdfdfdsf], which is not actually good, because we would like the whole string to be one single token [asfsdfdfdsf asfsdfdfdsf]. We need then a rule to merge the two temporary words, which is the following:		From the application of the d-rule above, the tokenization result will be [asfsdfdfdsf][ ][asfsdfdfdsf], which is not actually good, because we would like the whole string to be one single token [asfsdfdfdsf asfsdfdfdsf]. We need then a rule to merge the two temporary words, which is the following:
	;(TEMP,%x)(BLK,%y)(TEMP,%z):=(%x&%y&%z,-BLK);		;<nowiki>(TEMP,%x)(BLK,%y)(TEMP,%z):=(%x&%y&%z,-BLK);</nowiki>
	The command & merges the nodes, as described in the [[UNL Grammar Specs]].		The command & merges the nodes, as described in the [[UNL Grammar Specs]].

imported>Martins at 19:46, 31 July 2012

2012-07-31T19:46:31Z

← Older revision		Revision as of 19:46, 31 July 2012
Line 28:		Line 28:
	*D-GRAMMAR: the same as above, to prevent hyper-segmentation.		*D-GRAMMAR: the same as above, to prevent hyper-segmentation.
	From the application of the d-rule above, the tokenization result will be [asfsdfdfdsf][ ][asfsdfdfdsf], which is not actually good, because we would like the whole string to be one single token [asfsdfdfdsf asfsdfdfdsf]. We need then a rule to merge the two temporary words, which is the following:		From the application of the d-rule above, the tokenization result will be [asfsdfdfdsf][ ][asfsdfdfdsf], which is not actually good, because we would like the whole string to be one single token [asfsdfdfdsf asfsdfdfdsf]. We need then a rule to merge the two temporary words, which is the following:
	:(TEMP,%x)(BLK,%y)(TEMP,%z):=(%x&%y&%z,-BLK);		;(TEMP,%x)(BLK,%y)(TEMP,%z):=(%x&%y&%z,-BLK);
	The command & merges the nodes, as described in the [[UNL Grammar Specs]].		The command & merges the nodes, as described in the [[UNL Grammar Specs]].