CLEA1500 - Revision history

imported>Martins at 14:22, 7 August 2014

2014-08-07T14:22:32Z

← Older revision		Revision as of 14:22, 7 August 2014
Line 52:		Line 52:
	*[[UNL Dictionary Specs]], which is essential to understand the dictionary structure		*[[UNL Dictionary Specs]], which is essential to understand the dictionary structure
	*[[UNL Grammar Specs]], which is essential to understand the grammar structure		*[[UNL Grammar Specs]], which is essential to understand the grammar structure
	*[[X-bar]]
	*[[Default Dictionary]]
	*[[English dictionary]]
	*[[Default grammar]]
	*[[English grammar]]
	It is also interesting to make a test drive with [[IAN]].		It is also interesting to make a test drive with [[IAN]].

	== Notes ==		== Notes ==
	<references/>		<references/>

imported>Martins at 14:20, 7 August 2014

2014-08-07T14:20:28Z

← Older revision		Revision as of 14:20, 7 August 2014
Line 7:		Line 7:
	#Training corpus		#Training corpus
	##Prepare the training corpus		##Prepare the training corpus
	##*<u>If your native language is not English</u>: translate (manually) the ~~100~~ sentences of the corpus [[UCA1]] from English into your native language. Be as close as possible to the original, and provide one single translation for each sentence. This will be your input document file, and your goal will be to provide (automatically, through [[IAN]]) the UNL graphs for each sentence. See an example of input file [http://www.unlweb.net.br/resources/UCA1/uca1_fra.txt here].		##*<u>If your native language is not English</u>: translate (manually) the 50 sentences of the corpus [[UCA1]] from English into your native language. Be as close as possible to the original, and provide one single translation for each sentence. This will be your input document file, and your goal will be to provide (automatically, through [[IAN]]) the UNL graphs for each sentence. See an example of input file [http://www.unlweb.net.br/resources/UCA1/uca1_fra.txt here].
	##*<u>If your native language is English</u>: use the corpus [[UCA2]] as your input document file.		##*<u>If your native language is English</u>: use the corpus [[UCA2]] as your input document file.
	##Save the translated text (without the English original) in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>IAN>NL FILES.		##Save the translated text (without the English original) in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>IAN>NL FILES.

imported>Martins at 14:20, 7 August 2014

2014-08-07T14:20:07Z

← Older revision		Revision as of 14:20, 7 August 2014
Line 1:		Line 1:
	CLEA<sup>1500</sup> certifies that you have reached a standard of knowledge concerning the development of natural language grammars for [[UNLization]] and may officially participate in the UNL Research. The certificate consists of a set of ~~100~~ sentences that must be UNLized with IAN.		CLEA<sup>1500</sup> certifies that you have reached a standard of knowledge concerning the development of natural language grammars for [[UNLization]] and may officially participate in the UNL Research. The certificate consists of a set of 50 sentences that must be UNLized with IAN.

	== Goal ==		== Goal ==

imported>Martins at 18:46, 4 August 2014

2014-08-04T18:46:35Z

← Older revision		Revision as of 18:46, 4 August 2014
Line 1:		Line 1:
	~~CLEA1500~~ certifies that you have reached a standard of knowledge concerning the development of natural language grammars for [[UNLization]] and may officially participate in the UNL Research. The certificate consists of a set of 100 sentences that must be UNLized with IAN.		CLEA<sup>1500</sup> certifies that you have reached a standard of knowledge concerning the development of natural language grammars for [[UNLization]] and may officially participate in the UNL Research. The certificate consists of a set of 100 sentences that must be UNLized with IAN.

	== Goal ==		== Goal ==
	In ~~CLEA1500~~, you are expected to provide the lingware (dictionary and grammars) necessary to '''UNLize''' a translated version of [[UCA1]].		In CLEA<sup>1500</sup>, you are expected to provide the lingware (dictionary and grammars) necessary to '''UNLize''' a translated version of [[UCA1]].

	== Instructions ==		== Instructions ==
Line 17:		Line 17:
	##*[[N-grammar]], which is used to segment<ref>This is not necessary, because the corpus is already segmented.</ref> the text into sentences and to normalize the input text, resolving contractions, abbreviations and other phenomena that can be handled without the dictionary (i.e., by manipulating isolated strings of characters);		##*[[N-grammar]], which is used to segment<ref>This is not necessary, because the corpus is already segmented.</ref> the text into sentences and to normalize the input text, resolving contractions, abbreviations and other phenomena that can be handled without the dictionary (i.e., by manipulating isolated strings of characters);
	##*[[T-grammar]], which is used to transform the list structure (i.e., the natural language sentence) into a graph structure (the UNL graph);		##*[[T-grammar]], which is used to transform the list structure (i.e., the natural language sentence) into a graph structure (the UNL graph);
	##*[[D-grammar]], which is used to improve the performance of the [[tokenization]] and of the [[transformation]].<ref>D-rules used in tokenization are used either to prevent wrong lexical matches or to provoke right lexical matches. D-rules used in ~~transformaton~~ are used to either prevent or provoke the application of certain rules.</ref>		##*[[D-grammar]], which is used to improve the performance of the [[tokenization]] and of the [[transformation]].<ref>D-rules used in tokenization are used either to prevent wrong lexical matches or to provoke right lexical matches. D-rules used in transformation are used to either prevent or provoke the application of certain rules.</ref>
	##Save the NL-UNL grammars in a plain text (.txt) file with UTF-8 encoding and upload them to the corresponding tabs in UNLWEB>UNLDEV>IAN.		##Save the NL-UNL grammars in a plain text (.txt) file with UTF-8 encoding and upload them to the corresponding tabs in UNLWEB>UNLDEV>IAN.
	##Test them and do the necessary changes until you get good results.		##Test them and do the necessary changes until you get good results.

imported>Martins at 18:37, 4 August 2014

2014-08-04T18:37:10Z

← Older revision		Revision as of 18:37, 4 August 2014
Line 2:		Line 2:

	== Goal ==		== Goal ==
	In CLEA1500, you are expected to provide the lingware (dictionary and grammars) necessary to UNLize a translated version of [[UCA1]].		In CLEA1500, you are expected to provide the lingware (dictionary and grammars) necessary to '''UNLize''' a translated version of [[UCA1]].

	== Instructions ==		== Instructions ==

imported>Martins at 18:35, 4 August 2014

2014-08-04T18:35:37Z

imported>Martins at 18:34, 4 August 2014

2014-08-04T18:34:50Z

← Older revision		Revision as of 18:34, 4 August 2014
Line 38:		Line 38:
	\|-		\|-
	\|align=center\|English		\|align=center\|English
	\|[http://www.unlweb.net.br/resources/UCA1/~~UCA1_eng~~.txt ~~UCA1 in English~~]		\|[http://www.unlweb.net.br/resources/UCA1/uca1_eng.txt uca1_eng.txt]
	\|[http://www.unlweb.net.br/resources/UCA1/eng_unl_dic.txt eng_unl_dic.txt]		\|[http://www.unlweb.net.br/resources/UCA1/eng_unl_dic.txt eng_unl_dic.txt]
	\|[http://www.unlweb.net.br/resources/UCA1/eng_unl_ngrammar.txt eng_unl_ngrammar.txt]		\|[http://www.unlweb.net.br/resources/UCA1/eng_unl_ngrammar.txt eng_unl_ngrammar.txt]

imported>Martins at 18:34, 4 August 2014

2014-08-04T18:34:24Z

← Older revision		Revision as of 18:34, 4 August 2014
Line 31:		Line 31:
	!Language		!Language
	!Corpus		!Corpus
	!Dictionary~~<ref>Two dictionaries are necessary for each language: the language~~-specific dictionary, and the [[Default Dictionary]], which contains language-independent entries, such as punctuation signs and regular expressions. The default dictionary must be loaded after the language-specific dictionary.</ref>		!Dictionary
	!T-Grammar<ref>Three t-grammars are necessary for each language: the [[Standardization grammar]], the language-specific grammar, and the [[Default grammar]]. The standardization grammar and the default grammar are language-independent. The grammars must be loaded in this order: 1) standardization, 2) language-specific, and 3) default. </ref>		!N-Grammar
			!T-Grammar
	!D-Grammar		!D-Grammar
	!Output		!Output
	\|-		\|-
	\|align=center\|English		\|align=center\|English
	\|[http://www.unlweb.net.br/resources~~/corpus~~/UCA1/UCA1_eng.txt ~~UC-A1~~ in English]		\|[http://www.unlweb.net.br/resources/UCA1/UCA1_eng.txt UCA1 in English]
	\|[http://www.unlweb.net.br/resources~~/dic~~/UCA1/eng_unl_dic.txt ~~ENG-UNL Dictionary]<br />[http://www.unlweb.net.br/resources/dic/default_dic~~.txt ~~Default Dictionary~~]		\|[http://www.unlweb.net.br/resources/UCA1/eng_unl_dic.txt eng_unl_dic.txt]
	\|[http://www.unlweb.net.br/resources/~~grammar~~/~~s-grammar~~.txt ~~Standardization Grammar~~]~~<br />~~		\|[http://www.unlweb.net.br/resources/UCA1/eng_unl_ngrammar.txt eng_unl_ngrammar.txt]
	[http://www.unlweb.net.br/resources~~/grammar~~/UCA1/eng_unl_tgrammar.txt ~~ENG-UNL T-Grammar]<br />~~		\|[http://www.unlweb.net.br/resources/UCA1/eng_unl_tgrammar.txt eng_unl_tgrammar.txt]
	~~[http://www.unlweb.net.br/resources/grammar/nl_unl_tgrammar~~.txt ~~Default T-Grammar~~]		\|[http://www.unlweb.net.br/resources/UCA1/eng_unl_dgrammar.txt eng_unl_dgrammar.txt]
	\|[http://www.unlweb.net.br/resources~~/grammar~~/UCA1/eng_unl_dgrammar.txt ~~ENG-UNL D-Grammar~~]		\|[http://www.unlweb.net.br/resources/UCA1/eng_unl_trace.txt eng_unl_trace.txt]
	\|[http://www.unlweb.net.br/resources~~/output~~/UCA1/~~eng_unl~~.txt ~~ENG>UNL~~]
	\|}		\|}
	<br />		<br />

imported>Martins at 18:30, 4 August 2014

2014-08-04T18:30:20Z

← Older revision		Revision as of 18:30, 4 August 2014
Line 15:		Line 15:
	#Grammars<ref>In order to prepare the grammar, study the [[Grammar Specs]]. Next, take a look at the structure of the [[English_grammar\|English Grammar]] for a detailed example. In many cases, it is simpler just to localize the English grammars to your own locale rather than creating a whole grammar from the scratch. See the instructions at [[Localization]].</ref>		#Grammars<ref>In order to prepare the grammar, study the [[Grammar Specs]]. Next, take a look at the structure of the [[English_grammar\|English Grammar]] for a detailed example. In many cases, it is simpler just to localize the English grammars to your own locale rather than creating a whole grammar from the scratch. See the instructions at [[Localization]].</ref>
	##Create the NL-UNL (analysis) grammars necessary to UNLize the natural language sentences of the translated corpus. These grammars are the most difficult (and the actual goal) of the whole analysis task. There can be three different types of grammar:		##Create the NL-UNL (analysis) grammars necessary to UNLize the natural language sentences of the translated corpus. These grammars are the most difficult (and the actual goal) of the whole analysis task. There can be three different types of grammar:
	##*[[N-grammar]], which is used to segment<ref>This is not necessary, because the corpus is already segmented.</ref> the text into sentences and to normalize the input text, resolving contractions, abbreviations and other phenomena that can be handled without the dictionary (i.e., by manipulating isolated strings of characters).		##*[[N-grammar]], which is used to segment<ref>This is not necessary, because the corpus is already segmented.</ref> the text into sentences and to normalize the input text, resolving contractions, abbreviations and other phenomena that can be handled without the dictionary (i.e., by manipulating isolated strings of characters);
	##*[[T-grammar]], which is used to transform the list structure (i.e., the natural language sentence) into a graph structure (the UNL graph).		##*[[T-grammar]], which is used to transform the list structure (i.e., the natural language sentence) into a graph structure (the UNL graph);
	##*[[D-grammar]], which is used to improve the performance of the [[tokenization]] and of the [[transformation]]<ref>D-rules used in tokenization are used either to prevent wrong lexical matches or to provoke right lexical matches. D-rules used in transformaton are used to either prevent or provoke the application of certain rules.</ref>		##*[[D-grammar]], which is used to improve the performance of the [[tokenization]] and of the [[transformation]].<ref>D-rules used in tokenization are used either to prevent wrong lexical matches or to provoke right lexical matches. D-rules used in transformaton are used to either prevent or provoke the application of certain rules.</ref>
	##Save the NL-UNL grammars in a plain text (.txt) file with UTF-8 encoding and upload them to the corresponding tabs in UNLWEB>UNLDEV>IAN.		##Save the NL-UNL grammars in a plain text (.txt) file with UTF-8 encoding and upload them to the corresponding tabs in UNLWEB>UNLDEV>IAN.
	##Test them and do the necessary changes until you get good results.		##Test them and do the necessary changes until you get good results.

imported>Martins at 18:29, 4 August 2014

2014-08-04T18:29:43Z

← Older revision		Revision as of 18:29, 4 August 2014
Line 17:		Line 17:
	##*[[N-grammar]], which is used to segment<ref>This is not necessary, because the corpus is already segmented.</ref> the text into sentences and to normalize the input text, resolving contractions, abbreviations and other phenomena that can be handled without the dictionary (i.e., by manipulating isolated strings of characters).		##*[[N-grammar]], which is used to segment<ref>This is not necessary, because the corpus is already segmented.</ref> the text into sentences and to normalize the input text, resolving contractions, abbreviations and other phenomena that can be handled without the dictionary (i.e., by manipulating isolated strings of characters).
	##*[[T-grammar]], which is used to transform the list structure (i.e., the natural language sentence) into a graph structure (the UNL graph).		##*[[T-grammar]], which is used to transform the list structure (i.e., the natural language sentence) into a graph structure (the UNL graph).
	##*[[D-grammar]], which is used to improve the performance of the [[tokenization]] and of the [[transformation]]~~, by preventing~~ wrong lexical matches or ~~provoking~~ right lexical matches~~, or by preventing the application of wrong~~ rules or ~~provoking~~ the application of ~~right~~ rules.		##*[[D-grammar]], which is used to improve the performance of the [[tokenization]] and of the [[transformation]]<ref>D-rules used in tokenization are used either to prevent wrong lexical matches or to provoke right lexical matches. D-rules used in transformaton are used to either prevent or provoke the application of certain rules.</ref>
	##Save the NL-UNL grammars in a plain text (.txt) file with UTF-8 encoding and upload them to the corresponding tabs in UNLWEB>UNLDEV>IAN.		##Save the NL-UNL grammars in a plain text (.txt) file with UTF-8 encoding and upload them to the corresponding tabs in UNLWEB>UNLDEV>IAN.
	##Test them and do the necessary changes until you get good results.		##Test them and do the necessary changes until you get good results.

← Older revision		Revision as of 18:35, 4 August 2014
Line 7:		Line 7:
	#Training corpus		#Training corpus
	##Prepare the training corpus		##Prepare the training corpus
	##*<u>If your native language is not English</u>: translate (manually) the 100 sentences of the corpus [[UCA1]] from English into your native language. Be as close as possible to the original, and provide one single translation for each sentence. This will be your input document file, and your goal will be to provide (automatically, through [[IAN]]) the UNL graphs for each sentence. See an example of input file [http://www.unlweb.net.br/resources~~/corpus~~/UCA1/uca1_fra.txt here].		##*<u>If your native language is not English</u>: translate (manually) the 100 sentences of the corpus [[UCA1]] from English into your native language. Be as close as possible to the original, and provide one single translation for each sentence. This will be your input document file, and your goal will be to provide (automatically, through [[IAN]]) the UNL graphs for each sentence. See an example of input file [http://www.unlweb.net.br/resources/UCA1/uca1_fra.txt here].
	##*<u>If your native language is English</u>: use the corpus [[UCA2]] as your input document file.		##*<u>If your native language is English</u>: use the corpus [[UCA2]] as your input document file.
	##Save the translated text (without the English original) in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>IAN>NL FILES.		##Save the translated text (without the English original) in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>IAN>NL FILES.
	#Dictionary<ref>Instead of creating a whole new dictionary from scratch, you may try localizing the English dictionary available at [http://www.unlweb.net.br/resources~~/dic~~/UCA1/eng_unl_dic.txt eng_unl_dic.txt]. Note that "localization" is not the same as "translation". You may need other features (in English, for instance, nouns do not have gender or case) or other entries. In any case, the resulting dictionary should reflect your translated version of the corpus (i.e., all entries appearing in your translated version of the corpus should appear in the dictionary). For further information on localization, see [[Localization]]. For information on the dictionary structure, see [[Dictonary Specs]]. For an explanation of the structure of the English dictionary, see [[English_dictionary\|English Dictionary]]. In case you need additional features, use only the tags available at the [[tagset]].</ref>		#Dictionary<ref>Instead of creating a whole new dictionary from scratch, you may try localizing the English dictionary available at [http://www.unlweb.net.br/resources/UCA1/eng_unl_dic.txt eng_unl_dic.txt]. Note that "localization" is not the same as "translation". You may need other features (in English, for instance, nouns do not have gender or case) or other entries. In any case, the resulting dictionary should reflect your translated version of the corpus (i.e., all entries appearing in your translated version of the corpus should appear in the dictionary). For further information on localization, see [[Localization]]. For information on the dictionary structure, see [[Dictonary Specs]]. For an explanation of the structure of the English dictionary, see [[English_dictionary\|English Dictionary]]. In case you need additional features, use only the tags available at the [[tagset]].</ref>
	##Create a dictionary for your training corpus. Your dictionary must contain <u>all and only the word forms appearing in your training corpus</u>, and must comply with the [[Dictionary Specs]].		##Create a dictionary for your training corpus. Your dictionary must contain <u>all and only the word forms appearing in your training corpus</u>, and must comply with the [[Dictionary Specs]].
	##Save the NL-UNL dictionary in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>IAN>DICTIONARIES.		##Save the NL-UNL dictionary in a plain text (.txt) file with UTF-8 encoding and upload it to UNLWEB>UNLDEV>IAN>DICTIONARIES.
Line 22:		Line 22:
	#Evaluation		#Evaluation
	##Export the actual output of IAN (range=1-100, trace-level=NONE).		##Export the actual output of IAN (range=1-100, trace-level=NONE).
	##Check the F-measure of the actual output at UNLWEB>UNLARIUM>TOOLS>F-MEASURE.<ref>The F-measure compares the actual output (exported from IAN) with the expected output (available at [http://www.unlweb.net.br/resources~~/corpus~~/UCA1/uca1_unl.txt uca1_unl.txt])</ref>		##Check the F-measure of the actual output at UNLWEB>UNLARIUM>TOOLS>F-MEASURE.<ref>The F-measure compares the actual output (exported from IAN) with the expected output (available at [http://www.unlweb.net.br/resources/UCA1/uca1_unl.txt uca1_unl.txt])</ref>
	##If the F-measure > 0.9, upload the corpus, dictionary and grammars to VALERIE (UNLWEB>VALERIE>CLEA1500) in order to have your exercise evaluated.		##If the F-measure > 0.9, upload the corpus, dictionary and grammars to VALERIE (UNLWEB>VALERIE>CLEA1500) in order to have your exercise evaluated.