F-measure: Difference between revisions
imported>Martins No edit summary |
imported>Martins No edit summary |
||
| Line 37: | Line 37: | ||
A result is considered '''"CORRECT"''' when the difference between the actual result and the expected result is less than 30%<br /> | A result is considered '''"CORRECT"''' when the difference between the actual result and the expected result is less than 30%<br /> | ||
WHERE<br /> | WHERE<br /> | ||
The difference between the actual and the expected result is calculated by the formula | |||
(exceding_words+missing_words)/(total_words) | (exceding_words+missing_words)/(total_words) | ||
WHERE | WHERE | ||
Revision as of 21:23, 1 April 2013
In the UNL System, the F-measure (or F1-score) is the measure of a grammar's accuracy. It considers both the precision and the recall of the grammar to compute the score, according to the formula
F-measure = 2 x ( (precision x recall) / (precision + recall) )
In the above:
- precision is the number of correct results divided by the number of all returned results
- recall is the number of correct results divided by the number of results that should have been returned
- RETURNED
A result is considered "RETURNED" when the output is a graph (i.e., all the nodes are interlinked) made only of UW's (i.e., without natural language words)
- CORRECT
A result is considered "CORRECT" when:
- The discrepancy of relations between the actual and the expected output is less than 0.3; AND
- The discrepancy of UW's between the actual and the expected output is less than 0.3; AND
- The overall discrepancy is less than 0.5;
WHERE
Discrepancy of relations is calculated by the formula:
(exceding_relations + missing_relations)/total_relations
Discrepancy of UW's is calculated by the formula:
(exceding_UW + missing_UW)/total_UW
Overall discrepancy is calculated by the formula:
((3*(exceding_relations+missing_relations))+(2*(exceding_UW+missing_UW)+(exceding_attribute+missing_attribute))/((3*total_relations)+(2*total_UW)+(total_attribute))
- exceding_relations is the number of relations present in the actual output but absent from the expected output
- missing_relations is the number of relations absent from the actual output but present in the expected output
- total_relations is the sum of the total number of relations in the actual output and in the expected output
- exceding_UW is the number of UW's[1] present in the actual output but absent from the expected output
- missing_UW is the number of UW's[1] absent from the actual output but present in the expected output
- total_UW is the sum of the total number of UW's[1] in the actual output and in the expected output
- exceding_attribute is the number of attributes[2] present in the actual output but absent from the expected output
- missing_attribute is the number of attributes[2] absent from the actual output but present in the expected output
- total_attribute is the sum of the total number of attributes[2] in the actual output and in the expected output
- RETURNED
A result is considered "RETURNED" when the output is a list of natural language words (i.e., without any UW).
- CORRECT
A result is considered "CORRECT" when the difference between the actual result and the expected result is less than 30%
WHERE
The difference between the actual and the expected result is calculated by the formula
(exceding_words+missing_words)/(total_words)
WHERE
- exceding_words is the number of words present in the actual output but absent from the expected output
- missing_words is the number of words absent from the actual output but present in the expected output
- total_words is the sum of the total number of words in the actual output and in the expected output
References
- ↑ 1.0 1.1 1.2 For the sake of comparison, UW's appearing in the source position are considered to be different from UW's appearing in the target position of a relation. Scopes are ignored.
- ↑ 2.0 2.1 2.2 For the sake of comparison, attributes appearing in the source position are considered to be different from attributes appearing in the target position of a relation. Optional attributes (such as .@def and .@indef) are ignored.
