DOMAIN RESTRICTION ON VEUSZ SOFTWARE

By using the standard NLP practice of eliminating punctuation (e.g., separators) and stopwords (e.g., keywords) we find that code is less repetitive and predictable than was suggested by previous work. Furthermore, 40% of all 2-grams end in a separator, implying that a model for autocompleting the next token, would have a trivial separator as top suggestion 40% of the time. For example, the syntax of a language, especially the separators e.g., semi-colons and brackets, make up for 59% of all uses of Java tokens in our corpus. We find that much of the apparent “naturalness” is artificial and is the result of language specific tokens.

DOMAIN RESTRICTION ON VEUSZ SOFTWARE

The first part replicates recent works that have concluded that software is more repetitive and predictable, i.e. This work is organized as a “manuscript” thesis whereby each core chapter constitutes a submitted paper.

In the second part, we study how well English discussions of source code can be aligned with code elements to create parallel corpora for English-to-code statistical machine translation. In the first part of the thesis, we study the predictability of source code and determine how well source code can be represented using language models developed for natural language processing. Analyzing source code using computational linguistics and exploiting the linguistic properties of source code have recently become popular topics in the domain of software engineering.

Domain restriction on veusz