2020 CWE Top 25 Analysis
The purpose of this blog is to provide the community with insight into the data analysis activities associated with the development of the “2020 Common Weakness Enumeration (CWE™) Top 25 Most Dangerous Software Weaknesses (CWE Top 25).” It is intended to supplement the methodology previously published on the 2020 CWE Top 25 page on the CWE website, and to provide further transparency into the technical process behind calculating the final list.
NVD Data Operations
The process to create the 2020 CWE Top 25 began on April 23, 2020 by downloading vulnerability data (in JSON format) from the National Vulnerability Database (NVD) for the years 2018 and 2019. There were 16,169 entries from 2018, and 15,370 entries from 2019 (total 31,539). We removed 1,694 of these entries classified as “REJECTED,” yielding 29,845 entries to analyze.
The remaining data was then processed by a series of scripts to parse the JSON, extract information needed for performing the analysis, and generate reports that determined which entries to analyze. These reports split the data into smaller subsets of entries related to: (1) automated keyword matching; (2) sources that might contain insufficient information; (3) CWE Categories (removing Categories was a focus of the 2019 CWE Top 25); or (4) CWE Classes.
The information extracted for each entry included the Common Vulnerabilities and Exposures (CVE®) Identifier (CVE ID), original CWE(s) mapping by NIST, CVE description, and reference(s) were collected in a file. Given the overall size of the data, the reports were kept small with roughly 50–100 entries in each, allowing the analysts to process them one at a time. Different milestones were created, roughly in order of increasing complexity of the CWE mappings, with entries mapped to high-level weaknesses (i.e., Class-level CWEs) scheduled near the end of the analysis exercise. The bulk of the entries considered for analysis came from the 2019 NVD data, since many entries from 2018 had already been analyzed for the 2019 CWE Top 25.
In preparation for the final 2020 CWE Top 25 calculation, the CWE Team downloaded a new version of NVD on August 11, 2020, which contained 32,143 entries. After excluding 1,782 entries classified as “REJECTED” CVEs (yielding a total of 30,361), there were 516 new entries to be considered for analysis in the August 11 dataset, compared to the April 23 dataset. This change occurs because vulnerabilities from previous years are added to NVD on a regular basis due to numerous factors in vulnerability disclosure and publication. The CWE Team did not perform a remapping analysis on these newer entries, but the 2020 Top 25 list was calculated using the August 11 version. This ensured that the official CWE Top 25 list would be as up to date as possible and minimize apparent inconsistencies that might be detected by other parties attempting to replicate the 2020 CWE Top 25 calculations by downloading the latest version of NVD. We compared the CWE Top 25 calculation for the April 23 dataset with the official calculation of the August 11 dataset and found no significant change in rankings.
Mapping Priorities
Prioritizing which areas to concentrate on for this year’s analysis effort was crucial, and one that was determined based on an overall feedback received from the community. Last year’s list focused on remapping entries in NVD that were mapped to CWE Categories. This year’s goal was to further improve the NVD data by also analyzing mappings to high-level CWE Pillars and Classes (e.g., CWE-20, CWE-200, CWE-119, and CWE-284). We continued to analyze NVD entries mapped to CWE Categories, especially when the entries appeared to be relevant to the weaknesses that would likely be in the CWE Top 25 or On-the-Cusp lists. Fortunately, since Categories are no longer used in NVD as of September 2019, this is the last time we had to account for them.
Analyzing NVD entries mapped to high-level CWE Pillars and Classes allowed us to identify more specific mappings that we considered to be more appropriate. Many of these mappings were normalized to what we call a Base-level of abstraction, which sits in between higher-level (Class and Pillar) and lower-level (Variant) abstractions. We believe that Base-level CWE entries are the most appropriate when mapping vulnerabilities to their related weaknesses. This goal was appropriately balanced by evaluating other CWE subject areas as seen throughout the entire dataset to provide a comprehensive scope of analysis. This was imperative because with so many mappings to high-level weaknesses such as CWE-20, CWE-200, and CWE-119, analyzing all of them would have meant less effort for other weakness types in the CWE Top 25 and On-the-Cusp lists, and a heavy bias toward those few weakness types being analyzed. Since all NVD entries couldn’t be looked at, we focused on all the major areas seen in the data to provide sufficient coverage. For example, we created batches of reports related to access control (authentication/authorization/permissions/privileges), injection (with many Base-level weaknesses besides OS command injection), etc. Once we laid out a plan, appropriate reports (as described above) were created for the analysis.
The end result was a mapping analysis for thousands of CVE entries. In August 2020, we provided 10,295 mappings to NIST for 8,882 unique CVEs. We excluded 181 of our mappings to CWE-20 (count of 143) and CWE-200 (count of 38) when the associated CVEs already mapped to more specific CWEs, although these mappings were used in our Top 25 calculation. In September 2019, we had sent 3,381 mappings to NIST for 3,321 unique CVE-2018-xxxx entries. Combining our mappings for 2018 and 2019 CVEs, we produced 13,286 total mappings for 11,673 unique CVEs (5,367 CVE-2018-xxxx and 6,306 CVE-2019-xxxx). Note that the numbers do not add up precisely because some CVEs were analyzed in both years.
Keyword Matcher
Milestone 1 of the data analysis began with what we call keyword matches. The raw NVD data was analyzed by a script to look for CWE ID, CWE Name, CWE Alternate Name, and common CWE-relevant phrasing available in the CVE descriptions. If the matcher detected one of these fields, then it would “guess” a CWE mapping; if the CWE mapping differed from what was in the NVD entry, then the guessed mapping would be proposed to the analysts to verify. For example, the phrase “SQL injection” or “SQLi” would suggest a mapping to CWE-89, although it should be noted that this is only an illustrative example, since SQLi and other easily-described weaknesses such as XSS were already well-handled by NVD. The analyst would then read the description, and if they agreed with the guessed mapping then it was kept; otherwise the analyst would find and suggest another CWE(s), if applicable. All analysts’ suggestions were verified by the tech lead of the task, and only mutual agreements were selected as the final mappings. The tech lead review was important to ensure some reasonable consistency in mappings across all team members.
Manual Analysis
Other milestones consisted of reviewing all NVD entries mapped to a specific CWE (e.g., CWE-20). We tried to avoid subjectivity introduced by the analysts by requiring the analyst to quote the relevant portion of the description or reference(s), and then reviewing the selection via independent verification.
Manual analysis started with the analysts reading the associated CVE descriptions to identify a phrase or sentence that would indicate a weakness. If they found something, then they would assign an appropriate CWE(s) along with a comment indicating why they mapped to that CWE(s). The goal of this exercise was to provide the most specific mapping possible without being concerned with the abstraction level of the chosen CWE. If the analysts didn’t find any weakness information, then they skimmed through the reference(s). It is important to note that given the limited time and resources, the analysts couldn’t read every single reference in detail or search the web for other references not listed in the NVD entry. Useful references often contained additional details related to a specific exploit (such as those published on Exploit-DB, PacketStorm, etc.); a vendor’s blog post; the original bug report (which may show source code diffs or describe the mistake to the developers); or, a security advisory by a third-party vulnerability researcher or coordinator. If the CWE analysts found appropriate details, then they would capture them with a CWE mapping as specified above.
If analysts still didn’t find any weakness information from the references, then that NVD entry was marked with the “NVD-CWE-Insufficient-Info” label (called “CWE-noinfo” in NVD) to indicate that there was not enough information available to determine the underlying weakness. Other than mapping to specific CWEs and using “NVD-CWE-Insufficient-Info”, we also used these labels: “CWE-GAP”, where a CVE clearly identified an area of weakness, but that wasn’t covered in CWE; “CWE-RESEARCH”, where we needed to dig deeper to understand the relationships within CWE as well as investigate the weakness area further; and “UNSURE”, where it wasn’t clear which CWE(s) would be mapped (if any). Again, all analysts’ mappings were verified by the tech lead, and mutually agreed mappings were finalized.
This table identifies the levels of abstraction from the originally mapped CWEs and the results from our remappings, showing the shift to lower-abstraction CWEs. This demonstrates that, with sufficient data and time spent on CWE mappings, more precise weakness information is sometimes available. Note that these totals do not match the total number of mappings we performed, because some entries could not be mapped to a CWE weakness.
Below are some stats for the special labels we identified above. For brevity, we have limited each table so that it only covers the most commonly encountered CWEs as originally mapped by NVD.
NVD-CWE-Insufficient-Info: Total: 2,951 CVEs (1,528 from year 2018, 1,423 from year 2019). Note that CWE-264, CWE-254, CWE-399, and CWE-255 are Categories, which are no longer being used in NVD as of September 2019.
CWE-GAP: Total: 101 (78 from year 2018, 23 from year 2019). Notice that these are Categories or high-level CWEs. Less than 1 percent of entries seemed to suggest a CWE gap, demonstrating CWE’s comprehensiveness with respect to publicly known vulnerabilities.
CWE-RESEARCH: Total: 545 (102 from year 2018, 443 from year 2019). The most popular original NVD maps labeled CWE-RESEARCH.
UNSURE: Total: 447 (171 from year 2018, 276 from year 2019)
It is understood that our approach to mapping can certainly be improved, as highlighted by authors from NIST, INMETRO, and UADB-Senegal in “Measurements of the Most Significant Software Security Weaknesses” (to be published at ACSAC in December 2020 https://www.acsac.org/2020/program/papers/). We strive to improve our methodology with each year’s CWE Top 25 list and will certainly consider community members’ thoughts on that. If you have ideas on how our list and processes associated with it can be improved, please reach out to us at cwe@mitre.org.
Most Common CWEs Analyzed
While our remapping tasks covered a wide variety of CWEs in the original NVD entries, mostly due to keyword matches, these were the most commonly analyzed CWEs:
For the entries that we remapped, there were 78 unique original CWEs. After remapping, this produced 406 unique CWEs. The following table identifies the 20 CWEs that we most frequently remapped to. Note that some entries such as CWE-122 and CWE-121 are not part of the 1003 view as used by NVD, so these were “normalized” to CWE-787 in the 2020 CWE Top 25 calculation and in the updated mappings as sent to NIST:
The following are the most common remap combinations, ignoring NVD-CWE-Insufficient-Info (which is covered in a different section). Notice how the lack of sufficient information (by our assessment) effectively forced the selection of CWE mappings to high-level Classes or Pillars (CWE-284, CWE-264, CWE-200, CWE-20). In hundreds of cases, we had to preserve the original high-level mapping because there was no lower-level detail; notice the number of original mappings to CWE-20 that remained at CWE-20 after remapping, as well as CWE-200 and CWE-119.
During this analysis activity, we noticed that many entries were originally mapped to CWE-77 (“Command Injection”), but deeper analysis of references made it clear that the underlying weakness was related to CWE-78 (“OS Command Injection”). We believe that this indicates a lack of distinction between the two phrases, even though there are numerous command languages that do not involve OS commands.
Also notice that many NVD entries were originally mapped to the high-level CWE-119 Class, but they were remapped to a lower-level Base or Variant weakness (e.g., the CWE-787 Base, or the CWE-122/CWE-121 Variants). These changes were often found by keyword matching (e.g., “heap-based buffer overflow” for CWE-122) or by closely examining references. We also decided that the phrase “memory corruption” was sufficient to indicate some kind of out-of-bounds write, so in the absence of any other details, we remapped those to CWE-787. Based on this logic, the term “Memory Corruption” was moved from CWE-119 to CWE-787, under “Alternate Terms.”
View 1003
Once the analysis was completed and verified, it was then put through our script to normalize our mappings based on the CWE View “Weaknesses for Simplified Mapping of Published Vulnerabilities”, known as “View 1003”. This view is used by NIST in performing their mappings within NVD and hence all remappings that we perform must eventually be to CWEs found in this view. To satisfy this requirement, we used the defined parent/child relationships within CWE to convert our specific mappings to the appropriate weakness found in View 1003. Note that certain weaknesses weren’t part of View 1003 prior to the release of the 2020 CWE Top 25 but were deemed necessary for inclusion: CWE-77, CWE-917, and CWE-1236. These entries have been added to View 1003 to provide more coverage, and NIST has already accepted this change and is implementing them in their process.
With the View 1003 requirement met, our scoring methodology was then leveraged to determine the final 2020 CWE Top 25 and On the Cusp lists.
Frequent Lack of Weakness Details in CVE Descriptions
CVE descriptions typically consist of elements such as product name, affected versions, whether the attacker is local or remote, technical impact such as code execution or denial of service, and affected components or parameters. These elements are useful for vulnerability management, but not for mapping to CWE. CWE mapping analysis requires knowledge of the underlying developer mistake(s) that actually led to the specific vulnerability as identified in a CVE entry. In some cases, attack terminology (e.g., “SQL injection”) is sufficient to indicate the weakness, but in the absence of such commonly used phrases, it can be difficult to determine the problem. For the purposes of CVE and NVD, users generally do not need to understand the underlying weakness at all, since it does not affect their decision-making. Also, some sources may intentionally obscure details in order to make it more difficult for potential attackers to figure out the underlying issue.
For these reasons and others, weakness information is often omitted, or there is minimal effort to diagnose and appropriately describe the underlying mistake within CVE descriptions. There is a strong dependency on the original author of the CVE description, or for references to provide technically deep explanations of the vulnerability, such as bug reports in open source products, or detailed exploits. The identification of the underlying weakness is therefore shifted to NVD analysts, and/or to the CWE Team as part of the CWE Top 25 remapping analysis.
The quality of weakness analysis across the entire CVE repository could be improved significantly by the original vendors, by either providing deeper technical details, or in providing lower-level CWE mappings. We urge vendors who do not provide a CWE(s) mapping in their publications, or only use high-level CWE(s), to provide appropriate weakness information and mappings.
Handling Insufficient Information in CWE Mapping
As described in the previous section, there are many CVE vulnerability descriptions that do not include enough information to determine what the underlying weakness is because knowledge of the weakness is not essential for performing vulnerability management and prioritization. This poses a special kind of problem for mapping a CVE entry to the appropriate CWE(s).
Many CVE entries are published by vendors who only describe the impact of the vulnerability without providing details of the vulnerability itself. For example, over 2,900 CVEs from 2018 and 2019 did not have sufficient information to determine the underlying weakness. In other cases, the CVE description covers how the vulnerability is attacked — but this does not always indicate what the associated weakness is.
There are several ways in which a CVE entry and/or its references do not contain sufficient information to map to a specific CWE, or at least force a map to high-level Classes:
- impact emphasis — the information only describes the impact of the vulnerability, e.g., “read data” or “obtain sensitive information.” Without knowing the specific programming error that allows this kind of impact, the analyst might be forced to use a high-level CWE such as CWE-200, which is only intended for weaknesses that explicitly insert sensitive information into output, but could be interpreted as being useful for any issue involving a loss of confidentiality. In many other cases, however, a guess is not even possible, with phrases such as “read files,” “execute code,” or “cause a crash.”
- attack emphasis — the available information might only describe the attack that is being conducted, and it is not always clear what weakness is being exploited. For example, if a long input to a program causes a crash, the cause of the crash could be due to a buffer overflow, a reachable assertion, excessive memory allocation, an unhandled exception, etc. These all correspond to different, individual CWEs. In other cases, the attack generally indicates the underlying weakness, such as XSS (CWE-79) or SQL injection (CWE-89) or buffer overflows (CWE-119 or its descendants). Ironically, detailed bug reports that include crash dumps from fuzzers effectively contain all the details needed for developers to replicate and fix an issue — but it’s often buried within the malformed input that the fuzzer has found, and the underlying mistake (and associated patch) are not obvious, forcing reliance on the bug report’s title or brief description.
- generic attack phrases — these phrases give no indication of the mistake at all. Examples include “crafted request,” “malicious input,” “malformed object,” etc. These give no indication of the associated weakness; the analyst would need to be able to determine how the input is manipulated, and what software mistake prevents the input from being “converted” or otherwise neutralized to a safe value. In these cases, there may be a temptation to map to CWE-20 in absence of specifics, but many weaknesses — such as poor error handling — occur even if validation is correct.
- generic software behaviors — sometimes, the information only covers the software behavior that contains the weakness. For example, “does not properly [parse input]” only identifies the behavior of parsing, which is a normal behavior of most programs; the phrase itself does not specify the programmer error. The weakness is not that parsing is used; but the mistake is not identified. Another common example is “does not properly handle [regular behavior]” or “error in [generic behavior].”
- reliance on limited, high-level CWE — some vendors or researchers may only use high-level CWEs. In many cases, this might be the only weakness information available. The use of high-level CWEs could be due to (1) insufficient training for mapping to lower-level CWEs; (2) reliance on older subsets of CWEs (e.g., from the CWE-635 view, a set of 19 mostly high-level CWEs as used by NVD from 2008 to 2016, and reused by many other organizations); or (3) the vulnerability discloser’s reluctance to provide more specific details.
- use of generic weakness phrasing — for many entries, a phrase could be used that identified a high-level or generic weakness, such as “input validation error,” which aligns with the name of CWE-20.
We did not track the rationale for each individual decision to map to NVD-CWE-Insufficient-Info, but we were able to detect some common patterns in the comments by CWE analysts when they were forced to map to NVD-CWE-Insufficient-Info. For example, comments for 989 remappings suggested that the information was “impact-only” or “impact-focused,” and 285 remappings contained statements about “generic behavior.” Note that these two categories overlapped, i.e., some CVE descriptions with emphasis on impact also specified a generic software behavior.
Note: There may be different opinions within the CVE/NVD community about how to handle low-detail situations, but we as a community need to determine the best way to handle this going forward.
Conclusion
Mapping analysis of vulnerability data to CWE is a complex task, made even more difficult when the data itself isn’t complete or informative. As a result of this mapping activity, we plan to have an ongoing discussion with vendors and other vulnerability publishers, requesting them to provide more specific weakness-related information or more precise CWE mappings, and for us to offer better guidance and tools to the community.
