The Domain Name System (DNS) remains a foundational element of internet functionality, yet it is inherently vulnerable to various cyber threats due to its lack of built-in security features. In response, DNS-layer security mechanisms such as DNSSEC, DNS-over-TLS (DoT), and DNS-over-HTTPS (DoH) have been introduced to bolster security and privacy. However, the implementation of these protocols has raised concerns regarding their impact on network performance. This systematic review examines literature from 2015 to 2025, analyzing the dual effects of DNS-layer security on threat mitigation and network efficiency. Using a structured PRISMA-based methodology across six major academic databases, 23 peer-reviewed review articles were analyzed. The findings highlight the effectiveness of modern AI and ML-based approaches in enhancing encrypted DNS traffic detection, the performance trade-offs associated with encryption techniques, and the need for adaptive and privacy-aware security frameworks. The review also identifies key gaps and emerging trends, proposing directions for future research, including quantum-resilient systems and lightweight models for resource-constrained environments.