In this work, a new nanocomposite, polythiophene (PTH)/carboxylic acid-functionalized multiwalled carbon nanotubes (MWCNTCOOH) and reduced graphene oxide (RGO), was prepared by an in situ oxidative polymerization route with different mass ratios (2%, 5%, 10%, and 20%). The synthesized PTH/MWCNTCOOH-RGO were then used to modify a glassy carbon electrode (GCE) to construct a voltammetric sensor for sensitive detection of Hg(II). The modified polymers were described by different microscopic, spectroscopic, and thermal analysis techniques. Electrochemical features of the new electrodes were identified by cyclic voltammetry (CV) and square wave voltammetry (SWV) techniques. The PTH/MWCNTCOOH-RGO (10%)/GCE exhibited high performance for Hg(II) detection compared to other nanocomposites. The PTH/MWCNTCOOH-RGO (10%)/GCE showed a low detection limit (0.009 µM); wide linear range (0.1–25 µM); and excellent stability, reproducibility, and selectivity. The practical application of the PTH/MWCNTCOOH-RGO (10%)/GCE was confirmed by the detection of Hg(II) in a tap water sample, and the recovery was between 110.7 and 96.79%.