Cellulose and its derivatives emerge as highly beneficial candidates for solid electrolytes. Their advantages include low-cost production, no leakage, biodegradability, solid-state stability, ease of processing, and good electrochemical stability. Although a strong ionic conductivity is essential for cellulose-based solid electrolytes, the relatively low ionic conductivity of current cellulose-based solid electrolytes remains a substantial challenge. This research seeks to address this critical gap with the development of a novel composite polymer electrolyte (CPE). Our approach utilizes a primary polymer matrix of cellulose acetate, which is supplemented by nanofillers, polyoxometalate (POM), along with LiPF6 as the lithium salt and (polyethylene glycol) PEG as a plasticizer. The strategic incorporation of nanofillers is crucial as they enhance ionic conductivity and aid in the formation of pathways for ion movement within the electrolyte. The optimized CPE with a 5 wt.% nanofiller demonstrates an electrochemical stability window greater than 5 V, a lithium-ion transference number of 0.58, and an impressive ionic conduction of 2.2 × 10−4 S cm−1 at room temperature (RT). The prepared CPE indicated 86.03% Coulombic efficiency, the charge capacity value of 169.7 mAh/g, and capacity retention was more than 98% after 25 cycles at 0.5 C. Also, the electrolyte has good compatibility with lithium-metal anodes, which allows the lithium symmetric cells to run steadily for over one hundred hours at 0.3 mA cm−2 current density.