Some patterns of symbols are sources of power. Mysterious magical spells? In fantasy, yes --- but in real life, enchantments are carried by resonant words: stirring speeches, catchy slogans, Shakespearean sonnets, alliterative aphorisms, and so forth.
Many benefits flow from memorizing great symbol-sequences. There's a simple pleasure in having lovely language at one's tongue-tip, just as there is in mentally replaying a nice tune or envisioning a pretty picture. Well-wielded words, internalized, help one write and speak better: quotation, allusion, and more subtle riffs on successful patterns all become easier. And a bit of rote study can strengthen the old bean, as Arnold Bennett noted in Mental Calisthenics (see ZhurnalAnniversary2).
What to learn "by heart"? Obviously there's no one answer; it's a choice every individual has to make, depending on interests, background, sophistication, native abilities, and time available to invest.
My own list? Thus far, it reflects a mental attic that's embarrassing in its dust and disorganization. Like many, I've got potshards of historic political speeches rattling around the cranial cavity: Lincoln's Gettysburg Address; the Preamble to the US Constitution; and the initial sentences of the Declaration of Independence. From Shakespeare I can summon a few spirits from the watery deep, including most of the Henry V "Crispin's Day" speech (see CrispinCrispian), much of Hamlet's soliloquy, the final lines with which Puck ends Midsummer Night's Dream, and the "Shall I compare thee to a summer's day?" sonnet. In other spheres poetic, Yeats's "Leda and the Swan" coexists peacefully with Shelley's "Ozymandias". Richard Wilbur's clever "Mind" vies with fragments of Barrett, Browning, and Bible verses. Silly limericks and double dactyls sporadically surface into consciousness like annoying background music, as do catchphrases from Monty Python and Douglas Adams, or lyrics from Beatles and Bob Dylan.
But the first poem I learned under my own initiative? It was the summer of '69, and I was a high school soon-to-be senior, away from home for the first time, at a National Science Foundation mathematics enrichment program (see BookhouseBoy). I picked up a poetry textbook left behind in a dusty classroom at Southern Methodist University, and it fell open to Robert Graves's "The Naked and the Nude". Needless to say, with a title like that a teenager was going to read on ... but what I found was funny intellectual wordplay along with some philosophical musings on death and destiny. No regrets --- it was worth memorizing.
But my bottom line: not much to report from several decades of lackadaisical study. Better get to work, eh? The list to learn is long ...