Sunshine Superman by Donovan was the first album I ever bought, circa 1969. It cost me $5.25 from Palings in George Street, Sydney, and I have it still. It was on Epic with a yellow label and it was a thing of mystery and wonder. The title track was the hit single but that was just the door into a strange world. On the album there’s ‘Season of the Witch’, with its classic line ‘beatniks out to make it rich’, which summed up the exotic new world of the ’60s that was seductive and dangerous; ‘The Trip’, a Dylanesque story about taking LSD; ‘The Fat Angel’ with the admonishment ‘fly Jefferson Airplane, get you there on time’; or ‘Bert’s Blues’, a slice of steamy, slightly sleazy jazz blues that pays tribute to guitar player Bert Jansch.

Sunshine Superman was like a rock that landed in the pond – the ripples just went out and out. ‘Season of the Witch’ is a trip all in itself, across its many and varied versions (including Mike Bloomfield’s and Stephen Stills’ 16-minute improvised version, or the numerous versions used in films of the ’60s as few songs sum up the dark side of the ’60s better). Then there are the resonances through people like producer Joe Boyd, who named his production company Witchseason and went on to bring Nick Drake and Fairport Convention, the Incredible String Band and R.E.M. to the wider world. And then there are all of the other versions by Terry Reid, Julie Driscoll, Hole, Luna, Dr. John, Lou Rawls, Joan Jett, Richard Thompson and Karen Elson. Just that one song will lead you into a whole pantheon of music. There is also the journey you could take with Bert Jansch, following the trail led by the song ‘Bert’s Blues’. Jansch was one of the masters of English music, whose unique guitar style was the bedrock of Jimmy Page’s style and from Page whole generations of heavy metal. And, of course, Page himself plays on ‘Sunshine Superman’. Although only the title track of Sunshine Superman got any radio play, the meat of the LP was all in the album-only songs.

The point of this lengthy trip down my memory lane is to illustrate how albums can be more than the sum of their parts. There have been songs that changed the way people saw the world, for example: ‘I Want to Hold Your Hand’ (the Beatles), ‘(I Can’t Get No) Satisfaction’ (the Rolling Stones), ‘That’s All Right (Mama)’ (Elvis Presley) or ‘Smells Like Teen Spirit’ (Nirvana). But the album as a piece of work was something that you could explore and live with. (I use the past tense because in this digital age albums have a more precarious, uncertain place.)

The classic album format was a long time coming. In April 1955 Frank Sinatra released In the Wee Small Hours, one of the first 12-inch vinyl records. The album contained songs that all fitted around the theme of despair and lost love but they were all expressed with a lush emotionalism. Sinatra wanted the album to make a singular statement. Sinatra’s success notwithstanding, singles dominated music in the 1950s and early 1960s. The demands of Top 40 radio determined the shape of pop music, as songs needed to be around three minutes to accommodate advertisers.

Artists pumped out singles and twice a year compiled those singles with whatever happened to be around to make an LP. There were titles like Ella Fitzgerald Sings The Cole Porter Songbook, which did have a conceptual framework. And then, as jazz took on greater artistic aspirations, works appeared like Miles Davis’ Kind of Blue (page 126), which was an album statement. The individual tunes existed on their own but were greatly enhanced by being within the larger format of the album.

The merging of the Beatnik era, the folk music boom and pop music, which occurred in the early 1960s, gave a greater artistic depth to popular music. By 1965, when Bob Dylan was planning Highway 61 Revisted, even the lead track ‘Like a Rolling Stone’ was too long to be a single – early versions were split across two sides of a 45 rpm disc. Dylan’s song was sufficiently powerful to get radio airplay despite its six minutes-plus length, but that was still an anomaly.

Then in 1967 the Beatles released Sgt. Pepper’s Lonely Hearts Club Band, an album that was conceived to fit an overarching concept. After Sgt. Pepper’s, artists began to think of their work differently. We had an age of rock operas and concept albums and triple albums and whatever else. The album, because of its higher unit price and greater profitability, changed the music business and turned it into a profit-making machine.

But it was in the late 1980s that the rot began to set in. The compact disc format provided digital quality reproduction, a more resilient object and 70 minutes with which to record music. In retrospect, 40 minutes – the optimum running time of a vinyl LP – is the ideal format for a popular music ‘long player’. CDs just go on too long.

Now we have digital downloads. Yes, the compression required for internet distribution does degrade the sound, but more damaging is the way that these single tracks are disassociated from any context, possibly robbing the songs of meaning and certainly divorcing them from being part of a bigger narrative. The beauty of the LP was that in listening to ‘Sunshine Superman’ you also found ‘Season of the Witch’. That serendipity will happen less and less as time goes on.

My personal history notwithstanding, Sunshine Superman doesn’t make this list of the 100 best albums of all time. Nor does In the Wee Small Hours or Sgt. Pepper’s. The methodology we adopted for creating the list was that, regardless of genre or era, the albums had to be ‘the best’; full of undeniable artistry and internal qualities that have continued to resonate through the years. All the records included here are of huge, rare artistic merit. They aren’t necessarily the biggest sellers or even the most influential (although very often they are both of these).

For us, Sgt. Pepper’s is a line in the sand. Too often critics and lists have praised Sgt. Pepper’s because of its game-changing qualities, encouraging popular artists to focus on a body of work rather than just individual songs, and because it marked the end of the Beatles as a performing (read: teenybop) band and the beginning of them being a more experimental (read: serious) studio band. And obviously it was incredibly successful – the first genuine blockbuster album. However, while Sgt. Pepper’s may have boasted a high concept, the conceptual thread is weak and it is arguably the band’s most patchy collection of songs. Yes, there are moments of greatness (‘A Day in the Life’, ‘Lucy in the Sky with Diamonds’, ‘She’s Leaving Home’) but equally there are weak, throwaway moments (‘Good Morning Good Morning’, ‘Lovely Rita’, ‘When I’m Sixty Four’). The spirit of Sgt. Pepper’s was, in fact, better articulated by the Kinks on The Kinks Are the Village Green Preservation Society (page 220).

For the next 30 years artists thought in terms of creating albums – building a format that could tell multi-dimensional stories. Fleetwood Mac’s Rumours (page 44), for instance, famously documents the conflicts of the couples within the band. These soap operas are contained within one of the most complex, orchestrated pop operas. Similarly the Beach Boys’ Pet Sounds (page 58) and Van Morrison’s Astral Weeks (page 32) are works that push the boundaries of what constitutes popular music. Layla and Other Assorted Love Songs by Derek and the Dominos (page 248) is a double album dedicated to unrequited love, and you’d be hard pressed to find a novel or a film on the subject that is as nuanced, emotive and rigourous. All of these statements needed the space of an album to really get into the nitty gritty and properly tell their story. ‘Layla’ by itself is a terrific song edited into a single, but then you add the piano coda and it stretches to seven minutes and its beauty truly unfolds. Look at it in the context of the rest of the album – from the elegiac, medieval Persian love poem, ‘I Am Yours’, to the desperate Freddie King blues, ‘Have You Ever Loved a Woman’, and you’re starting to see the full picture. Iggy and the Stooges’ Raw Power (page 112) is the diary of a madman with a backbeat. No Russian author has been more creatively deranged than Iggy Pop and James Williamson were when making that record. The Ramones, who seemed completely dedicated to the glory of the 45 rpm single, made a debut album – in a shorter time than most groups take to make a single – that was an art statement of the order of Andy Warhol’s soup cans, from its songs to its amazing black and white cover.

So, in every case we went for the definitive collection of songs that, when put together on the one release, were bigger than the sum of their parts. Perhaps the only arbitrary restriction that we loosely imposed was a limit of two albums per artist – and it was a restriction that we failed to adhere to. In reality that restriction was a self-regulator, another reminder to ourselves to stay true to the course and constantly challenge our own choices. We have also kept one eye on the notion of the album as a discrete statement in itself. For this reason we have not included greatest hits collections or the like. Almost all of these albums are as the artists themselves intended them. This has meant that we don’t have an Elvis Presley album. (The Sun Sessions LP that occurs on many ‘best of’ lists was compiled as a ‘greatest hits’.) Two of the LPs from the 1950s, by Little Richard and Chuck Berry, were compiled from different sessions but they do tell a consistent story. In a sense they prefigure the shape that albums took on in the 1960s.

This is mostly a list of rock & roll and pop records. We thought that stretching across too many genres would ultimately make a list like this redundant, so we have stuck to the brief of rock & roll and its near neighbours. There are other books that obviously can be written about jazz or country or blues music alone. We have only included albums from those genres where they crossed over to sit within the broader definition of popular music. Having said that, we were cognisant of the different shapes that contemporary popular music takes. The key albums in soul, rap, R&B, alternative, punk, metal and hardcore are all represented on this list, but only because they forced themselves on the list of ‘the best’.

Obviously, this is not the first project to codify the 100 best albums of all time. Bearing that in mind, we thought that for the list to have real value we needed to rethink some of these records, challenge some of the perceptions that are held about them, and spark some debate. There are controversial choices here, such as the Monkees – often derisively referred to as the Pre-Fab Four. The Monkees remain an interesting harbinger of the way that music has evolved over the past 10 years through synergies with television. Comparing the career trajectory of the Monkees with any number of TV talent show victors is a salutary lesson in what ails the music business. (It’s also worth noting that the Monkees’ album Headquarters – page 156 – featured country rock before the Byrds and Bob Dylan got there.)

The Top 10 is always problematic. There is an accepted canon among music critics of certain albums that represent a peak achievement – Astral Weeks and Pet Sounds are two of them. These records were not major hits on release, so it was important to re-examine the albums and put them to the test rather than just reiterate critical platitudes. Of the other records in the Top 10, Nirvana’s Nevermind (page 28) was the catalyst for a generational change in popular culture, and it’s unlikely that we will ever see another album that so clearly draws a line in the sand between eras. Public Enemy’s It Takes A Nation of Millions to Hold Us Back (page 52) was similarly the album that gave rap music a new sonic and literary vocabulary. The Velvet Underground & Nico was for white rock & roll what Public Enemy was for rap. It gave us punk and alternative music and it still sounds fresh today. Rumours has never had the critical cachet it deserves. Not even the Beatles achieved a complete collection of pop songs burnished so exquisitely as Fleetwood Mac did on Rumours. By the same token, no singer-songwriter ever exposed themselves as nakedly as Joni Mitchell did on Blue (page 36). That album is one of the most unvarnished, raw and minimal records ever made.

We hope to create controversy and trigger debate, but only because that kind of discourse is healthy, and any and all focus on these great artistic monuments is positive and deserved. There’s no point in just having an opinion, though. For each of these 100 entries we carried out extensive research, filing through interviews with artists, producers and musicians. We have looked behind the velvet rope to bring out the story behind the album and to see how much of the final product was the intention of the artist.

Our methodology for compiling these lists was to take our own hard-won opinions and match them against lists from reputable sources – NPR radio, Mojo magazine, Rolling Stone magazine, NME, Pitchfork, the BBC, Metacritic and more. Our unique position is that we are Australian writers, so therefore we hope this book avoids the UK- and US-centric biases that usually dominate this debate. It was important when compiling the list to take those subjectivities into consideration – which is why Donovan’s Sunshine Superman hasn’t made the cut. But each of the 100 albums that has made it here has the richness and the texture of that record.

Although it’s fair to say that rock & roll did have a golden age that ran from the Beatles to Nirvana, it’s also true that music continues to surprise, inspire and illuminate our lives and times. Hopefully this book will encourage you to reconsider, revisit, recalibrate … and rejoice in the magic of these great albums.

Toby Creswell