Wadada Leo Smith portrait by Scott Groller

This year saw jazz continue its steady march into the mainstream. There might not have been a crossover jazz hit on the level of Kamasi Washington's The Epic, but like Kendrick Lamar's To Pimp A Butterfly in 2015, David Bowie's Blackstar set new standards for pop's engagement with the genre. Hiring saxophonist Donny McCaslin's band for his swansong was a masterstroke: the energy and invention they bring to Bowie's songs puts Black Star lightyears ahead of The Next Day's session muso rock. McCaslin returned the favour by covering two Bowie songs on his own album Beyond Now (Motema), underlining the influence of rock form and attack on contemporary jazz. McCaslin's anthemic and tightly structured approach can be a little too slick for weirdos like me, but he creates an undeniably powerful sound, aided, in part, by superhuman drummer Mark Guiliana.

If Blackstar is a brilliant example of rock musicians drawing on jazz, Esperanza Spalding's Emily's D+ Evolution (also co-produced by Tony Visconti) comes from the opposite direction. As the history of fusion shows us, jazzers can come a cropper when tackling rock for the simple reason that they're too damn good. But while the bassist and her band can certainly shred (and then some) they avoid the pompous showboating of the worst jazz rock. Her exquisite songwriting and inspired arrangements recall Joni Mitchell, Prince and Funkadelic, as well as the freakier moments of D'Angelo and Erykah Badu. And with its vocal hockets and intricate guitars 'Rest In Pleasure' beats the Dirty Projectors at their own game. Bow down.

The widely noted love-in between West Coast jazz and hip hop has led to a steady stream of releases from Kendrick Lamar's collaborators. Robert Glasper recruited the likes of Stevie Wonder, Bilal and Erykah Badu to remix and reinterpret Miles Davis on Everything's Beautiful. The results, however, were overly reverent, with a slightly dated nu-soul sound that his guests have long since moved on from. His latest album with the Robert Glasper Experiment, Art Science (Blue Note), mined an '80s r&b sound, the in-the-pocket grooves leaving too little space for improvisation. Finding the right balance between repetition and spontaneity has always been the challenge in any fusion of pop and jazz forms, and playing against the locked grooves of hip hop presents its own set of problems and opportunities. Trumpet player and To Pimp A Butterfly contributor Josef Leimberg sparks a cosmic jazz bifter on Astral Projections, slathering Space Echo horns, celestial choirs and toothsome bass clarinet over head-nodding beats. The sense of interstellar drift gives the instrumentalists space to stretch out, while the guest vocalists (Kurupt, Georgia Anne Muldrew) bring a great deal of character.

If the jazz and hip hop mainstream has perhaps played it a little safe, then Steve Lehman's Sélébéyone (Pi Recordings), which I discuss in detail in my top ten below, throws caution to the wind by attempting to integrate the wiggiest avant-jazz with the weirdest underground hip hop and electronica. As such, it's perhaps closer in spirit, if not sound, to the Afro futurist jazz experiments of acid house outlier Hieroglyphic Being, than the laid-back grooves of the West Coast mainstream. Hieroglyphic Being, aka Jamal Moss, delivered one of 2015's best albums in We Are Not The First, an audacious collaboration with jazz and improv heavyweights like Marshall Allen, Greg Fox and Shelley Hirsch. On New Year's Day, Moss joined British improvisers Orphy Robinson and Mark Sanders for a session at London's Café Oto, the heady results of which I reviewed for this column back in April. Black Focus (Brownswood), by London's Yussef Kamal strikes a pleasing balance between rare groove and contemporary bass music, creating a sound that is, as I wrote elsewhere, pure headphone umami. That jazz is big enough to embrace these different mutations is a cause for celebration.

The US avant garde delivered instant classics from Wadada Leo Smith and Henry Threadgill, plus very fine albums from William Parker, Ivo Perelman, Tyshawn Sorey, Mary Halvorson and Taylor Ho Bynum. Peter Evans and Nate Wooley continue to extend the possibilities of the trumpet, and following a several terrific projects of their own (the electronically detourned jazz-rock of Wooley's Argonautica and the grindhouse synth atrocities of Evans' Genesis are my pick of the bunch) they appeared together on Wooley's otherworldly sound installation Polychoral (Mnóad). Perhaps the most startling debut from a New York group this year, Elder Ones' Holy Science (Northern Spy) saw virtuoso vocalist Amirtha Kidambi channeling Hindu devotional music, Alice Coltrane and Linda Sharrock over hypnotic harmonium riffs and hard-driving avant jazz. On the epic closer 'Kali-Yura', producer Weasel Walter puts the rhythm section through the psychedelic grinder, rendering this already heady music all the more disorientating.

Relatively young scenes in Portugal, Poland and Lithuania are breathing new life and ideas into the European avant jazz and improv world, while veterans like Peter Brötzmann continue to seek out new collaborators. The Wuppertal Walrus's album with pedal steel maverick Heather Leigh, Ears Are Filled With Wonder (Not Two/Trost) is particularly notable. Her woozily psychedelic guitar brings out Brötzmann's lyrical loverman side, but the album is still an intense listen, with the duo forging an intoxicatingly weird take on the blues. Their live shows have been great too – not to be missed. The new generation of Scandinavian avant-jazzers continue to make great strides, with Mette Rasmussen, Anna Högberg and Sofia Jernberg releasing impressive leader albums or contributing to larger projects.

A new generation of players is also emerging in the UK, with several comfortably straddling the mainstream and avant garde. Sarathy Korwar, Laura Jurd's Dinosaur, and Huw Williams' HON all releasing promising debuts that combine accessibility with experimentation, while Sloth Racket presented a distinctly British take on the structured avant jazz of Downtown New York. One of the most exciting players on the London scene, Anglo-Danish saxophonist Julie Kjær delivered a highly inventive and engaging debut in Dobbeltgænger (Clean Feed), featuring the peerless rhythm section of Steve Noble and John Edwards. Glasgow lost a founding figure of its free music scene in bassist George Lyle, but the great man's final concert – a trio with Daniel Carter and Fritz Welch – has just been released on IORRAM. With festivals like Brighton Alternative Jazz Festival, Counterflows, GIOfest, Tusk, and LUME all supporting more adventurous jazz – not to mention the regular events popping up across the country – the future of this music is in good hands, whatever the Tory Brexit bastards throw at us.

Before I unveil my top ten releases of the year, a quick roundup of essential reissues. The second in Aum Fidelity's David S. Ware's archival series - following last year's incredible Apogee collection - Live In Sant'Anna Arresi 2004 finds the late saxophonist scaling the basalt columns of Matthew Shipp's piano. The set offers abundant proof of the pair's apparently telepathic interplay, as they shape a duo language that is both challenging and supportive. The final passage of 'Tao Flow Pt. 2' has it all: gorgeous balladry, deep gospel testifying and ecstatic leaps into the unknown. No visit from the jazz Santa is complete without Sun Ra's Singles: The Definitive 45s Collection, 1952-1991 (Strut). It even contains one of the sweetest festive songs of all time in the cosmic doo-wop of 'It's Christmas Time'. Finally, in addition to being one of the best labels for new music, Vilnius's No Business continues its sterling archival work, unearthing free jazz treasures from William Hooker, Peter Kuhn, Howard Riley and Roy Campbell Jr's Nu Band.

Ten. Konstrukt feat. Graham Massey & David McLean – Live At Islington Mill
(Astral Spirits)

Istanbul's interstellar jazz collective Konstrukt are tQ favourites and this live set from Salford's Islington Mill is one of two strong releases this year, the other being a collaboration with the brilliant Oxford-based pianist Alexander Hawkins. Konstrukt have worked with heavyweights like Marshall Allen, Peter Brötzmann and Keiji Haino, but sometimes it's the less starry collaborations that yield the most potent cosmic gravy. David McLean of Manchester's Tombed Visions label and Graham Massey of 808 State are the guests here. Contributing guitar, bass clarinet and fx, Massey's presence nudges the band towards an expansive sound that reimagines 70s Miles or Lanquidity-era Sun Ra through acid house and space rock. To mix my UK science-fiction references, a zarjaz thrill-ride through space and time.

Nine. William Parker feat. Lisa Sokolov & Cooper-Moore – Stan's Hat Flapping In The Wind
(Centering)

Every year brings a clutch of great William Parker recordings and 2016 is no exception. The master bassist joins drummer Gerard Cleaver on the Brazilian saxophonist Ivo Perelman's superb Art Of The Improv Trio Vol. 3 (Leo), and explores global folk forms with Joe Morris and Hamid Drake on Eloping With The Sun's magical Counteract This Turmoil Like Trees And Birds (Rogue Art). There's also the majestic Song Sentimentale (Otoroku), documenting Parker, Drake and Peter Brötzmann's January 2015 residency at London's Café Oto. Stan's Hat Flapping In The Wind sees singer Lisa Sokolov and pianist Cooper-Moore performing the songbook from Parker's unproduced musical, the fabulist tale of a spiritual journey. As his albums with Raining In The Moon attest, Parker is a great songwriter, and on Stan's Hat he synthesises his classic r&b influences with the American musical tradition and dash of modernist art song. Sokolov's formidable vocal prowess won't be to everyone's taste, but she interprets these songs magnificently, channelling the spirit on the gospel numbers, and sounding hushed and fearful on the chilling 'The Death Of Death'. Cooper-Moore is more than an accompanist, but even at his wildest, he serves the song. If the unabashedly spiritual numbers don't appeal, then the gorgeous homages to departed souls, among them Ornette Coleman, David S. Ware and Jeanne Lee, certainly will.

Eight. RED Trio/John Butcher - Summer Skyshift
(Clean Feed)

Portugal's avant jazz scene is among the strongest and most distinctive in Europe, perhaps due to the country's geographical isolation and relative lack of an improvised music tradition. RED Trio, featuring Rodrigo Pinheiro on piano, Hernani Faustino on bass, and Gabriel Ferrandini on drums are one of the country's finest units, revitalising the piano trio format. Taped at Lisbon's Jazz Em Agosto festival in 2015, Summer Skyshift sees them renew their vows with British master saxophonist John Butcher. From Pinheiro's right hand come luminous ripples of piano, as Faustino's bass evokes gathering stormclouds. Ferrandini is quite brilliant, circling an unfixed centre with fluttering polyrhythms and balletic cymbal moves. Butcher has great chemistry with this ensemble, and they accommodate his extended techniques beautifully, while also willing him to unleash some gutsy free jazz honks alongside the birdsong. Butcher-philes should also check out Tangles, his head-turning session with piano genius Matthew Shipp and analogue synthesiser guru Thomas Lehn on Fataka. Members of RED Trio, meanwhile, have appeared on a host of fine releases. Ferrandini dazzles on Rodrigo Amado Motion Trio's excellent Freedom & Desire (Not Two), while he and Faustino both appear on splendid new albums from No Business.

Seven. Tyshawn Sorey – The Inner Spectrum Of Variables
(Pi Recordings)

At only 35, drummer and composer Tyshawn Sorey straddles New York's avant jazz and contemporary classical scenes with aplomb. Largely composed, this extended work for double trio is opened up to improvisation through Sorey's Butch Morris-inspired conduction techniques. Spread over two discs, this monumental work takes in Bach-like contrapuntalism, resonant drones, jagged piano vamps, and catfights between violin and viola. There's even room for a lyrical coda of straight-ahead piano jazz balladry, before the Venusian abstractions of 'Reverie' and the diverse topographies of the final three movements. The Ethio-jazz section of 'Movement IV' is particularly inspired, successfully transposing modal melodies to violin and viola, while Sorey deftly expands on a syncopated groove. The Inner Spectrum Of Variables is a hugely impressive and often moving work, establishing Sorey as a worthy heir to creative composers like Anthony Braxton and Henry Threadgill.

Six. Shabaka & The Ancestors – Wisdom Of Elders
(Brownswood)

Recorded with a crack squad of South African musicians, Wisdom Of Elders is a glorious communion of spiritual jazz, Caribbean music and fiery Johannesburg jazz. British-Barbadian composer Shabaka Hutchings is a marvel throughout, his tenor a conduit for gorgeous melodies, hot rhythmic vamps and tightly coiled ecstasies. His interplay with drummer Tumi Mogorosi gives the album its particular energy, with the saxophonist deftly negotiating the tension between freedom and form. Trumpeter Mandla Mlangeni is another stand-out, cutting through Nduduzo Makhathini's impressionistic Rhodes colouring with sharp silver beams. Siyabanga Mthembu's stirring vocals are pitched somewhere between the pulpit and the opera house, bringing grit and class. As inventive and enjoyable as his albums with Sons of Kemet and the Comet Is Coming have been, Wisdom Of Elders is Hutchings' finest achievement to date. And don't forget to check out his South African bandmates on Amandla Freedom Ensemble's marvellous Bhekisizwe.

Five. Steve Lehman - Sélébéyone
(Pi Recordings)

With Sélébéyone, composer and saxophonist Steve Lehman has created an uncompromising fusion of avant jazz, underground hip hop, and spectral composition. The results are mind-boggling, with the MCs and instrumentalists negotiating Lehman and Maciek Lasserre's asymmetric beats and alien harmonies with remarkable skill and invention. The manic Wolof chatter of Senegalese rapper Gaston Bandimic is a brilliant match for HPrizm's gruff New York philosophising, and both MCs break new ground in terms of phrasing and flow. Lehman and Lasserre leap into the fray with spiralling saxophone riffs and bracing squalls, while drummer Damon Reid runs rings around the programmed beats. One of the year's most innovative and exciting releases.

Four. Charlie Haden Liberation Music Orchestra - Time/Life
(Impulse!)

Before his passing in 2014, Charlie Haden and his wife Ruth Cameron had planned a Liberation Music Orchestra album dedicated to environmental issues. While the project was never realised in his lifetime, two pieces on an ecological theme were recorded at a Belgian festival in 2011: Bill Evans and Miles Davis's 'Blue In Green' and Haden's own 'Song For The Whales'. Arranged by the incomparable Carla Bley, Haden's chief collaborator in all iterations of the Orchestra, these are deeply moving performances, full of grace and fire. Following Haden's death, Cameron invited Bley to complete the album with some of the latter's own compositions. The ensemble arrangement of 1968's 'Silent Spring' is eerie and foreboding, while the new piece, 'Time/Life' is a gorgeous requiem for Haden, in which each musician pays personal tribute with a solo. 'We're here for a reason, and that's to make sure this universe stays beautiful and wonderful and brilliant,' says Haden in his closing remarks, 'it's so important to remember how precious this life is.' Poignant as it is to hear these words, Time/Life is a beautiful testament to Haden's unerring sense of humanity and justice.

Three. Linda Sharrock/(In) The Abyssity of the Grounds – GODS
(Golden Lab)

Linda Sharrock is best known for her astounding vocal contributions to her late ex-husband Sonny Sharrock's free jazz masterpiece Black Woman and her very fine collaborations with Wolfgang Puschnig. GODS is her first major release following a debilitating stroke. Sharrock's vocal range may be limited, but she has found a deeply expressive new language in the growls, moans and wails she unleashes over (In) The Abyssity Of The Grounds' avant-jazz tumult. Theresa Eipeldauer is a superb vocal foil, inhabiting the higher registers Sharrock can no longer reach, while reedist Mario Rechtern, guitarist Max Bogner and drummer Didi Kern play with a divine fury, pushing Sharrock into realms that are both harrowing and ecstatic. This limited edition triple-vinyl release from Manchester's Golden Lab sold out within weeks: bring on the repress. And if Sharrock and her band are playing in your town, cancel all other plans: their Glasgow show back in February was fiercely transcendent.

Two. Henry Threadgill - New Locks and Old Verbs
(Pi Recordings)

Henry Threadgill deservedly won a Pulitzer for his 2015 album In For A Penny, In For A Pound and Old Locks & Irregular Verbs is equally stunning. A tribute to the late great Butch Morris, Old Locks sees Threadgill putting his saxophone and flute aside to focus on conducting his Ensemble Double Up through a structured improvisation based on his unique interval-based system. 'Part 1' opens with an expressive dialogue for David Virelles and Jason Moran's pianos. The instrumental doubling extends to the twin alto saxophones of Roman Filiu and Curtis MacDonald, who enter after three minutes with a statement of gleaming, angular melodicism. Cellist Christopher Hoffman and tubaist Jose Davila bring unusual colour and movement to the mid-to-low registers, while Craig Weinrib's tightly wound drums maintain an underlying sense of propulsion and swing on even the knottiest of passages. This is ingenious music, but it has great emotional resonance, not least in 'Part 4', where Threadgill unveils a deeply uplifting gospel theme.

One. Wadada Leo Smith – America's National Parks
(Cuneiform)

Choosing between this album and A Cosmic Rhythm With Each Stroke, Wadada Leo Smith's sublime set with pianist Vijay Iyer on ECM, was no easy task, but ultimately it's the great trumpeter/composer's paean to the American landscape which has it. The six-movement suite is scored for the Golden Quintet, in which cellist Ashley Walters joins pianist Anthony Davis, bassist John Lindberg and drummer Pheeroan akLaff. The combination of Smith's trumpet and Walters' cello is inspired, opening up new melodic and colouristic possibilities, while the rhythm section plays with masterful sensitivity and grace. There is a beautiful sense of space and light to this music, but the exquisitely laconic Smith always surprises, never resorting to banal evocations of the rivers, forests and mountains that inspire him. At 75, Smith is a true master, and America's National Parks is one of his most visionary works: an environment to explore and cherish.