Two years ago this week, I moved to Peckham from Norfolk. I didn't really choose Peckham – the location came about more or less accidentally, and I had no idea that the area was undergoing gentrification. Indeed, with the exception of one or two streets in the vicinity of Peckham Rye, the large common where William Blake saw angels and which inspired Muriel Spark to write a subtly brilliant novel, there wasn't that much evidence that this was taking place.

More recently, however, and particularly since the opening of an Overground spur connecting Peckham to Hackney, there's been a tangible shift in the cultural atmosphere of SE15. It seems that not a week goes by without an empty shop being refitted as a Jay Rayner-pleasing restaurant catering for the deep-of-pocket; meanwhile, the fashionably wasted pour in for club nights at the Bussey Building and Canavans. This shift provides broad sociological context for non-profit arts organisation Bold Tendencies' staging of the recent London Contemporary Music Festival in the multi-storey car park off Rye Lane.

For the last few summers, BT have been running arts events in the space, and hope to eventually convert it into a 'creative hub' open all the year round. Their hosting of the LCMF – also managed by an NFP arts commissioning agency, Sound Four – suggested a push towards making this ambition a reality, and performers and composers of the calibre of Glenn Branca and Tony Conrad duly arrived to provide eight nights of avant-garde music against the backdrop of the city spreading away to the north. What I saw of the concerts was impressive, especially given that tickets were free: a fired-up Branca leading the UK premiere of new work Twisting in Space during a thunderstorm, electronic drones harmonising with passing trains on the first Sunday afternoon, and an exceptional collaboration between Conrad and Jennifer Walshe the same evening.

Nevertheless, various controversies have emerged around the LCMF. Some critics expressed concern that the venue was acoustically unsuited to the kinds of music – including many durational pieces working with minimal volumes – programmed. Others have mumbled that not all of the pieces performed still seem particularly radical in 2013. However, the most provocative debate was initiated by Ben Beaumont-Thomas in the Guardian. This review argued that, while there much to applaud about the festival, the choice of Philip Corner's Piano Activities, in which the titular instrument is literally destroyed, pointed to a wider lack of political conscience on the part of the organisers. Beaumont-Thomas contended that "Smashing and violating this instrument that gave so much over its life, at an (admittedly free) event high above one of the most deprived areas in London, is indulgence bordering on immorality". The implication here seems to be that certain privileges need to be checked around the matter of aesthetic extremism (if one can really call a fifty-year-old piece 'extremist') being staged in a community as marked by socio-economic inequalities as Peckham.

This leads, inevitably, to certain questions about the role of experimental art in the wider scheme of gentrification. King's College-based sociologist Tim Butler, who specialises in urban geography and regeneration projects in inner London, has proposed optimistically that the flourishing of the arts in deprived areas need not be seen (or need not be seen solely) in terms of broader economic patterns linked to housing prices which lead ultimately to the displacement of communities by moneyed newcomers. For Butler, art projects in areas like Brixton and London Fields reflect a middle-class attempt to end, rather than further, social segregation: this is, he suggests, perhaps a 'logical outcome of the good faith perhaps implied by the ascetic liberal consciousness.'

In Britain 2013, however, it can often be hard to sustain belief in either the 'ascetic liberal consciousness' or the notion that it implies 'good faith'. Many apparent gestures of ethical conscience make claims to social legitimacy which are at best spurious, while the Big Society, which is styled as a democratising force, operates to consolidate wealth for certain groups while excusing reductions of funding for services essential to the economically vulnerable. In the context of Peckham, Bold Tendencies and the LCMF, alarm bells start to ring when one notes that the Evening Standard report paraphrases Hannah Barry, Bold Tendencies' founder, as saying that the project 'is an embodiment of the Big Society'. Indeed, all the memes of Cameron-era fake DIY are there – cheerful job-creation estimates, corporate sponsorship and bonhomous affirmations of the capacity of 'gigs' and 'poetry readings' to instantiate social change.

Bearing this scepticism in mind, it's also important to think about how the critique of art as a component of gentrification might itself have some reactionary underpinnings. Such a position is potentially haunted by the assumption that the people of (say) Peckham are inherently uninterested in the avant-garde. Now, I'm unwilling to make determinate claims about how many 'genuine' locals were present at the LCMF performances and, certainly when I attended, the audience was predominantly white and blinged up with all the accoutrements of middle-class cultural capital. This is not to say, however, that to be working-class is to be uninterested in difficult art of the kind on display in the car park. Moreover, the implication that this is the case – that the simple Peckhamites just want that knackered old piano for school assemblies – carries with it the impression of cultural capital being accumulated to, and consolidated by, Butler's 'liberal ascetic consciousness'.

This is a longstanding problem in Britain. The poetry we learn in school tends towards the banal and paraphrasable (think Larkin or Duffy) because (liberal) middle-class curriculum setters have continually returned to a notion of 'relevance' which resists the idea that kids from council estates might find that T.S. Eliot or Ezra Pound speak to them. The experimental poet and critic Andrew Duncan covers this in some detail in his polemical The Failure of Conservatism in Modern British Poetry, coming to the conclusion that it ultimately represents an attempt on the part of the middle class to ring-fence an area of culture – the 'difficult' bit – for themselves. Some find this argument paranoid, but it's telling that our broadsheets still tend to regard 'authentic' working-class art through the paradigm of 1950s social realism and Not Letting The Bastards Grind You Down.

Against this foible of the British cultural imagination, it's instructive to consider just how much of the UK's experimental art since World War Two has been made by practitioners from backgrounds that can in no way be considered privileged. Indeed, Derek Bailey, one of the foremost members of the free improvisation scene that came to prominence in the late 1960s, was the son of a Sheffield barber and found his way into radical music by playing guitar in nightclub bands. Likewise, B.S. Johnson, arguably postwar Britain's most uncompromising experimental novelist, was a working-class evacuee who left school at sixteen. As for poetry, the list of avant-garde writers who grew up in at least relatively poor environments includes Tom Pickard, Tom Raworth, Peter Riley and Brian Catling. The supposition that it's only the wealthy and so-called 'educated' who have a stake in the leftfield is a dangerous one to make.

The debate around gentrification, then, can lack nuance on both sides. Advocates of regeneration tend to be too hasty in their claims that 'cultural hubs' and arts festivals can, as the saying goes, 'breathe life' into deprived areas. Against this, the belief that avant-garde work is a purely middle-class concern brings with it its own set of assumptions, and these also need to be challenged. When all is said and done, a better way to help inner-city communities maintain their identities would be a tighter regulation of the property market – rent controls can help a working-class family keep close to their roots far better than a blanket resistance to incursions by experimental artists. In this case, the finger needs to be pointed at policy makers and councils rather than at straw men.