In writing, we have many formulas and rules. Yes, rules are meant to be broken, but only the ones who know them can break them well. However, we can't break too many at once.
I am by no means an expert in experimental fiction. I gravitate to classical and modern literature. Even in these circles, writers buck the rules.
This assumes we know the rules and accept the rules, and maybe this is the topic for another post.
One example is Anthony Burgess's Clockwork Orange. This novel breaks several rules, using creative dialect (a linguistic experiment in Russian vocabulary), a dislikable protagonist, and a difficult Christian theme. Despite the rules that this novel breaks and the controversy surrounding the publication of the first edition and release of the film, the novel generally follows most of the other guidelines for novels: conflict, plot, rising action, denouement, characterization, and setting (although we could discuss these elements at length).
Point is, the book is a balance of experiment and rules. Burgess didn't throw out the entire rule book for fiction in writing this, but followed some expected patterns.
One more example that I have fallen back on in recent months: Bright Lights, Big City. I have found this novel strangely fascinating as I have reflected on this. Certainly, the 2nd person POV is experimental in its persistence, but the rest of the novel is rather expected. A man is doing too many drugs in 1980s New York City after his model/wife has left him. He loses his job and does many other foolish things. Really, the plot and protagonist is unremarkable. Even the denouement is rather expected for a post-modern novel. But the writing and POV are quite remarkable.
Genres are even trickier. I've already deleted this section since I'm geeking out too much. Maybe this is another discussion.
In the end, I believe, we need to know the rules and expectations. We can only break a few (maybe only one or two?) at a time, and do so carefully, balancing the expected with the new.