Documentation often suffers because clients and developers alike fail to recognise its importance, instead prioritising development activities to demonstrate progress and value. Documentation supports the entire lifecycle of software development: from helping individual developers recall why they built something in a certain way, to enabling continuity between developers, to facilitating user adoption. Thus, insufficient documentation can undermine the success of an entire project.
The Agile Manifesto (Beck et al., 2001) came out in response to an excess of poor bureaucratic documentation promoting an approach based on direct collaboration. Eventually, this was translated into the production of “hundreds of written ideas, sketches, visualizations, scenarios, and storyboards from which to generate requirements” (Maiden and Jones, 2010). The process has a lot of advantages for the software development cycle, but the resulting documentation is normally very disorganized and disconnected. I lived the progressive adoption of agile methodologies and, during my career, I went from having long boring documents covering everything from A to Z in the early 2000s to the current age of kanban boards full of unsorted tickets that rarely describe a feature with more than a sentence. I agree with the manifesto that the main goal is to produce working software, and indeed agile is an improvement compared to waterfall. However, it is time to admit that the iterations beyond the first six to twelve months become harder and harder without comprehensive documentation.
References
Beck K. et al. (2001) The Agile Manifesto. Available from https://agilemanifesto.org/iso/en/manifesto.html
Maiden, N., & Jones, S. (2010) Agile requirements can we have our cake and eat it too?. IEEE software, 27(3), 87-88.