Action guides evaluation

In 2007 the Legal Services Commission asked Advice Services Alliance Advicenow to produce two 'self-management' guides to be made available to CLS Direct callers who don't qualify for legal help, and the general public. By 'self-management' we mean guides that help people through the whole process of dealing with a problem including knowing when and how to get help. We gave this approach the more user-friendly name of 'action guides'

The guides include information on the issue; a guide to the actions needed to resolve the problem; and material providing the knowledge and skills to manage the problem effectively, and encourage the reader to seek further advice.

These resources aren't a substitute for good advice and representation, but they are intended to help those who may not be able to get that help.
After looking at research by the Legal Services Commission, and consulting with the advice sector we decided to produce guides on Disability Living Allowance/Attendance Allowance appeals and interviews under caution for suspected benefit fraud:

The process of producing these guides was quite challenging for a number of reasons:

Producing materials that start from the user's point of view and try to tell them what they need to know about the reality of a situation, rather than just how it should work, is difficult and time-consuming. We had to involve both users and the professionals that worked with them in the production process. We are extremely grateful that people were prepared to work with us in this way.
Sometimes there were disagreements about the way that the law worked or local differences in how agencies or systems operated.
There was controversy about whether we should be producing guides on these topics at all. This ranged from the view that a little knowledge could be a dangerous thing, and that users should be properly represented, to concerns that we were producing guides to help people cheat the system.

Evaluation
The guides were independently evaluated following publication. The research set out to evaluate the effectiveness of the action guides by talking to people who used the guides, advice workers and other intermediaries.

The results were generally positive. It was found that the approach of focusing on specific audiences and identifying specific issues that are amenable to self-management in some or all of their aspects can be used for any number of issues. The delivery techniques deployed in these guides: both for readability/appeal and so usability, and their useful and accessible content, should be readily adaptable for other future guides.

However, there were strong and sometimes opposing views on the guides and a number of specific comments were made on the content.

You can read the evaluation report here: An evaluation of Advicenow self-management guides (172 KB)
For ASA Advicenow's response to the report click here: ASA Advicenow response to self-management guides evaluation (110 KB)

Policy
During the process of developing the guides, and then evaluating them, a number of issues arose which pointed to wider issues which could not be resolved by the guides. These issues of policy are drawn together by our evaluator in this document:
Policy issues arrising from the development and evaluation of Advicenow self-management guides (70 KB)

November 2008

« BACK: Reviewing and scoring information resources

NEXT: Advicenow Email Update »

Comments

I agree about the need for better adviser (and also arguably user) understanding about what materials aim to do, and how they might be used. It seems all too easy for people to dismiss materials by saying 'but my clients who are worst off/least capable won't understand them'. Give those clients some advice then. But don't assume that nobody can use them. Nonetheless, I think the comments about users with literacy and language issues indicate a need to consider the minimum levels of literacy required to use them (no matter how much care you take over the language, if people can't read or don't have basic comprehension skills, they're never going to get it without assistance).

As I read the report, I was quite struck by some of the differences in perceptions between users and advisers. I think this raises two questions:
- firstly, is it possible to produce a single guide which speaks equally well to users, family/friends, intermediaries, and generalist advisers?
- secondly, when evaluating information materials, is it safe to rely on the assumptions of advisers etc. as to what is a good resource for users?
My own answer to each of these would be 'no'. I think that the starting point ought to be a tailored approach on both counts. From this and other work it seems pretty clear that people need information which speaks directly to them, and I think the only way to find out what that really means is to ask them. Costs are obviously an issue here. But globally, there must be an enormous amount of time and money going into information provision.

Perhaps a case could be made for just a fraction of that to be diverted towards establishing an evidence base as to what constitutes good - and cost effective, materials?
One last point. I don't have a particular view on whether to use cartoons. But as indicated by some of the comments, there's a need to think about the language of visuals just as carefully as that of text. For my own part, I wonder about the use of Shakespeare (for I presume it to be he) (p12 of the DLA/AA guide). The intention seemed to be to illustrate somebody writing a statement - and who better one might think than 'our' most illustrious writer? But a substantial proportion of people who aren't the English middle classes don't know (or care) who Shakespeare was, and I suspect many more wouldn't have a clue what he was supposed to have looked like, so would have just seen a bald bloke from another time. To me, the connotation was that writing your statement may be a long and laborious process, and something you probably have to write in Olde English/legalese. But that's only my interpretation. A user may think differently!

A researcher, Wednesday 10 Dec 2008