In general, when creating any product or solution, it is crucial to test its usefulness. The same principle applies to building digital solutions for community users, especially in the context of development where users' demographics can be constrained by accessibility and literacy. Luckily, the Principles for Digital Development can serve as a guiding checklist for digital development implementers when designing solutions for the community. If this is the first time you're hearing about the Principles for Digital Development, we strongly suggest you take some time to familiarize yourself with them.
At Kawsang, one of the digital development principles that we find particularly important, based on our work with the community, is "Design with the user." We engage with the project implementing partners and validate the design of digital solutions with end users in the field. A decade ago, we had to explain to our partners/funders why solution design was not final and why field validation was necessary and important. Nowadays, development partners have a greater awareness of this importance and support such field validation activities.
Let us share with you our recent field testing of the upgraded version of our career guidance and counseling application, "Trey Visay."
After spending several weeks working with the Vocational Orientation Department (VOD) of the Ministry of Education, Youth, and Sport in Cambodia, our team created a prototype of the enhanced version that aligns with VOD's vision. Under the guidance and facilitation of VOD, we selected two schools in Battambang for field testing the prototype.
Our team (Sokly and Kimsan) traveled by bus from Phnom Penh to Battambang to conduct the field testing of the new enhancement of the Trey Visay application design at Net Yang High School and Anlong Vil High School in Battambang Province. A total of 48 students from grades 10 to 12 participated in our test.
We conducted the testing with small groups of five students at a time, following these steps:
We provided orientation to the students about the purpose of the application.
We installed the prototype app on their mobile phones.
We observed quietly how the students interacted with the application.
We asked the students to fill out a survey about their understanding and usefulness of the application.
We conducted one-on-one interviews with each student after they completed the survey.
Key insights from this field test include:
70% of students proactively provided comments and feedback.
85% of students were able to complete the test scenarios.
Installing the prototype on the students' phones worked well, as they were familiar with their devices.
We observed that students showed a preference for video content over reading text.
Students provided valuable feedback on the wording and terminology, suggesting the use of simpler terms they are familiar with.
In the paper survey, students gave higher scores overall to the application and provided fewer suggestions for enhancement. However, during the one-on-one interviews, they provided more critical feedback. This suggests that students are less critical in providing feedback on paper but more open during oral interviews. This is a valuable lesson for future field validation.
Sokly and Kimsan, the software developers from our team who conducted this field testing, share their insights in a Q&A session:
Q: If we could redo the field test, what would you suggest doing differently?
Sokly: If we could redo the test, I would update the survey questions by adding detailed keyword triggers in the questionnaire. Without those keywords, students seemed to have no idea. Compared to our field test in Kampong Cham province a few years ago, students there had received a career counseling session, so they were more familiar with the application process.
Kimsan: In the future, we should continue to combine the survey filling and one-on-one interviews. Although the survey responses were positive, through the interviews, we can delve deeper and obtain clearer answers, which may differ from what participants initially indicated. Essentially, the survey alone does not provide a qualitative response.
Q: Do you think it is worthwhile to have developers be part of this user testing?
Sokly: As a developer, it is beneficial to meet the end-users and observe their interaction with the prototype app. Sometimes, designs that I thought were intuitive turn out differently in practice. It is essential for us to witness their use and receive their feedback. I want to continue conducting field tests on the application because it allows me to learn from firsthand experience. It encourages me to think about the user's perspective during app development. Understanding what features users find useful or not helps me make informed decisions and provide input to my team.
Kimsan: This is my second time conducting a field test with end users. Without observing them using the app in real-life situations, I wouldn't truly know how intuitive it is. Additionally, I can learn about their behaviors and needs. I would like to conduct further field tests in the future as it enables me to better understand and address the real needs of users.