Image taken from http://metropolitanorganizing.com
Developing an Evaluation Rubric for Selection of Mobile Apps
Created by Sonny Dhaliwal, Jean Kloppenburg and Justin Mark
OLTD 508
March 2014
What is the overall driving force when selecting an app?
What should be considered in the “big picture” regardless of the type of device or operating system?
In discussing and sharing our thoughts around this question, we realized that we all felt the same way but chose to take a different approach. Instead of compiling one large intro and possibly losing the voice of the individual, we have decided to display all three.
Sonny Dhaliwal
As educators our first priority, aka the “big picture” is meeting the needs of our students. We have learning outcomes and student needs and someway, somehow we build connections, as long as these connections are made does it really matter what process we take, app or not. Too often technology is looked to as the solution, it is only a tool. Before introducing new technology to our students whether it is a device, software or an app we need to ensure that it is based on good pedagogy and have a clear idea of how it will be used. If we do not go through this process we can end up using technology for the sake of technology. The cool factor always helps, but should not be the driving force. What we should be looking at is how effectively it can deliver the intended learning outcome. There are many parts necessary in order for an app to be effective, including but not limited to: engagement, feedback quality, curriculum correlation, skill level, cost. If the app passes through all of these filters then it can be used as an option for students to use. Remember to provide as many options as possible, an app is not necessarily everyone’s preferred medium.
Jean Kloppenburg
Common criteria for app use mostly concentrate on the technical aspects of reliability, stability, fast loads, consistency and absence of advertisements. Walker (2011) states: “When evaluating apps for educational use, technical criteria are only the bare minimum; practitioners need to take a more focused look at the educational benefits the app offers”. In Langwitches Blog, a reader asked: “I want to use iPads in my Science class. What app is good for that?” The reply given sums up what should be the driving force when selecting an app:
“I am not comfortable with the level of disconnect between the teacher (who knows her/his students best) and the curriculum related skills and objectives and pedagogical relationship that needs to be in place for an app to be a match to use in a classroom or with an individual learner.” (Langwitches, 2012)
As Langwitches Blog(2011) points out, teachers need to be equipped with the curiosity and knowledge of:
· the value an app can bring to a learner (and being able to articulate the value)
· the connection from the app to curriculum content (and being able to demonstrate the depth of that connection)
· the possibilities the app can bring in order to amplify (exposing work to new literacies)
· the difference of using an app to automate and substitute a task versus inform and transform
· how to evaluate apps for their transformative potential.
The driving force behind choosing an app needs to be, first and foremost, related to the learning objectives. If my students need to understand and demonstrate their understanding of the exposure triangle in photography, they can create a narrated video using Videolicious or analyze the relationship with the use of Popplet. The apps are available for students to enhance their learning and hopefully create a transformation and amplification to their learning along the way.
Justin Mark
My take on the overall driving force of app selection is that the app must meet the learning outcome that you are hoping to achieve. When I was selecting apps for an iPod project that I was working on 3 years ago, I started with the prescribed learning outcomes and I selected apps that supported learning outcomes. The problem I encountered was that most apps focused on ROTE memory task to memorize vocabulary, while I put some value on this, it only represents a small aspect of the curriculum. In addition at that time we were struggling with Wifi and cost was another limiting factor. I had a group of students help me evaluate the apps and help design lessons for each app. Some of the most enduring apps were the tools like a Collins Dictionary app and a verb conjugating app that we found for free. As I’ve moved more toward an inquiry model in my practice I find I am much less prescriptive with apps and I often have students choose apps that help them achieve the learning goals that they are trying to achieve. It must be noted that this approach may change based on grade level as I tend to give my senior students more freedom than my junior ones. In an 8-12 program, I spend more time with the junior students teaching them how to use certain tools, and apps. In the readings one of the problems I was encountering was that there seemed to be a higher value placed on top of the Blooms Taxonomy pyramid, and a devaluation of memorization. In the Cantwell presentation, “Evaluation of Apps”, the question is posed, “is the app built on quality pedagogical principles?” (Cantwell) I would argue that this depends on your curriculum needs? In some cases there may be a need for lower skills like memorization on Blooms Taxonomy for example memorization of times tables, which I still believe is considered an essential skill even in 21st century learning models. I think this is a fundamental problem with many of the app evaluating rubrics, as they work on the principle that a single app should meet multiple higher learning functions, when maybe it would be more valuable to evaluate a group of apps that are each specialized in focusing on specific learning outcomes. For instance in the Jeanette Van Houten article, (2011, January 01). iEvaluate Apps for Special Needs. I disagreed with her Rubric, specifically the criteria - “Under types of Skills Practiced” it characterized Flashcard drill as “Weak” I think this type of assessment is missing the point, while many like to value problem based and simulation based learning over memorization some apps might specifically target memorization as a skill, for example a math game that focused on memorization of times tables. I think the point is that, it is very difficult for one app to fit all learning tasks and that’s the point. Teachers should seek to employ a variety of apps with different pedagogical purposes in order to meet the learning requirements of the curriculum and their students. Regardless any reflection of an App which evaluates its educational value is useful, but at the same time we may want to specifically evaluate each app based on the specific learning outcome we hope for it to achieve.
Rubric Categories
Curriculum Focus
According to Harry Walker (2011),” Defining what makes an app “good” varies depending on the audience.” Curriculum Focus is at the top of our importance list in our evaluation rubric because we are evaluating apps for educational use. The use of an app should support our student audience with strong connections to the skills and concepts being addressed. Walker points out that there are many quality apps for math and literacy, but locating higher thinking level apps for science and social studies are harder to find (2011). Putting an emphasis on the curriculum connection will enable rubric users to rate the app first based on the strength of the skills reinforced that match the intended concept.
Age and Grade Level
Many apps have age and grade levels attached to them and others have none at all. It is important to be aware of not only the age appropriateness of the app but also the clarity of the directions for use. An app may be too easy or too difficult for the intended age group. If instructions are confusing or unclear it can lead to frustration and disengagement from the learners. As an educator, I first choose an app that fits my learning outcomes and concepts then my next decision is making sure it is the proper age and grade level.
User Friendly
While this category may be somewhat subjective to the user and may depend on grade level we feel it is an important category to consider when evaluating apps. If apps are prone to crashing or instability this may be a limiting factor in employing them with students. While all apps may require some level of instruction to get started, some are much easier to use than others. In addition some apps are much more sophisticated and may have more functions and features to learn than others. Some apps may include built in tutorials, and help functions or may provide an intro tutorial to get students acclimated to the program. In conclusion while this can be a subjective rating, some programs are notoriously difficult for the user while others are presented in a way that makes it seem almost seamless for the user, and this needs to be taken into account when considering apps for students.
Cost
In a perfect world where school districts possessed unlimited budgets and resources, perhaps this category would be unnecessary. While this category doesn’t address the learning outcomes we felt it was still necessary to include as it represents a pragmatic reality when choosing educational apps. We have organized our cost rubric from free to $5.99 and up on the extreme. While free is usually preferable given the economic realities of department budgets in most schools in BC, sometimes free means limited functions or embedded and annoying advertisement.
Student Engagement
We choose to incorporate Student Engagement in our rubric because we felt that the likelihood of an activity to be successful would greatly diminish without it. There could be an app that perfectly aligns with the intended learning outcomes but if there are frustrations and distractions, these would be obstacles in the ability of a student comprehending the learning outcomes. Den Delimarsky does an excellent job of highlighting some of these which include Stability, Load times, Hangups, Ads and functionality. No matter what type of activity we are attempting we want students to “buy in”, be motivated which will help with taking ownership and all of this is intertwined with engagement.
Thinking Skills
We choose to incorporate Thinking Skills in our rubric because pedagogy should be driving all of our classroom activities. The time to complete an activity, the learning outcomes, HOTS skill level all need to be in alignment. The app needs to be able to meet or exceed the blooms taxonomy level needed or else the concept is only being superficially covered. Students need to be able to take the knowledge and understanding and apply it in critical and creative ways.
Differentiation
Differentiation is at the bottom of our list and I’m not sure it applies is necessary requirement for all apps, however it is still worth considering. In her blog post 7 Essential criteria for evaluating mobile educational applications. Mayra Villar, when considering differentiation or as she titles it personalization, writes:
The possibility of adjusting content and settings to meet specific needs
of the learner does not only guarantee engagement throughout the learning
process but also contributes to the acquisition of new knowledge and shows
the learner how to apply it to real-life situations.
Differentiation as stated above is a key component of meeting the specific needs of the learner and allowing the learner to not outgrow the app as quickly. This should be considered when evaluated an app and may apply to using the app for multi grade levels.
Created by Sonny Dhaliwal, Jean Kloppenburg and Justin Mark
OLTD 508
March 2014
What is the overall driving force when selecting an app?
What should be considered in the “big picture” regardless of the type of device or operating system?
In discussing and sharing our thoughts around this question, we realized that we all felt the same way but chose to take a different approach. Instead of compiling one large intro and possibly losing the voice of the individual, we have decided to display all three.
Sonny Dhaliwal
As educators our first priority, aka the “big picture” is meeting the needs of our students. We have learning outcomes and student needs and someway, somehow we build connections, as long as these connections are made does it really matter what process we take, app or not. Too often technology is looked to as the solution, it is only a tool. Before introducing new technology to our students whether it is a device, software or an app we need to ensure that it is based on good pedagogy and have a clear idea of how it will be used. If we do not go through this process we can end up using technology for the sake of technology. The cool factor always helps, but should not be the driving force. What we should be looking at is how effectively it can deliver the intended learning outcome. There are many parts necessary in order for an app to be effective, including but not limited to: engagement, feedback quality, curriculum correlation, skill level, cost. If the app passes through all of these filters then it can be used as an option for students to use. Remember to provide as many options as possible, an app is not necessarily everyone’s preferred medium.
Jean Kloppenburg
Common criteria for app use mostly concentrate on the technical aspects of reliability, stability, fast loads, consistency and absence of advertisements. Walker (2011) states: “When evaluating apps for educational use, technical criteria are only the bare minimum; practitioners need to take a more focused look at the educational benefits the app offers”. In Langwitches Blog, a reader asked: “I want to use iPads in my Science class. What app is good for that?” The reply given sums up what should be the driving force when selecting an app:
“I am not comfortable with the level of disconnect between the teacher (who knows her/his students best) and the curriculum related skills and objectives and pedagogical relationship that needs to be in place for an app to be a match to use in a classroom or with an individual learner.” (Langwitches, 2012)
As Langwitches Blog(2011) points out, teachers need to be equipped with the curiosity and knowledge of:
· the value an app can bring to a learner (and being able to articulate the value)
· the connection from the app to curriculum content (and being able to demonstrate the depth of that connection)
· the possibilities the app can bring in order to amplify (exposing work to new literacies)
· the difference of using an app to automate and substitute a task versus inform and transform
· how to evaluate apps for their transformative potential.
The driving force behind choosing an app needs to be, first and foremost, related to the learning objectives. If my students need to understand and demonstrate their understanding of the exposure triangle in photography, they can create a narrated video using Videolicious or analyze the relationship with the use of Popplet. The apps are available for students to enhance their learning and hopefully create a transformation and amplification to their learning along the way.
Justin Mark
My take on the overall driving force of app selection is that the app must meet the learning outcome that you are hoping to achieve. When I was selecting apps for an iPod project that I was working on 3 years ago, I started with the prescribed learning outcomes and I selected apps that supported learning outcomes. The problem I encountered was that most apps focused on ROTE memory task to memorize vocabulary, while I put some value on this, it only represents a small aspect of the curriculum. In addition at that time we were struggling with Wifi and cost was another limiting factor. I had a group of students help me evaluate the apps and help design lessons for each app. Some of the most enduring apps were the tools like a Collins Dictionary app and a verb conjugating app that we found for free. As I’ve moved more toward an inquiry model in my practice I find I am much less prescriptive with apps and I often have students choose apps that help them achieve the learning goals that they are trying to achieve. It must be noted that this approach may change based on grade level as I tend to give my senior students more freedom than my junior ones. In an 8-12 program, I spend more time with the junior students teaching them how to use certain tools, and apps. In the readings one of the problems I was encountering was that there seemed to be a higher value placed on top of the Blooms Taxonomy pyramid, and a devaluation of memorization. In the Cantwell presentation, “Evaluation of Apps”, the question is posed, “is the app built on quality pedagogical principles?” (Cantwell) I would argue that this depends on your curriculum needs? In some cases there may be a need for lower skills like memorization on Blooms Taxonomy for example memorization of times tables, which I still believe is considered an essential skill even in 21st century learning models. I think this is a fundamental problem with many of the app evaluating rubrics, as they work on the principle that a single app should meet multiple higher learning functions, when maybe it would be more valuable to evaluate a group of apps that are each specialized in focusing on specific learning outcomes. For instance in the Jeanette Van Houten article, (2011, January 01). iEvaluate Apps for Special Needs. I disagreed with her Rubric, specifically the criteria - “Under types of Skills Practiced” it characterized Flashcard drill as “Weak” I think this type of assessment is missing the point, while many like to value problem based and simulation based learning over memorization some apps might specifically target memorization as a skill, for example a math game that focused on memorization of times tables. I think the point is that, it is very difficult for one app to fit all learning tasks and that’s the point. Teachers should seek to employ a variety of apps with different pedagogical purposes in order to meet the learning requirements of the curriculum and their students. Regardless any reflection of an App which evaluates its educational value is useful, but at the same time we may want to specifically evaluate each app based on the specific learning outcome we hope for it to achieve.
Rubric Categories
Curriculum Focus
According to Harry Walker (2011),” Defining what makes an app “good” varies depending on the audience.” Curriculum Focus is at the top of our importance list in our evaluation rubric because we are evaluating apps for educational use. The use of an app should support our student audience with strong connections to the skills and concepts being addressed. Walker points out that there are many quality apps for math and literacy, but locating higher thinking level apps for science and social studies are harder to find (2011). Putting an emphasis on the curriculum connection will enable rubric users to rate the app first based on the strength of the skills reinforced that match the intended concept.
Age and Grade Level
Many apps have age and grade levels attached to them and others have none at all. It is important to be aware of not only the age appropriateness of the app but also the clarity of the directions for use. An app may be too easy or too difficult for the intended age group. If instructions are confusing or unclear it can lead to frustration and disengagement from the learners. As an educator, I first choose an app that fits my learning outcomes and concepts then my next decision is making sure it is the proper age and grade level.
User Friendly
While this category may be somewhat subjective to the user and may depend on grade level we feel it is an important category to consider when evaluating apps. If apps are prone to crashing or instability this may be a limiting factor in employing them with students. While all apps may require some level of instruction to get started, some are much easier to use than others. In addition some apps are much more sophisticated and may have more functions and features to learn than others. Some apps may include built in tutorials, and help functions or may provide an intro tutorial to get students acclimated to the program. In conclusion while this can be a subjective rating, some programs are notoriously difficult for the user while others are presented in a way that makes it seem almost seamless for the user, and this needs to be taken into account when considering apps for students.
Cost
In a perfect world where school districts possessed unlimited budgets and resources, perhaps this category would be unnecessary. While this category doesn’t address the learning outcomes we felt it was still necessary to include as it represents a pragmatic reality when choosing educational apps. We have organized our cost rubric from free to $5.99 and up on the extreme. While free is usually preferable given the economic realities of department budgets in most schools in BC, sometimes free means limited functions or embedded and annoying advertisement.
Student Engagement
We choose to incorporate Student Engagement in our rubric because we felt that the likelihood of an activity to be successful would greatly diminish without it. There could be an app that perfectly aligns with the intended learning outcomes but if there are frustrations and distractions, these would be obstacles in the ability of a student comprehending the learning outcomes. Den Delimarsky does an excellent job of highlighting some of these which include Stability, Load times, Hangups, Ads and functionality. No matter what type of activity we are attempting we want students to “buy in”, be motivated which will help with taking ownership and all of this is intertwined with engagement.
Thinking Skills
We choose to incorporate Thinking Skills in our rubric because pedagogy should be driving all of our classroom activities. The time to complete an activity, the learning outcomes, HOTS skill level all need to be in alignment. The app needs to be able to meet or exceed the blooms taxonomy level needed or else the concept is only being superficially covered. Students need to be able to take the knowledge and understanding and apply it in critical and creative ways.
Differentiation
Differentiation is at the bottom of our list and I’m not sure it applies is necessary requirement for all apps, however it is still worth considering. In her blog post 7 Essential criteria for evaluating mobile educational applications. Mayra Villar, when considering differentiation or as she titles it personalization, writes:
The possibility of adjusting content and settings to meet specific needs
of the learner does not only guarantee engagement throughout the learning
process but also contributes to the acquisition of new knowledge and shows
the learner how to apply it to real-life situations.
Differentiation as stated above is a key component of meeting the specific needs of the learner and allowing the learner to not outgrow the app as quickly. This should be considered when evaluated an app and may apply to using the app for multi grade levels.
App Evaluation Rubric
PDF File with selected evaluated apps included
project2oltd508-1.pdf |
References:
Cantwell, K. (n.d.). Evaluation of apps. https://docs.google.com/presentation/d/1arPb0STMDosbKNX-gncwKJkBFqFfhoghcqZeJryPzfs/edit#slide=id.g15a3d4
92_0_19
Delimarsky, D. (2011, January 12). What makes an app a good app – 7 pointers. Retrieved March 13, 2014 from http://dotnet.dzone.com/articles/what-makes-app-good-app-10
Langwitches Blog. ( 2012, May 27) Evaluating Apps with Transformative Use of the iPad in Mind. Retrieved from http://langwitches.org/blog/2012/05/27/evaluating-apps-with-transformative-use-of-the-ipad-in-mind/
Pronovost, R. (2012, February 16). Technology for Technology’s Sake. Education Week Teacher. Retrieved March 13, 2014 from http://blogs.edweek.org/teachers/teaching_ahead/2012/02/technology_for_technologys_sake.html
Reed, J. (n.d.). What makes a good educational ipad app. Retrieved March 13, 2014 from http://www.drjonathanreed.co.uk/wordpress/2011/05/what-makes-a-good-educational-ipad-app/
Villar, M. A. (2012, December 06). 7 Essential criteria for evaluating mobile educational applications.
Schrock,K .(2013). Critical Evaluation of Mobile Apps. Retrieved from http://www.ipads4teaching.net/critical-eval-of-apps.html
Vincent, T. (2012, March 4) Ways to Evaluate Educational Apps. Retrieved March 13, 2014 from http://learninginhand.com/blog/ways-to-evaluate-educational-apps.html
Walker, H. (2011). Evaluating the Effectiveness of Apps for Mobile Devices. Retrieved from http://gpsk12.org/PD/iPads/HarryWalkerEvaluating.pdf