Lessons on learning analytics from a taxi driver

I've been on a surprise strip. You book a trip for three day but without having a clear where you are going to. Very exciting! We had to be at Schiphol by 5 am and although we were still sleepy: the taxi driver was not. He talked a lot, probably to stay awake .... He was very enthusiastic about a new system with data in his taxi. His story reminded me occasionally of an episode of Black Mirror:  Nosedive with a society based upon likes. Everyone gives each other points/likes and you need to have a certain number of points to be able to live somewhere in a certain appartement. It does not work out well for the hero of the story.

The wonderful taxi system

The fantastic system of our taxi driver consists of a point system (it is called cicada) adopted by his taxi company. Every driver earns points with his or her driving style. At first, he first did not care about his score, until it turned out that a female colleague scored highest. Knowing this, stimulated him to want to improve his score. Our driver was already at 98. For a long time his scores remained at 96 and he thought you could never get 100, but ... by keeping his hands on the wheel he was able to increase his score. I asked him what you have to do as a taxi drive to score this high. It comes down to riding evenly, not accelerate or brake suddenly, but also keep your hands on the wheel for example. He knew exactly how many seconds he could be on the radio before his score went down. It became interesting when a training place was released to become an executive driver. Our (black) driver was not nominated. Then someone thought about the scores, and wondered whether the scores should be an indication. Since our driver had the highest score he earned the place in the training and became an executive taxi driver.

From taxi to learning analytics

Clive Shepherd discussed the new skillset that L&D-ers need to develop. One of the new skills is interacting with media, and that includes the use of data. L&D also has to deal with new data which are available because of working with online media. Just think of the data you collect in your LMS or during a webinar. Although I was on vacation, I could not help myself to think about the parallels between the taxi system and learning analytics. With learning analytics you use data to gain insight into how people learn and how to support learning. The taxi data are aimed at a better driving style. The positive thing about the points system is that our driver was not nominated for the training, but because of his points he could prove that he was the best. Could you also provide a more objective assessment using learning analytics? For example, looking for the real experts? What also worked really well is the gamification element. He wanted to get a high score and therefore changed his driving style. Tip is therefore to make learning analytics understandable for learners themselves.

People always tweak the system

One more thing: you have to pay attention to what you measure with your system / learning analytics. Know what you are measuring. Combine it with observations. In the black mirror episode the system has very bad effects because people are going to judge each other. So be aware of the effect on people. Who wins, who looses?

Your data are always indicators, such as the number of minutes hands off the wheel. It was clear that people are always trying to outdo the system. For example, a taxi driver can turn on the radio 5 times in succession for 30 seconds and in between put his hands on the wheel to avoid a lower score. The question is whether that is safer than changing the radio at once. So know what you are actually measuring and ways users may tweak the system. You should never just look at the figures and forget to use your common sense.

By the way our surprise trip took us to Rome!

Monitoring informal learning with a Learning Record Store

I wrote this blogpost last year with Saskia Tiggelaar and Lisan van der Lee. It took me some time to translate it into English. It was written on the basis of a session with Ben Betts organized by MOOCfactory called 'Meeting more minds'. 

Experience API, XAPI, Learning Record Stores, TinCan API do you know what this is about? It all sound rather technical. Last year I participated in a fringe event during the Learning and Technologies conference and it was funny to be emerged into a topic I didn't have a clue what is was. I didn't even know what to ask. Nevertheless I think this is interesting for all learning professionals working with 70-20-10. In this blogpost I explain what a Learning Record Store is, what the challenges are and why you should be interested as learning professional.

Point of departure: a diverse online learning landscape 
We face similar challenges in monitoring learning as learning professionals, which is no coincidence. In the past five years many organizations worked hard to make educational materials available to employees. These e-learning efforts have taken all kind of shapes, blended learning, tutorials, games and apps. Most organizations also have a rich learning management system (LMS) with herein (self) developed materials appropriate to the particular training needs of the organization. As organizations begin to develop their own content, a considerable body of material becomes available within the LMS. In addition, content on websites like Youtube, Vimeo, blog sites are increasing. We are therefore becoming increasingly conscious that the making of "new" content is not necessary, because it is already available elsewhere. Enter content curation. Why invest in creating if you can use existing content? By curating content I mean that you select existing content and make it available to employees so that they can access the materials whenever they want to. We may call this a learning landscape. The challenge for the design of a good learning landscape lies in looking beyond borders and use of materials outside your own LMS, such as websites and video's. The requires systems flexibility and implications for the way you monitor the use of resources.

Ben Betts: Use any platform which seems appropriate, do not try to connect everything together, but make sure you have collected the data from the different systems in one place- the Learning Record Store.

A place for collecting all results: the Learning Record Store
The  Learning Record Store is the place where you store about about the use of all learning resources, whether in your LMS or other platforms, or social media.

Ben Betts: “it is actually a quite boring piece of software because it is nothing else than a database”.

Below in the graph you see an example where the data is collected in one place. A condition is that the data from the different platforms use Experience in API (XAPI). The definition of xapi "a standard way of talking about our experiences in using data.

A practical example: children in a museum
Ben Betts shared a few practical examples The first is the Ann Arbor museum,  a children's museum in the US. The children visit the museum and get a name tag. The tag can monitor what the children do at the museum, and this data is stored on the LRS. The teachers can then see which answers are given and what is popular. The teacher may use this information to adapt his class teachings.

Do you want to have a Learning Record Store in your organization?
Good for the Ann Arbor museum but why would you want a Learning Record Story in your organization? It is interesting to see what a LRS can do for an organization. Especially if you have the ambition to monitor both formal and informal activities of employees it might be useful. The most important and innovative feature of such a store is that it can collect data about different activities and different platforms. The main purpose of the monitoring is to improve the learning landscape. However, this is still a very wide target. That leads us to the most important question you have to answer as an organization: what data do you collect and why? One can think of several reasons; you can collect data to:

• Generate user feedback about the learning interventions (. eg e learning modules) that you have offered. Use this feedback to improve the learning interventions continuously;
• Find out which interventions the users choose to meet their learning needs;
• Predict what the needs / issues are a target group to arrive at advice for future interventions.

To answer the question what the purpose will be to collect and analyze this data, but there are some stepping stone questions. The question is: what is your vision on learning and development? This can be formulated in terms of competencies, but it can also involve social, personalized learning. L & D professionals are increasingly placing the learner at the centre, the learners must be able to take its own route and engage in workplace learning. So if you wish to monitor whether what you're doing as L & D department for your employees really works, formulate questions and hypotheses. This will help you get a grip on the data you will need to collect.

What about the privacy of employees, we can simply collect their learning data as an organization? Privacy and consent is an important issue on Learning Record Stores. On the one hand, there is no guarantee that the data is 100% safe. On the other hand, there are already a lot of data about employees in an organization. Sometimes these data are not or hardly used. Most important is to be transparent to employees on how to deal with their data. A good practice (which was also used in the example of the museum) is to anonymize the data. There is a  code of practice developed by JISC. This might give you some inspiration. 

In 10 years we might all walk with our own learning passport? 
Will the employee be better off with more self control and keep their own data? An interesting idea is that the employees themselves might have full control over their own data in the Learning Record Store- what we call a learning passport in this blog. It might gradually increasing the employee responsible. Ben Betts used the metaphor of a coffee card to describe the learning passport. As a coffee card is already partially filled, it motivating a customer to continue filling the coffee card. Our task for the passport of the employee to partially fill it with formal learning activities and the employee fills in the rest. This allows the employee to take its learning direction as much as possible in their own hands and the employee can decide which experiences are added to the passport. However, someone commented that many people have lost their diploma from school or university .. so how interested will people be in their own learning passport?

Where to start? Just start somewhere!  
The big question is the group was: where do you start? Do you start with a solid plan and legal support or do you start with smaller experiments? Should you form a data team or you can do it yourself as L & D professionals? On the one hand it is good to take an organizational perspective and to collaborate with other departments to also be able to link performance data for example. On the other hand, it is good to get going  for instance working with data that are already there. So you build a clearer case and you know better what you would like in the future.

Read my former blogpost "from intuition to know for sure" about how you can start at the level of a course to analyze. If you want to read a basic explanation of  Xapi by Learnovate click here

From intuition to knowing for sure: a case of applying learning analytics at course level

Blogpost is written in collaboration with Francois Walgering from MOOCfactory and Sibrenne Wagenaar from Ennuonline.

More and more trainers and facilitators use online platforms for online interaction to facilitate learning. This is akin to face-to-face facilitation but also has its own dynamics. For example, how do you know if an online article is read? Or how much time a participant spends viewing a video or doing an assignment? Which participants connect online? How to judge whether valuable conversations place? In a face-to-face setting you can ask process questions to participants: "do you need more time for this discussion?" And "an extra assignment about ... seems to make sense." You can observing interactions and sense the group's enthousiasme. You may adjust the learning process gradually as a result.

In an online environment you miss these kind of observations. Online, some data can be helpful. Data collected by the system, such as degree of online activity, time spent viewing source, reached level, length of discussions. The value of these data is in analyzing and interpreting: what we call learning analytics. We recently experienced the power of learning analytics and gladly share this experience with you.

The case: the food and nutrition security course


We have designed and facilitated a five-week online course (a so-called SPOC- Small Private Online Course) "Food and Nutrition Security". About 90 people took part, and we have worked with them in Curatr, a learning platform that supports social learning. Four of the five weeks had a specific theme in the fifth week was reserved to work on practical cases, brought forward by participants and experts. The participants received a certificate after earning a minimum number of points per week and writing a case reflection. 19 people have received the certificate.

Online monitoring- using your intuition
During the course, uur most important source of information were feedback from participants on the online platform and e-mail messages, about content as well as the process. We had a picture of important issues, issues that led to discussion and effective learning activities. For instance, we received many compliments on the weekly interactive webinars. Another example: after an upgrade of the software technical problems arose which discouraged participants. And a number of participants informed us that they appreciated the content very much, but got in trouble with new ad hoc task at work. By following discussions and explicit interaction with a number of participants, we have developed a feel for the quality of the learning process. A group of enthusiastic participants began to emerge as a core group. Intuitively we felt that we were on the right track with this online learning program. But ... what can the data can tell us?

Online monitoring- analytics
Curatr collects behind the scenes lots of data using Experience API (xAPI). Beyond the general data such as name, surname, e-mail, organization and function you also get very good insight into the 'User'- and' Usage 'data. What is the difference between user and usage data?

User data is: all qualitative data you collect within the platform:

• All threads are started by participants;
• All reactions started in several discussions;
• All sources (videos, PDFs, blogs, websites) which were added by participants;
• Any responses to learning activities offered (eg reflection questions, quiz, open questions)

Usage data are the quantitative data (see illustration)

• How many comments have been posted by participants;
• How many resources have been added by a participant;
• How many comments by those who have liked;
• Who what when level has been reached;
• Number of participants who completed the SPOC.

Learning Analytics involves looking at the data, analyzing it and acting on the results. We can perform learning analytics at different levels . The depth of analysis affects the reliability of the actions we arrive at. You could say that learning analytics roughly consists of five steps:

1. Visualize: You are viewing the data, for instance, 100 participants who completed the SPOC, 20 participants in the test with a level one completed. This qualitative data is generally speaking easy to generate from a learning platform.
2. Clustering: The data that you see, can be clustered: all activities of a participant, all threads which have been started, all the sources added. Together they form perhaps a cluster 'involvement' or 'quality'.
3. Relationships: For instance "after a mail by the facilitator the activity on the learning platform has gone up." Or "the most important activity on the learning platform is to discuss with each other. "
4. Patterns: activities that have repeatedly proven themselves. We see for example, by looking at different SPOCs that 80% of participants watching a video stop after three minutes. 
5. Actions for the future: Based upon the pattern 'participants stop a video after 3 minutes' we may decide to use only videos within that period, or ensuring that the core of the message is transmitted within the first three minutes of the video.

Learning analytics in the case of the Food and Nutrition Security SPOC

In the case of the Food and Nutrition Security course we have taken a number of peaks at the data like people logged in and the leadershipboard during the course. Afterwards, we dived deeper into the data, with extensive analysis and interpretation. This proved really valuable and has given us new insights regarding the manner of participation of participants and the motion when it comes to social learning. A few examples:

• Most of the participants who have ended high in the leadership board, earned points by contributing to the discussions. This group was also very visible to us and we had them 'spotted'. However, there appeared to be two participants who earned quite a few points, purely by studying sources. They had not responded online and had therefore remained invisible to us. While they actively participated in the course.
• We were also curious to know about sources and questions that entice good discussions. 'Good' a qualitative concept and difficult to measure. But some threads stood out because of the large number of responses from participants. We have analyzed those discussions discovered that the striking aspect here was the contribution of an expert / coach: the fact that the expert herself was involved in the discussion by asking questions, giving an opinion, made the discussion very lively.
• A diverse group took part in this course, and we were wondering if we would see differences in participation between certain "groups" (eg working in the Netherlands versus working in Asia, or working for NGOs versus working for government). This certainly was the case. Some "groups" had a much more explicit contribution to the social learning than other groups. Important to know when designing new courses! In the illustration, you can see the visualized total scores of the different groups.

Some of our lessons about using learning analytics as online facilitator

• Using learning analytics can be very supportive of your role as an online facilitator. You can find confirmation for possible interventions. In the beginning as an online facilitator you may be scared or disappointed by the fact that not all participants are active online. Why is that? Is it unclarity, disinterest? If data show that they participants have accessed the available resources but have not commented on the platform can may react different and try to stimulate reactions. You might invite those who did not respond specifically to do so.
• It is valuable to combine data from the 'system' with own feelings and observations, such as reactions of participants by mail. Thus observations and analytics can reinforce each other.
• Before you start with the learning process make an inventory of your questions and possible indicators and numbers. What would you like to see? Are you happy with 10 active participants? What sources are you doubtful about? If you do this before the start, you will know better what information will help you to gain insights during the course. 
• Make a plan for how often want to use and analyze data during the course. In 'Food and Nutrition Security' case we analyzed more deeply after the course. Of course we held an eye on the obvious data during the project (who log in and who contributed) but we could have put the data to better use. An exmple: afterwards we compared the activity in the various user groups. We could have done this even earlier and spend more energy to involve some groups. 
• The learning analytics as described took place at course level. You can also go one step further and compare the data from the corresponding course with data from other courses. The results from such analysis can contribute to conclusions that fuel future designs.
• Take time for accessing the data. Plan it in! Learning analytics is still a relatively new activity for many online facilitators. And certainly if you want to monitor during the course by means of data, it requires a regular look at the dashboard, analysis and opinion to determine your future interventions.
• And ... start conversations with people involved to interpret the data! Check the data with curiosity and discuss it with those involved. Feeling and intuition are crucial. However, sometimes data offer new perspectives but often just confirms a certain intuition. This may then be a catalyst in order to take action. In our case we had weekly progress discussions, which might have benefitted from some data interpretation. 

Last but not least ... be careful about how you use your data. Be transparent about your intention with it. Tell participants very clearly what you will do with the data. After all, you are working with data on the performance of individuals in a learning environment.

Data is beautiful (but not many learning professionals believe this)

I read the book 'Big learning data'. I do like playing with data and always liked mathematics.  

Learning data is a new topic - quite big and still evolving. In the book a referal is made to 1000 self-assessments by learning professionals rating their skill set. Data interpretation was one of the lowest scoring skills of learning professionals!. Much higher scored presenting, facilitating skills etc. In other words, learning professionals are usually not the first to dive into numbers. Most betas are not learning professionals.

What is learning analytics? Learning analytics is about the use of data for learning and improving learning processes. In the cartoon above, for example, you see that the fortune teller used Facebook as a source of information to predict the the future. Smart of her ofcourse :). As a professional learning you can now do like the fortune teller using data (information) online. The book focuses on the use of big data in organizations to support, especially large-scale data learning. What I miss in the book is where you may practically start within an organization, even though they explain you can start with a training dashboard where you systematically collect data.

In this blog post I will try to propagate more use of data by learning professionals by making it small and practical and looking at three levels:

1. the level of your own online learning network (also called personal learning network PLN) 
2. the level of an online course or course 
3. the level of an organization

I think that sometimes you have data available at hand that you are not  (yet) using as a professional but could improve your work. And on the other hand, there are new tools like Google Analytics or Twitter Analytics that you can use to start collecting data.

Level 1: the level of your Personal Learning Network (PLN) 
At the level of your own online network you can measure a lot, depending on what your goals are. Think of it as a form of feedback to collect and analyze feedback. For example: you can measure the number of retweets on Twitter. We have an en_nu_online account on twitter where we share a tip everyday. I follow the number of retweets with the aim to see which tips are populair. I do this mainly to with the "my tweets, retweeting." column in Hootsuite. Every month I try to gather the totals and assemble those in an excel sheet. I have noticed that a lot of very practical tweets are retweeted  - this helps me to focus the upcoming tips focus. See also my blog post "Do not follow your number of followers, but you mentions and retweets. New to Twitter is that you can also turn on your analytics. I did this yesterday and you get a lot of information about your tweets.

Level 2: the level of an online cursus or learning trajectory 
For the course 'learning and changing with new media, we use an online platform, a Ning platform. Within this platform, you can also make use of data. For example you can see which topics were given a lot of responses. We use this type of information as we go through redesigns. But apart from the number of responses you can see the number of views. I use this information when I receive few reactions. Sometimes people read it but it is still a heavy topic to respond to. Most platforms do have data, and you want more than you could make use of Google Analytics. Qualitatively you might analyse the content of a course for instance with a wordcloud (eg. wordle or tagxedo). In Moodle I often monitor the participants who have not logged on for 5 days.

Level 3: the level of an organization or network 
At the level of an organization or school learning analytics is a bit more complex. Within an organization or school is it really a major project since you also have to look at the performance data and dashboards which already exist. It is best when you can make a link between performance and assessment and training / informal learning. Who can play what role in such a project? Think of the Research and Development, Learning and Training (HRD), Data scientists and management departments. Perhaps a good start within an organization or school to see what data you actually use . Sometimes gathering data in an excel sheet can be a big step. Additionally, you can think about an organizational challenge to solve. How can you start collecting to progress in this issue? So start with a question. I actually think the book 'measuring the networked non-profit' van Beth Kanter en Katie Delahaye Paine  might be a more practical book than 'big learning data'. A quote from that book is 'deciding what to measure is 90% of the process'.

Tags: analytics, learning