Building Assessment for the Digital Future

Recent conversations with colleagues at AdelaideX have us wondering what assessment will look like in the future. When I think about that, the questions that come to mind are:

  • What isn’t working now about how we assess learners? and
  • What might assessment that leverages the potential of technology to more efficiently and effectively assess at scale look like?
Part technology, part alien, but human underneath…?

The latter is the more intriguing and attractive question, isn’t it? The first few ideas that came to my mind were:

Read More »

Schrodinger’s LMS: the future is kinda almost here

In reply to Mike Goudzwaard:

Mike, who I had the absolute pleasure of meeting at the edX Global Forum in 2015, makes an interesting case about the issues facing the current iteration of LMSs and a suggestion for fixing it. He points to a future LMS that is clean, simple and heavily reliant on integration; where the “LMS” that an institution uses is focuses on collating learners into a learning group (i.e. a course), presenting a variety of learning activities (via LTI or API integration) and extracting all of that delicious grade and ‘engagement’ data out into an SIS.

What would this LMS look like? In my view, it would have three things:
1) a course roster with stellar SIS integration
2) a gradebook
3) a rock-star LTI and API

That’s it! Oh, except it would also be open source, students would control their own data, including publishing any of their work or evaluations to the block chain, and you could host it locally, distributed, or in the cloud. Never mind the pesky privacy laws (or lack thereof) in the country hosting your server, because the LMS is back on campus. Not connected to the internet? That’s okay too, because there is a killer app that syncs like a boss


Which, I’ll admit, sounds wonderfully clean and smooth, like a marble kitchen benchtop after you’ve cleaned all the remains of a delicious dinner and wiped away the crumbs until it shines and gleams, full of promise…


A couple of thoughts came to mind as I read Mike’s post:

  1. Is it just me, or does this sound like a stripped down Moodle, i.e. could it be, maybe, almost, already here or within our grasp?
  2. Is there anyone in the LMS market now who could deliver this?
  3. Would this necessarily result in a better experience for learners, educators, IT staff? Where the educators I have in mind are those laggards who resist change and are uncomfortable with technology

Before I go too much further, let me point out that at this stage I have personally worked with (i.e. completed a course in, built in, delivered in, designed for) more than 5 LMSs – Blackboard Learn while at university and then later as a Solutions Engineer, Moodle (including the Moodlerooms delivery) while at NetSpot/Blackboard and then ICAC, and then MOOCs in Canvas, Open2Study (built on Moodle and Drupal) FutureLearn, Coursera and, now, edX.*

Some have more of this feature, some do that feature better, some structure the course in this way, some guide you to design a course in that way, some offer more flexibility as a designer v a learner, etc.

The similarities, though – presentation of learning content/activities, learner management (comms, enrolments, progress, etc), and learning management (lists of courses, gradebook) – definitely exist, if in overall functionality and not in the exact manner of workflow.

So, is there an existing potential “Future LMS”? Reading through Mike’s description, Moodle and its open-source development and modular nature came to mind.

Moodle has:

  1. SIS integration (limited by default, extensible via plugin, incredibly easy via Moodlerooms extension) and display of courses
  2. A gradebook with rather in-depth functionality
  3. LTI and API integration (I’ll admit my knowledge of implementing these is limited, so AFAIK it’s not the best in the LMS market, but it certainly is easy to install an LTI activity into a course as I’ve tried)

Moodle also ticks some of the other boxes he suggests:

  • Open-source? Yes. And with a rather robust revenue-generating model based on royalties from organisations that provide services
  • Multiple hosting/distribution options? Yes, with numerous universities managing their own installation here in Australia through to managed hosting with a third-party organisation, and more recently small-scale cloud-hosting through Moodle HQ itself

Some of Mike’s other points I either don’t know or would, to my knowledge, need considered development:

  • Learner-owned data – Moodle doesn’t, as far as I know, truly support the learner-ownership of data and privacy, but that seems more about policies and the ability to store backups as a user level
  • Student’s republishing their own content? Well, you can already use an e-portfolio, a la open-source Mahara to push student-created content from Moodle to a more visible container of personal content and reflection. I personally would love to see a combination of Mahara functionality and Blackboard’s MyEdu extensibility and looks, because I have a concern about learners being able to save their Mahara portfolio and be able to display that anywhere meaningfully. It could also be an opportunity to extend the Mozilla Open Badges framework to store examples of the activity completed to earn a badge?
  • Offline support? Definitely possible, depending on site administration and how the course has been developed. The Moodle mobile app supports (according to the app information in iTunes) offline ‘browsing’ of course content. To me, it’s the same idea as Spotify’s ‘save to offline’ functionality (yes I pay for Premium Spotify. I don’t know how you couldn’t with those stupid ads), allowing the user control  of which songs/artists/playlists are available offline, and then syncing your plays later.
  • Killer app that supports syncing? Definitely possible. Moodle’s mobile app is, in my opinion, underwhelming, although I’ve not used it in a while. But something similar to Evernote or OneNote would be amazing – multiple types of files, notifications, etc.

In my opinion, Moodle presents a feasible option for developing the “Future LMS” Mike proposes. It would, however, require a significant pivot in Moodle HQ’s strategic plan which is (unfortunately?) unlikely given the shift to more directly involve Moodle users in the development priorities with the advent of the Moodle Users Association.

So, if not Moodle, then who?

Well, there’s Instructure with their Canvas LMS…

Canvas’ extensive support and proven development (especially in LTI) could make them a front-leader in this, and they do have an open-sourced version of their software to offer users an option besides Instructure-managed cloud-hosted. However, it is not the same kind of open-source in the same way that Moodle is (something about AGPL v GNU GPL , but is similar to Kaltura’s open-source community version v managed/supported hosted version.

On the other hand, they have, I believe, the vision, drive and funding to be able to create a “future LMS” and perhaps open-source it once developed. A business model could include offering cloud-hosting, as they currently do, and using that funding to invest in the further development of LTI, APIs, offline caching, etc.

D2Ls Brightspace

Don’t make me laugh. They appear, to me, such a second-rate runner in this race (judging their prevalence in Australian HE/K12/Corporate and overseas), their software seems to inspire very little excitement and they’re private-equity-owned that I’ll just leave it there.

Blackboard Learn/Moodlerooms?

Blackboard is possibly in the best position to be able to offer the functionality Mike is suggesting, with their market ownership and (relative) certainty with private equity ownership and 2 LMS platforms to present to the market. However, their private equity ownership would, I imagine, forestall any decision to remove the ability to force receive large regular payments from clients.

Their Moodlerooms LMS is probably the only option for them to develop this future LMS. Blackboard could leverage the existing functionality of Moodle (perhaps by forking their Moodlerooms platform?), invest funds in further development in LTI (which will also benefit the Learn and other Blackboard platform), build a better Moodle app, get better integration with both Mahara and MyEdu, open-source the lot and offer hosting services (similar to the existing Moodle partner model and Moodlerooms business model – essentially a managed hosted Moodle with additional enhancements as standard) to generate revenue.

Let’s be honest though, it’s unlikely any of these companies will be interested in making this investment, because, duh, capitalism.

It may be something addressed by a new startup, sure. From what I’ve seen, however, they seem pretty focused on recreating what is currently available but making it easier to administer/mobile friendly/prettier/etc. Not a lot of innovation in the actual functionality or architecture that I can see.

Which begs the question, if this future LMS isn’t available now and it’s unlikely to be developed by any company on the current horizon, is it possible? And would it be worthwhile, even if it could be developed?

A significant concern I have is that the level of abstraction required to make this future LMS function well is massive. Massive in terms of the coding (the interoperability and likely change in software design), scale (the number of applications or pieces of software that would be integrating with the LMS) and the overall global movement towards interoperability and integration – it would be so significant as to render it not quite, but almost,  impossible. The LTI standard has been v1.0 since 2010 and it’s not pervasive. And this future LMS requires interoperability of not only data (to share back to the gradebook) but also functionality.

I raise this because how would the learner (or even the educator/designer) experience be if upon clicking on a link to a piece of content or learning activity, it is completely different to every other learning experience had until that point? How would university teams, or IT, support that variablity? How would educators?

David Jones, from USQ, posted a thought-provoking blog recently about the mismatch of mental models and ICT:

Koehler and Mishra (2009) have this to say

Digital technologies—such as computers, handheld devices, and software applications—by contrast, are protean (usable in many different ways; Papert, 1980); unstable (rapidly changing); and opaque (the inner workings are hidden from users; Turkle, 1995).On an academic level, it is easy to argue that a pencil and a software simulation are both technologies. The latter, however, is qualitatively different in that its functioning is more opaque to teachers and offers fundamentally less stability than more traditional technologies. By their very nature, newer digital technologies, which are protean, unstable, and opaque, present new challenges to teachers who are struggling to use more technology in their teaching. (p. 61)

With that in mind, I can’t help but wonder, would the level of digital literacy required from both learners and educators to navigate and create (respectively) a digital learning experience with confidence, ease and purpose be beyond the levels of the average user?

In my own experience, even with the consistent and structured environments of spaces like Facebook and Blackboard Learn there are users that find it incredibly difficult without explicit assistance. How much would this be exacerbated with an environment that is extended almost infinitely with plugins to other systems?

For instance, consider the differences between YouTube and Vimeo. Incredibly simply in terms of functionality in a learning context, it should amount to simply uploading, embedding and viewing a video. Except each platform have different approaches to uploading a video, especially the privacy options, embedding a video (a user generates the embed code slightly differently and there are different analytics information available in each platform), and while selecting to play a video is pretty much the same, the workflow to share a video or view captions is slightly but significantly different – different icons, different lexicon.

What kind of learning experience would this provide for learners? To enter one system, then, when they select to view one activity they experience one interface with its own rules and lexicon, then return to the LMS and select a new activity only to have to navigate a new interface and lexicon.

What kind of support would be required for learners and educators from an institutions IT department to make this work?

From what I can see, success in this future LMS would require a level of abstraction and standardisation of functionality across a variety of platforms and technologies in order to provide a reasonable consistency of user experience such that the learner can focus on the learning rather than trying to understand and navigate a new system.

In an ideal world, users, at both learner and educator roles, would be familiar enough with a variety of platforms and functionality workflows that this could be overcome. However, as David says:

Digital technologies are opaque. It’s not easy to get a handle on the models that underpin the design and implementation of digital technologies. 

To make this future LMS a success would require a concerted effort to remove the opaqueness of almost all digital technologies, even those that might not consider themselves a part of the edtech world.

How much time, money and effort are learning institutions, let alone edtech and other companies, prepared to commit to this global holistic movement to open up and standardise functionality?

Would this level of standardisation required foster or inhibit innovation in learning? I’m not sure, but my initial thought is that it would be likely to inhibit innovation – in the long-term, on the part of providers of learning content, because of the constraints of the interoperability demands, and in the short-term, on the part of educators looking to take advantage of interoperable content but restricted by the LTI/API compatible options while the world wide web wakes up to the opportunities available in edtech.

And so, this future LMS Mike Goudzwaard proposes is an intriguing concept in its beautiful clean lines, but it seems to be a paradox

  • It is both available already now (kind of) and only in the future (pending development)
  • It is both possible (considering the options of organsiations available who could develop it) and impossible (given the organisations available are highly unlikely to develop it and the relatvei unattractiveness for to-be-developed startups)

The answer to the LMS debate? I don’t have it. And I’m unconvinced it’s a problem that needs solving in developing a completely new system.

Michael Feldstein posed a valid question about whether the procurement processes of large client organisations have a part to play in the way the LMS has developed (and failed)

From my experience, on both vendor- and client-side, I can certainly relate to a lot of what he outlines.

So, perhaps, there’s something to be said for (potential) clients considering how best to collate, articulate and communicate their needs to edtech vendors, rather than edtech vendors trying to second-guess. Maybe?

What do you think?


*As I explained to a colleague recently, this is not my first rodeo.

My kind of scaffold – part 2

Part 2 of the series of posts wherein I share the process I’ve used to try build a “different” scaffold for compliance-based courses.

Building wall of mismatching windows. By Edgaras Maselskis, via Unsplash. CC0
In a previous post, I spoke about how I used a combination of butchered action-mapping and Bloom’s taxonomy to create course objectives with actionable verbs and that provided more direct mapping to activities. Here, I want to share the course design and activity design approach, and why I think this goes beyond what many (especially in corporate/compliance training) believe constitutes ‘eLearning’ – the “Next” button.

One of the common pitfalls of a Learning Management System (Moodle, Blackboard Learn, Canvas) is that they don’t necessarily support the creation of learning ‘content’ that is typically expected in corporate environments. Instead, the majority of them support discrete activities that are collated together to form a learning experience rather than a coherent and simple flow*. Another common downfall arising out of using LMS’s is that, despite the built-in activities common in LMS’s, all too often they’re used to house individual resources (pdf files, URLs, word documents, videos) and then an in-built quiz to assess the learning. Having worked on both the vendor and the user sides of the LMS/edtech world, I would purport that the root of this both in the technology and the typical user – but that’s a conversation for another day.Read More »

A stuttering spluttering start

The rambling story of my first few weeks…


I don’t know about you, but the memories of ‘health and safety’, ‘accepted conduct’ or other general induction and compliance courses don’t fill me with excitement. So, I’m rather excited about the the challenge of trying to deliver compliance training in a way that’s effective and not completely mind-numbing.

The first hurdle that sprang itself up at me was the content. One of my team’s objectives is to raise awareness of legal obligation imposed on all public officers in South Australia, an obligation that is outlined in deep legalese which, unless you have experience reading and interpreting the law you’re likely to struggle to comprehend it. And the stakes are high – they are obliged to know, and failure to comply can mean pretty hefty consequences. And so, we have a responsibility to ensure information is available that helps people understand their obligations and how to prevent corruption, misconduct and maladministration.Read More »

A new path to wander

It’s been a while, hasn’t it? It’s not that I haven’t thought of you often, of what I would say or things you might enjoy. It’s just, well, time and my own inability to organise myself…


You might notice a difference in the types of things I post from here on out in comparison to previous posts. And that’s because, well, here’s my official news: I left Blackboard in March. Although I tweeted about it a few times, there are some who still didn’t know. I wasn’t there as long as some people, certainly not as newsworthy as others, but the positive comments on twitter when I shared my news were rather soothing to my frail ego.Read More »

Measuring the competencies of Outcomes

Recently, I’ve been working more closely with Registered Training Organisations (RTOs) and their online learning needs. We’ve been bringing on more and more clients to the Moodlerooms platform (link) because of a number of exciting enhancements on top of core Moodle. One of those that I’m really excited about is a new version of Outcomes – tracking, aligning and reporting on learning outcomes or competencies at a course level.

Currently in core Moodle (on which Moodlerooms is based), outcomes don’t quite match what educators are looking in terms of managing learning outcomes:

You can choose a list of outcomes for the site, and you can choose a subset of those for the course, and you can assign outcomes to various activities, and even set grades (via scale) for them in assignments.  This helps somewhat for course design and some grading, but there is no transference into competencies, and there is no concept of progress tracking for students based on these. (Martin Dougiamas, ‘Outcomes, Stage 2’)

The Moodle outcomes system as it exists today does not accomplish what customers demand and require, due to some key missing functions, including the ability to easily import hierarchical standards and the ability to map and track outcomes on quiz questions and rubric rows. This limits Moodle adoption in K12, corporate, for-profit, and, to a lesser extent, higher education, specifically community colleges. (Outcomes Specification, Moodle Docs)

Read More »

Badges – ringeth the eportfolio death knell?

Last night, Steve Wheeler pondered “Do open badges signal the death of the e-portfolio? #learningpoollive“. The sentiment of that conversation is that, yes, Steve and others believe they do.

I disagree.

The key reason I don’t believe that badges signal the death of the eportfolio is the purpose for their respective initial creation, ongoing development and real-life application.Read More »