Building Assessment for the Digital Future

Recent conversations with colleagues at AdelaideX have us wondering what assessment will look like in the future. When I think about that, the questions that come to mind are:

  • What isn’t working now about how we assess learners? and
  • What might assessment that leverages the potential of technology to more efficiently and effectively assess at scale look like?
Part technology, part alien, but human underneath…?

The latter is the more intriguing and attractive question, isn’t it? The first few ideas that came to my mind were:

Read More »

Schrodinger’s LMS: the future is kinda almost here

In reply to Mike Goudzwaard:

Mike, who I had the absolute pleasure of meeting at the edX Global Forum in 2015, makes an interesting case about the issues facing the current iteration of LMSs and a suggestion for fixing it. He points to a future LMS that is clean, simple and heavily reliant on integration; where the “LMS” that an institution uses is focuses on collating learners into a learning group (i.e. a course), presenting a variety of learning activities (via LTI or API integration) and extracting all of that delicious grade and ‘engagement’ data out into an SIS.

What would this LMS look like? In my view, it would have three things:
1) a course roster with stellar SIS integration
2) a gradebook
3) a rock-star LTI and API

That’s it! Oh, except it would also be open source, students would control their own data, including publishing any of their work or evaluations to the block chain, and you could host it locally, distributed, or in the cloud. Never mind the pesky privacy laws (or lack thereof) in the country hosting your server, because the LMS is back on campus. Not connected to the internet? That’s okay too, because there is a killer app that syncs like a boss


Which, I’ll admit, sounds wonderfully clean and smooth, like a marble kitchen benchtop after you’ve cleaned all the remains of a delicious dinner and wiped away the crumbs until it shines and gleams, full of promise…


A couple of thoughts came to mind as I read Mike’s post:

  1. Is it just me, or does this sound like a stripped down Moodle, i.e. could it be, maybe, almost, already here or within our grasp?
  2. Is there anyone in the LMS market now who could deliver this?
  3. Would this necessarily result in a better experience for learners, educators, IT staff? Where the educators I have in mind are those laggards who resist change and are uncomfortable with technology

Before I go too much further, let me point out that at this stage I have personally worked with (i.e. completed a course in, built in, delivered in, designed for) more than 5 LMSs – Blackboard Learn while at university and then later as a Solutions Engineer, Moodle (including the Moodlerooms delivery) while at NetSpot/Blackboard and then ICAC, and then MOOCs in Canvas, Open2Study (built on Moodle and Drupal) FutureLearn, Coursera and, now, edX.*

Some have more of this feature, some do that feature better, some structure the course in this way, some guide you to design a course in that way, some offer more flexibility as a designer v a learner, etc.

The similarities, though – presentation of learning content/activities, learner management (comms, enrolments, progress, etc), and learning management (lists of courses, gradebook) – definitely exist, if in overall functionality and not in the exact manner of workflow.

So, is there an existing potential “Future LMS”? Reading through Mike’s description, Moodle and its open-source development and modular nature came to mind.

Moodle has:

  1. SIS integration (limited by default, extensible via plugin, incredibly easy via Moodlerooms extension) and display of courses
  2. A gradebook with rather in-depth functionality
  3. LTI and API integration (I’ll admit my knowledge of implementing these is limited, so AFAIK it’s not the best in the LMS market, but it certainly is easy to install an LTI activity into a course as I’ve tried)

Moodle also ticks some of the other boxes he suggests:

  • Open-source? Yes. And with a rather robust revenue-generating model based on royalties from organisations that provide services
  • Multiple hosting/distribution options? Yes, with numerous universities managing their own installation here in Australia through to managed hosting with a third-party organisation, and more recently small-scale cloud-hosting through Moodle HQ itself

Some of Mike’s other points I either don’t know or would, to my knowledge, need considered development:

  • Learner-owned data – Moodle doesn’t, as far as I know, truly support the learner-ownership of data and privacy, but that seems more about policies and the ability to store backups as a user level
  • Student’s republishing their own content? Well, you can already use an e-portfolio, a la open-source Mahara to push student-created content from Moodle to a more visible container of personal content and reflection. I personally would love to see a combination of Mahara functionality and Blackboard’s MyEdu extensibility and looks, because I have a concern about learners being able to save their Mahara portfolio and be able to display that anywhere meaningfully. It could also be an opportunity to extend the Mozilla Open Badges framework to store examples of the activity completed to earn a badge?
  • Offline support? Definitely possible, depending on site administration and how the course has been developed. The Moodle mobile app supports (according to the app information in iTunes) offline ‘browsing’ of course content. To me, it’s the same idea as Spotify’s ‘save to offline’ functionality (yes I pay for Premium Spotify. I don’t know how you couldn’t with those stupid ads), allowing the user control  of which songs/artists/playlists are available offline, and then syncing your plays later.
  • Killer app that supports syncing? Definitely possible. Moodle’s mobile app is, in my opinion, underwhelming, although I’ve not used it in a while. But something similar to Evernote or OneNote would be amazing – multiple types of files, notifications, etc.

In my opinion, Moodle presents a feasible option for developing the “Future LMS” Mike proposes. It would, however, require a significant pivot in Moodle HQ’s strategic plan which is (unfortunately?) unlikely given the shift to more directly involve Moodle users in the development priorities with the advent of the Moodle Users Association.

So, if not Moodle, then who?

Well, there’s Instructure with their Canvas LMS…

Canvas’ extensive support and proven development (especially in LTI) could make them a front-leader in this, and they do have an open-sourced version of their software to offer users an option besides Instructure-managed cloud-hosted. However, it is not the same kind of open-source in the same way that Moodle is (something about AGPL v GNU GPL , but is similar to Kaltura’s open-source community version v managed/supported hosted version.

On the other hand, they have, I believe, the vision, drive and funding to be able to create a “future LMS” and perhaps open-source it once developed. A business model could include offering cloud-hosting, as they currently do, and using that funding to invest in the further development of LTI, APIs, offline caching, etc.

D2Ls Brightspace

Don’t make me laugh. They appear, to me, such a second-rate runner in this race (judging their prevalence in Australian HE/K12/Corporate and overseas), their software seems to inspire very little excitement and they’re private-equity-owned that I’ll just leave it there.

Blackboard Learn/Moodlerooms?

Blackboard is possibly in the best position to be able to offer the functionality Mike is suggesting, with their market ownership and (relative) certainty with private equity ownership and 2 LMS platforms to present to the market. However, their private equity ownership would, I imagine, forestall any decision to remove the ability to force receive large regular payments from clients.

Their Moodlerooms LMS is probably the only option for them to develop this future LMS. Blackboard could leverage the existing functionality of Moodle (perhaps by forking their Moodlerooms platform?), invest funds in further development in LTI (which will also benefit the Learn and other Blackboard platform), build a better Moodle app, get better integration with both Mahara and MyEdu, open-source the lot and offer hosting services (similar to the existing Moodle partner model and Moodlerooms business model – essentially a managed hosted Moodle with additional enhancements as standard) to generate revenue.

Let’s be honest though, it’s unlikely any of these companies will be interested in making this investment, because, duh, capitalism.

It may be something addressed by a new startup, sure. From what I’ve seen, however, they seem pretty focused on recreating what is currently available but making it easier to administer/mobile friendly/prettier/etc. Not a lot of innovation in the actual functionality or architecture that I can see.

Which begs the question, if this future LMS isn’t available now and it’s unlikely to be developed by any company on the current horizon, is it possible? And would it be worthwhile, even if it could be developed?

A significant concern I have is that the level of abstraction required to make this future LMS function well is massive. Massive in terms of the coding (the interoperability and likely change in software design), scale (the number of applications or pieces of software that would be integrating with the LMS) and the overall global movement towards interoperability and integration – it would be so significant as to render it not quite, but almost,  impossible. The LTI standard has been v1.0 since 2010 and it’s not pervasive. And this future LMS requires interoperability of not only data (to share back to the gradebook) but also functionality.

I raise this because how would the learner (or even the educator/designer) experience be if upon clicking on a link to a piece of content or learning activity, it is completely different to every other learning experience had until that point? How would university teams, or IT, support that variablity? How would educators?

David Jones, from USQ, posted a thought-provoking blog recently about the mismatch of mental models and ICT:

Koehler and Mishra (2009) have this to say

Digital technologies—such as computers, handheld devices, and software applications—by contrast, are protean (usable in many different ways; Papert, 1980); unstable (rapidly changing); and opaque (the inner workings are hidden from users; Turkle, 1995).On an academic level, it is easy to argue that a pencil and a software simulation are both technologies. The latter, however, is qualitatively different in that its functioning is more opaque to teachers and offers fundamentally less stability than more traditional technologies. By their very nature, newer digital technologies, which are protean, unstable, and opaque, present new challenges to teachers who are struggling to use more technology in their teaching. (p. 61)

With that in mind, I can’t help but wonder, would the level of digital literacy required from both learners and educators to navigate and create (respectively) a digital learning experience with confidence, ease and purpose be beyond the levels of the average user?

In my own experience, even with the consistent and structured environments of spaces like Facebook and Blackboard Learn there are users that find it incredibly difficult without explicit assistance. How much would this be exacerbated with an environment that is extended almost infinitely with plugins to other systems?

For instance, consider the differences between YouTube and Vimeo. Incredibly simply in terms of functionality in a learning context, it should amount to simply uploading, embedding and viewing a video. Except each platform have different approaches to uploading a video, especially the privacy options, embedding a video (a user generates the embed code slightly differently and there are different analytics information available in each platform), and while selecting to play a video is pretty much the same, the workflow to share a video or view captions is slightly but significantly different – different icons, different lexicon.

What kind of learning experience would this provide for learners? To enter one system, then, when they select to view one activity they experience one interface with its own rules and lexicon, then return to the LMS and select a new activity only to have to navigate a new interface and lexicon.

What kind of support would be required for learners and educators from an institutions IT department to make this work?

From what I can see, success in this future LMS would require a level of abstraction and standardisation of functionality across a variety of platforms and technologies in order to provide a reasonable consistency of user experience such that the learner can focus on the learning rather than trying to understand and navigate a new system.

In an ideal world, users, at both learner and educator roles, would be familiar enough with a variety of platforms and functionality workflows that this could be overcome. However, as David says:

Digital technologies are opaque. It’s not easy to get a handle on the models that underpin the design and implementation of digital technologies. 

To make this future LMS a success would require a concerted effort to remove the opaqueness of almost all digital technologies, even those that might not consider themselves a part of the edtech world.

How much time, money and effort are learning institutions, let alone edtech and other companies, prepared to commit to this global holistic movement to open up and standardise functionality?

Would this level of standardisation required foster or inhibit innovation in learning? I’m not sure, but my initial thought is that it would be likely to inhibit innovation – in the long-term, on the part of providers of learning content, because of the constraints of the interoperability demands, and in the short-term, on the part of educators looking to take advantage of interoperable content but restricted by the LTI/API compatible options while the world wide web wakes up to the opportunities available in edtech.

And so, this future LMS Mike Goudzwaard proposes is an intriguing concept in its beautiful clean lines, but it seems to be a paradox

  • It is both available already now (kind of) and only in the future (pending development)
  • It is both possible (considering the options of organsiations available who could develop it) and impossible (given the organisations available are highly unlikely to develop it and the relatvei unattractiveness for to-be-developed startups)

The answer to the LMS debate? I don’t have it. And I’m unconvinced it’s a problem that needs solving in developing a completely new system.

Michael Feldstein posed a valid question about whether the procurement processes of large client organisations have a part to play in the way the LMS has developed (and failed)

From my experience, on both vendor- and client-side, I can certainly relate to a lot of what he outlines.

So, perhaps, there’s something to be said for (potential) clients considering how best to collate, articulate and communicate their needs to edtech vendors, rather than edtech vendors trying to second-guess. Maybe?

What do you think?


*As I explained to a colleague recently, this is not my first rodeo.

Rape ‘Artistry’ should be “UnAustralian”

The following is a copy of an email I sent to Minister Dutton this morning, after reading that more ‘workshops’ are scheduled to be delivered in Australia early this year by a group renowned for perpetuating and perpetrating violence and abusive behaviour towards women and, perhaps worst of all, profiting off the incitement of this deplorable and despicable behaviour in others. If you agree, you can contact Minister Dutton and/or sign the Change.Org petition to deport/revoke the visas of those involved

Dear Minister Dutton,

I write to you to request that the visas of anyone involved with Real Social Dynamics, including Julien Blanc, Owen Cook and Jeff Allen, be revoked because of their public history of perpetrating and inciting in others violence against women.

This current government has previously stated that

This man and this team are part of the domestic violence problem globally and contribute to the problem here in Australia. We know how many women died at the hands of intimate partners last year – the belief that it’s acceptable to behave in that way starts with the belief that behaviour such as belittling, physically restraining, physically harming, mentally abusing women is also acceptable. Our government needs to prove that women deserve respect. That decent human beings respect women. That healthy relationships form out of respect for women. That women are not sexual objects to be acquired and won. Our government can take action by refusing the visas and releasing a public statement as to the reasons behind it.

Do you think that women are less worthy than men, Minister Dutton? Do you think that women are less deserving of respect and safety? Do you believe that it is acceptable that men disrespect women? Do you think that men should have the right to commit violent acts against women?

If you allow these men to come to Australia and deliver their talks, you and your government will be saying exactly that.

I have signed this petition at Change.Org that has the following text:

Australia says NO to Jeff Allen and his teaching of violence against women.

Please consider acting as Scott Morrison did in the role of Immigration Minister when Jeff Allen’s colleague Julien Blanc tried to tour the country teaching men domestic violence tactics in workshop events, and revoke Allen’s visa.

Australia has lately been discussing violence a lot as we try to work out how to reduce the number of domestic partnerships which are fraught with violence and abuse. A good step in tackling this would be to say very clearly that we do not hand out visas for people to come to Australia and promote, and profit from the promotion of, domestic violence and specifically violence against women.

These men have built a business advising young men that their best option for a romantic or sexual encounter with a woman is to belittle her, physically restrain her, mentally deride her, and abuse her until she submits. These tactics are promoted on their social media and in videos from their seminars, and now they are touring the country preaching to young Aussie men this same dangerous mantra.

If this Coalition Government is serious about promoting equality for women, denouncing violence against women, preventing the growth of a culture of domestic and public violence of all varieties, a good indication of this would be not to issue business visas to people whose business model is founded on workshops which teach and promote abuse which may very well border on criminal.

Australia, like many other nations, requires that visitors and people who are wishing to migrate are of good character as outlined in the Department of Immigration policies. In making a determination on whether a person is fit for entry into the country, the Department applies a ‘character test’ which ultimately decides whether or not a person is granted a visa to enter Australia.

The Minister for Immigration and Citizenship (the Minister) and a decision maker in the Department have the power to make a decision to grant or refuse a visa on character grounds on a case-by-case basis. It’s important to note, that the Minister or the Department have the authority to revoke a visa that has already been granted if it is later established that the person has failed the character test.

A person is deemed to have failed the character test if:

• they have a substantial criminal record
• they have an association with another individual, group or organisation whom the Minister reasonably believes has engaged in criminal conduct
• they are likely to engage in criminal activities in Australia
• they are likely to harass, stalk, or molest another person while in the country
• they are likely to incite discord within the community, or a segment of the community within Australia
• they represent a danger to the community, or a segment of that community by being likely to become involved in activities that are disruptive, violent or threatening.

Jeff Allen represents a profit-making group advocating tactics which include harassing and stalking women, they advocate a mentality to women that does not value consent, and they represent a very real danger to the community, they are also profiting from a business model which encourages violent and threatening behaviours, it is very likely that Jeff Allen would fail the character test just as Julian Blanc retroactively did when his visa was revoked.

We ask that you apply the character test to Jeff Allen and any associates from Real Social Dynamics, to determine grounds to revoke visa approval.




It was a good day

Written this morning, on the train ride commute into the city… 

Today is a good day.

Forest scene - pic from unsplash

I feel rested after a week holidaying at home in NZ. I got to bathe in the wonderful comfort of family and familiarity, so warm that even the 40+ degree summer days in South Australia can’t quite compete. I got to see NZ’s beauty as new again through Mark’s eyes. I switched off from work completely – the first Xmas/New Year’s Day break in a few years where I could actually do that. And on my return home I got an afternoon/evening of doge cuddles, who smelled of home and happiness and love.

And now, it’s the first day of work for the calendar year.

I am optimistic again. Enthusiastic again. Feeling curious and empowered again.

I have big plans for today, and for 2016.

Recent years have felt like hard slog – demolishing, rebuilding, discovery, negotiating, crawling, recovery. Whereas previous years have required a complete demolish and rebuild of foundations – sometimes huddling in a caravan, avoiding leaks as torrential rain pours that halts any work – this year feels different. The foundations are complete and strong. The scaffold is there – it’s time to make it whole and beautiful.

How nice it feels.

Today is a good day.

Manakau, NZ - pic by Mark Drechsler

Just 5 minutes…

Reading through The Thesis Whisperer’s blog post about the exploring options for journal article titles (in the context of her workshop on writing a journal article in 1 week), I stumbled upon the her challenge to free-write for 5 minutes every day for 2 weeks.

Macbook laptop semi-open surrounded by dark

It so happens that every day, on average, I have 2x about 30 minutes of train travel of commute between home and work. I keep thinking I should be using this time more productively. I even had grand notions of using the time each day to draft/review/publish a blog post, in order to try and write at least once a week the myriad of thoughts that flow through my head.

So far, that’s happened sporadically.

But now, I think I want to try it again. I feel like it’s similar to quitting smoking – every time you quit you’re closer to being a non-smoker long-term. Every cigarette you don’t smoke you win, you save money, you increase your health. I really despise cigarette smoking and have tried multiple times to encourage my cousins to quit (once I made one cousin promise she would cut down one cigarette a week until she smoked none, because I’d seen that somewhere and it worked – it turned out she was just hiding her cigarettes and going down the path outside the house to smoke them. When I found out, I threw ALL her cigarettes in the bin, and threw water over them, just to be sure. That was almost a whole packet, I think. Perhaps $20 or so in those days?).

I digress.

I’ve tried to think of ways to make this happen for myself more easily.

I’ll admit, I’ve found the 20-25 mins or so of train commute (by the time I sit down and settle my bags, etc) to be a decent amount of time to write a few paragraphs about something I find interesting or that’s on my mind. It’s then turning it into publishing that’s the difficult part.

Perhaps if I tried a version of this 5 minute challenge from the Thesis Whisperer?

I’ve created a spotify playlist of instrumental songs that are approximately 5 minutes long (4:50 – 5:30 to be exact). The idea being I can use 5-10 minutes to write, 5 minutes to post.

I’m hoping – thinking that perhaps 15 minutes of almost-free-writing might work? I use evernote to capture all my notes, including interesting links, thoughts, snippets of blog ideas and drafting posts. I find it much easier to free-write on the computer sometimes. Other times, I use a notebook that I can then take a photo and save it up into Evernote and it gets scanned. It’s rather handy.

Again, I digress. It’s almost the end of 10 minutes now. I should summarise and have a point.

My point will be this: I’m going to, once again, try a new method to encourage reflective thinking in myself. And use this blog to do that.

Thanks Inger.

Head picture was selected from Unsplash, using the Unsplash Instant Chrome Extension. 

Learning that’s just-in-time to frustrate me

Last night some new strings arrived for my ukulele. I’ve only ever restrung a ukulele once (despite owning a zillion) so although I knew I was a complete n00b, I figured I ought to be able to work it out.

Last night someone in my house may have thrown a violent tantrum of frustration.

Don't tell me to be calm!!
That person may have been trying to change ukulele strings.

That person may have been me.

When faced with the task in which I am incredibly lacking in any knowledge or experience, i would turn either to someone knowledgeable nearby (in person or through PLN) or youtube. In this case, it was youtube. This was just a short manual activity, surely a video is going to be the best way to learn it. The situation, and my experience, made me think of the iterations of flipped classrooms and recorded lectures that I have experienced up until now – and why I’m still skeptical of the “down with lectures!” push that seems to be gaining traction.

In light of the recent conversations of the value of lectures (from many viewpoints), I wondered – what would have been the optimal way for me to learn this valuable, yet infrequently applied, knowledge?Read More »

Just because you can, doesn’t mean you should 

Or “The difference between being a Subject Matter Expert and being a teacher”

When I was in high school, I was Vice-Captain of the girl’s First XI cricket team. Not because I was especially good, in fact I was average-poor most days, but mostly because only 12 people turned up to tryouts. I was, however, pretty good at keeping spirits up on the field and also pretty good at translating and articulating coach’s instructions on how to execute a particular skill better, even if I couldn’t actually demonstrate it myself. I knew that when bowling you want your body to be almost side-on and the arm with the ball to make a wide arc with your upper arm grazing your ear as you step through.

When my friends struggled with algebra or calculus concepts, or how to interpret the wordy physics exercises or couldn’t wrap their heads around how to conjugate ‘venir’ v ‘ir’ in French, they didn’t often ask the teacher, they asked me. Whereas our teachers might simply repeat the formula or answer or just move on, I could come up with ways to explain or other examples that made more sense or different interpretations of the formula.

When I worked at a linen and haberdashery store during university, my colleagues who had worked there for years and knew the processes and products in and out struggled to work out how to impart that knowledge on me – preferring to make me stand and watch or rattling off step after step with no context.

All of which is to say, just because you know your shit, doesn’t mean you’re shit hot at how teach it to others.*Read More »

On winging it…

Unless you’ve worked with me, you may not know this, but I am the self-proclaimed Queen of “Winging It”. It’s the curse of procrastinators everywhere that what must eventually be done is done under duress extreme time pressure. Sooner or later, if you’re a bad learner not unlike me, that means throwing caution to the wind and just doing it.

The years I have spent honing this craft have resulted in an innate belief that I can do almost anything with enough pressure and confidence.

This weekend I was a headline performer at the Mandorah Ukulele and Folk Festival.

A few days out from the performance, I get the run sheet that says I’m to be onstage for up to 70 minutes (including setup – which takes all of 3 mins- and breakdown – which takes even less). This would present no problem, if I wasn’t used to and expecting a 40-45 minute set. Those extra 15-20 minutes means 4-5 songs plus preamble and potentially rearranging the order to maintain or recreate the flow. At the same time, I need to plan for contingencies such as playing faster than usual and getting through my songs quicker than expected, or the timing on the day running over and having to cut songs out with the call of “15 minutes to go”. Read More »

My kind of scaffold – part 2

Part 2 of the series of posts wherein I share the process I’ve used to try build a “different” scaffold for compliance-based courses.

Building wall of mismatching windows. By Edgaras Maselskis, via Unsplash. CC0
In a previous post, I spoke about how I used a combination of butchered action-mapping and Bloom’s taxonomy to create course objectives with actionable verbs and that provided more direct mapping to activities. Here, I want to share the course design and activity design approach, and why I think this goes beyond what many (especially in corporate/compliance training) believe constitutes ‘eLearning’ – the “Next” button.

One of the common pitfalls of a Learning Management System (Moodle, Blackboard Learn, Canvas) is that they don’t necessarily support the creation of learning ‘content’ that is typically expected in corporate environments. Instead, the majority of them support discrete activities that are collated together to form a learning experience rather than a coherent and simple flow*. Another common downfall arising out of using LMS’s is that, despite the built-in activities common in LMS’s, all too often they’re used to house individual resources (pdf files, URLs, word documents, videos) and then an in-built quiz to assess the learning. Having worked on both the vendor and the user sides of the LMS/edtech world, I would purport that the root of this both in the technology and the typical user – but that’s a conversation for another day.Read More »

My kind of scaffold – part 1

In the series of posts that follows, I want to share the process I’ve used to try build a “different”* scaffold for compliance-based courses.

Building facade, stairs and windows, shot from the bottom looking up.

After I wrote the post about my stuttering start to designing this course, I realised I had fallen in to an all-too-common trap of focusing on the activities first. Indeed, I was halfway committed to what type of content I would need to build, had made sweeping decision about what design approached use when I hadn’t yet confirmed what the purpose of the course was. Trying to explain my ideas and clarify the objectives sparked heated conversations in my team about what eLearning was and wasn’t and no clear agreement on learning outcomes.

I needed to take a step back, and focus on the bigger picture – what do we want our learners to achieve and how will we know that they’ve done it? What is the purpose for this course in the first place? From there, I was confident that we could reach an understanding on what the course would look like.Read More »