For my bachelor thesis I decided to work together with physicist, Jesper Bruun, on a project within physics education research. Jesper is also one of the authors behind a methodology for assessing response patterns in multiple choice inventories, called MAMCR. When applying this method on student responses from the Force Concept Inventory, we are ideally assessing the conceptions of physics within that group of students. For the interested reader, this is a brief summary of my bachelor thesis. If you want to read what I handed in, my thesis is published here.
In 1985 Hestenes, Wells, and Swackhamer gauged student understandings of physics phenomena by asking students to explain – conceptually – the physics occurring in various situations. Based on these interviews, they published an inventory in 1992 where for each question, the student is asked to distinguish the correct response item from several distractors. This inventory is called the Force Concept Inventory (FCI), and it has since then been widely used within physics education to assess teaching effects especially in introductory physics courses.
At the Niels Bohr Institute, the inventory is still distributed to first year students during their first physics courses which are on Newtonian mechanics. This data is what I worked with, kindly provided by professor Ian Bearden.
In 2017, Brewe, Bruun, and Bearden published an article that proposed a new method for identifying non-normative modules using network science on multiple choice responses. In short, the method consisted of creating a network of incorrect response items based on the student responses (by use of the bipartite projection on items), and identifying modules of responses using the clustering algorithm InfoMap on the “backbone” of the network which was extracted using the sparsification algorithm proposed here.
What I set out to do
For my project, we wondered if it was possible to include not only the incorrect response items but also the correct ones. The problem with including these response items was that they attracted all other response items to such a degree that it completely obscures the clustering analysis. However, it proved possible to only exclude some of the response items.
Reducing the network smartly
MAMCR is utilised on the item-item projection of the bipartite student-item network. Let me break it down for you. The kind of data we have is that each student connects to 30 response items like so:
Extending this to a larger number of students, we end up with a bipartite network where several students are connected to various response items depending on their choices. This is illustrated in the middle of the following picture. In network terms, we can then create a projection to either one of the sets of data such that items are connected through students, or students would be connected through items. The numbers in the item projection indicates the number of students that chose that connection. Such a network is called an undirected, weighted network, and this is what we will apply MAMCR to.
Next, we reduce the network using a sparsification algorithm, LANS (Locally Adaptive Network Sparsification), that evaluates the ‘importance’ of a connection for each response item in the network. This algorithm removes weak connections based on a significance level set by the researcher by locally comparing connections for each response item, and removing those that fall below the significance level. In the following example is how an adjacency matrix could change when LANS is applied. In my project, I chose to continue with the ‘lower’ diagonal as this keeps more connections than by using the ‘upper’ diagonal.
This results in a slightly reduced network. What I then proposed to do was to remove popular items from the network based on a frequency threshold. Choosing the right significance level and threshold frequency level were both up to the researcher. During most of the project, we chose to extract various information on each network, and manually tried to determine the most desirable parameter values based on a set of graphs. The computation time for this was extreme – we would set the frequency cutoff level first and extract values for a range of significance levels. We would repeat this for various threshold frequency levels with little variations. At the end of my project, I instead set the significance level very low (at 0.0001), and simply scanned over the threshold frequencies instead. This reduced computation time from hours or days to simply seconds. Below is an example of extracted values using the ‘new’ method.
After choosing the appropriate levels, we would run a clustering algorithm on the final network. For this project, we have chosen to use the InfoMap algorithm (available at http://www.mapequation.org). Using the item projection from before as an example, the algorithm might detect a modular solution with two modules:
Finally, we would end up with a modular solution to analyse. For one of the cohorts, the network below is one the modular solution I ended up with. Since then, I have continued with developing the method, and with investigating its validity and reliabilty.