Module 2: Logistic Knowledge Tracing and Performance Factors Analysis

KT Learning Lab 2: A Conceptual Overview

Logistic Knowledge Tracing

A broad framework for knowledge tracing models based on logistic regression Pavlik, Eglington, and Harrell-Williams (2021)

Speaker Notes:

Logistic knowledge tracing is a broad framework for knowledge tracing models based on logistic regression formulas.

Performance Factors Analysis

First member of the LKT family that ran in real-time Pavlik Jr, Cen, and Koedinger (2009)

Speaker Notes:

The first member of the LKT family that ran in real time was Performance Factors Analysis (Pavlik et al., 2009).

PFA

• Measures how much latent skill a student has, while they are learning

  • But expresses it in terms of probability of correctness, the next time the skill is encountered

  • No direct expression of the amount of latent skill, except this probability of correctness

Speaker Notes:

PFA measures how much latent skill a student has, while they’re learning. But expresses it in terms of probability of correctness, the next time the skill is encountered. Within PFA, unlike BKT, there’s no direct expression of the amount of latent skill, except the probability of correctness.

What is the typical use of PFA?

• Assess a student’s knowledge of topic X

• Based on a sequence of items that are dichotomously scored

  • E.g. the student can get a score of 0 or 1 on each item

• Where the student can learn on each item, due to help, feedback, scaffolding, etc.

Speaker Notes:

What’s the typical use of PFA?

• Assess a student’s knowledge of topic X, based on a sequence of items that are dichotomously scored

  • e.g., the student can get a score of 0 or 1 on each item

• Where the student can learn on each item due to a variety of things:

  •  help

  •  feedback 

  •  scaffolding, and so on

How does PFA differ from BKT?

Speaker Notes:

How does PFA differ from BKT, some key assumptions:

Key assumptions

• Each item may involve multiple latent skills or knowledge components

  • Different from BKT

• Each skill has success learning rate γ and failure learning rate ρ

  • Different from BKT where learning rate is the same, success or failure

Speaker Notes:

• Each item may involve multiple latent skills or knowledge components

  • This is a big difference from BKT. In BKT, there is only one skill per item, and that means that you can only use BKT in situations where you’ve designed the curriculum, so there’s only one relevant skill with a chance of being hard at any given time

  • PFA doesn’t make this assumption, thus it can be used a little more widely

• Also, in PFA, each skill has a success learning rate — gamma, and a failure earning rate — rho, which is different from BKT, where the learning rate is the same, success or failure

Key assumptions

• There is also a difficulty parameter β, but its semantics can vary – more on this later

• From these parameters, and the number of successes and failures the student has had on each relevant skill so far, we can compute the probability P(m) that the learner will get the item correct

Speaker Notes:

• PFA also has a difficulty parameter: beta, but its semantics can vary

• From these parameters, and the number of successes and failures that the student has had on each relevant skill so far, we can compute the probability P(m) that the learner will get the item correct

PFA

Speaker Notes:

PFA can be expressed mathematically as follows(Next page)

Let’s go over what each of these parameters means

Speaker Notes:

• First, you compute m, m is a function of beta — the difficulty, and the sum of the number of the successes and failures with their gamma and rho that students have had on every relevant skill in the problem, take that m, and then you put it through an exponential function

Reasonable Example

γ = 0.2, ρ = 0.1, β = -0.5

Actual	m	P(m)
	-0.5	0.38

Speaker Notes:

• You have a gamma of 0.2 and a rho of 0.1, we’re assuming that you improve your future performance double if you get success compared to if you get a failure, and the beta is negative 0.5, which means the item is a little hard. 

• The initial M, there are no successes or failures yet, is going to be negative 0.5. The probability that the student will get it right on the first attempt is 0.38. At this point, we have no information about the student.

Reasonable Example

γ = 0.2, ρ = 0.1, β = -0.5

Actual	m	P(m)
0	-0.5	0.38
	-0.5+(0.1)*1

Speaker Notes:

• Let’s say the student gets it wrong, now we update for the next opportunity m, and we take a negative 0.5, item difficulty is the same here. We add for that failure 0.1, times 1 failure so far,

Reasonable Example

γ = 0.2, ρ = 0.1, β = -0.5

Actual	m	P(m)
0	-0.5	0.38
	-0.4	0.40

Speaker Notes:

• and this gives us negative 0.4. 

  • So even though they got it wrong, we’re saying they still learned more than the evidence of them being wrong told us they won’t get it.

Reasonable Example

γ = 0.2, ρ = 0.1, β = -0.5

Actual	m	P(m)
0	-0.5	0.38
0	-0.4	0.40

Speaker Notes:

In fact, the actual 0,

Reasonable Example

γ = 0.2, ρ = 0.1, β = -0.5

Actual	m	P(m)
0	-0.5	0.38
0	-0.4	0.40
	-0.5+(0.1*2)

Speaker Notes:

• in this case, the m will be negative 0.5 plus 0.1 times 2. We’ve got two failures, which are 0.1 for rho. Negative 0.3 is the result,

Reasonable Example

γ = 0.2, ρ = 0.1, β = -0.5

Actual	m	P(m)
0	-0.5	0.38
0	-0.4	0.40
	-0.3	0.43

Speaker Notes:

Which gives us a P(m) of 0.43, and the actual is 1.

Reasonable Example

γ = 0.2, ρ = 0.1, β = -0.5

Actual	m	P(m)
0	-0.5	0.38
0	-0.4	0.40
1	-0.3	0.43
	-0.5+(0.1*2)+(0.2*1)

Speaker Notes:

• Now they have two failures and one success, the function is just a little more complex. It’s negative 0.5 the beta, plus 2 failures and 1 success,

Reasonable Example

γ = 0.2, ρ = 0.1, β = -0.5

Actual	m	P(m)
0	-0.5	0.38
0	-0.4	0.40
1	-0.3	0.43
	-0.1	0.48

Speaker Notes:

• which gives us a probability of m, a 0.48, and it’s kind of going up over time.

How Does PFA

• Represent when the student learns from an opportunity to practice?

• As opposed to just better predicted performance because you’ve gotten it right 

Speaker Notes:

Please follow the slides

How Does PFA

• Represent when the student learns from an opportunity to practice?

• As opposed to just better predicted performance because you’ve gotten it right

• Is it ρ ?

• Is it average of ρ and γ?

Speaker Notes:

Please follow the slides

Degeneracy in PFA Maier, Baker, and Stalzer (2021)

• Three degenerate cases

  • γ < 0

  • γ < ρ

  • γ = ρ = 0

Speaker Notes:

PFA can have degenerate models, and Maier et al., (2021) talk about three degeneracy cases that you can get, where gamma is less than 0, or gamma is less than rho, and where both gamma and rho equals 0.

What do each of these mean?

• When might you legitimately get them?

• ρ < 0

• γ < ρ

• γ < 0

Speaker Notes:

Degeneracy in PFA Maier, Baker, and Stalzer (2021)

• Three degenerate cases

  • γ < 0

  • γ < ρ

  • γ = ρ = 0

• One seemingly degenerate (but not) case

  • ρ > 0

• “It is worth noting that a fourth case when ρ > 0 – is not degenerate, due to the multiple functions the parameters perform in PFA. In this case, the rate of learning the skill may outweigh the evidence of lack of student knowledge that an incorrect answer provides. So long as γ > ρ, a positive ρ is conceptually acceptable.”

Speaker Notes:

There’s also a seemingly degenerate case that isn’t where rho is greater than 0, and Maier and her colleagues talk about this:

• In this 4th case, there’s no degeneracy because of the multiple functions the parameters perform in PFA. In this specific case, what might be happening is that the rate of learning the skill might be bigger than the evidence of lack of student knowledge, that an incorrect answer provides

• So even though rho is greater than zero, you get better when you get it wrong seems implausible, it isn’t necessarily degenerate for that reason. As long as the improvement associated with getting it right is greater than the probability associated with getting it wrong.

Degenerate Example (Case 1)

γ = -0.1, ρ = -0.5, β = -0.5

Actual	m	P(m)
0	-0.5	0.38
0	-1	0.27
1	-1.5	0.18
	-1.6	0.17

Speaker Notes:

First degenerate case:

• Where gamma is less than zero, which means that if you get it right, your predicted future performance is worse

• This student got it wrong the first two times and their predicted performance drops, we should expect that, but they got it right the third time, and their predicted performance still drops from 18% correct to 17% correct. The drop is not as extreme for right is wrong, but still, they get it right and the system thinks they’re doing worse, and this is degenerate within the PFA framework  

Degenerate Example (Case 2)

γ = 0.1, ρ = 0.2, β = -0.5

Actual	m	P(m)
0	-0.5	0.38
0	-0.3	0.43
1	-0.1	0.48
	0	0.5

Speaker Notes:

An example of degenerate case two:

• In this case, rho is greater than gamma, which means you get better if you get it wrong than if you get it right. What you can see here is the student gets it wrong a couple of times, and they’re probably getting it right on the next one, which goes up more than when they get it right

Note

• Values of ρ below 0 don’t actually mean negative learning

• They mean that failure provides more evidence on lack of knowledge

• Than the learning opportunity causes improvement

Speaker Notes:

Another thing to note:

• Values of rho below 0 don’t mean negative learning, it’s not like the student is unlearning

• What they mean is that failure provides more evidence of student’s lack of knowledge than learning opportunities, which causes improvement 

  • That’s why it’s not degenerate to have rho above zero

Addressing Degeneracy Maier, Baker, and Stalzer (2021)

• Simply bound γ and ρ

• Does not reduce model performance substantially (just like BKT)

• What causes degeneracy? We’ll come back to this in a minute

Speaker Notes:

How do you address degeneracy? 

• There’s a simple approach in Maier et al., (2021), which is simply bound rho and gamma, just like BKT, that doesn’t substantially reduce model performance

Note

• Parameters in PFA combine information from correctness with improvement from practice improvement

• Makes PFA models a little harder to interpret than BKT

Speaker Notes:

Parameters in PFA combine information from correctness with improvement from practice improvement. This makes PFA models a little harder to interpret than BKT. 

Adjusting β

γ = 0.2, ρ = 0.1, β = -0.5  

Actual	m	P(m)
0	-0.5	0.38
0	-0.4	0.40
1	-0.3	0.43
	-0.1	0.48

Speaker Notes:

How about adjusting the beta?

• In this case, we’ve got a beta of negative 0.5.

Adjusting β

γ = 0.2, ρ = 0.1, β = -1.5

Actual	m	P(m)
0	-1.5	0.18
0	-1.4	0.20
1	-1.3	0.21
	-1.1	0.25

Speaker Notes

If we change it to negative 1.5, the same amount of improvement happens, but they start from a lower baseline.

Adjusting β

γ = 0.2, ρ = 0.1, β = +3.0 

Actual	m	P(m)
0	3.0	0.953
0	3.1	0.957
1	3.2	0.961
	3.4	0.968

Speaker Notes

• If we set beta to plus 3, then it starts with a really high baseline from the very start, we assume a student’s is 95.3% correct

• Incidentally, if we think of students being 95.3% correct, the problem is relatively unlikely that you want to give the student this item to study unless there’s some other good pedagogical reason

  • Maybe it’s a step that you have to do to get to another step, and it’s just too inconvenient to design a way that skips it

β Parameters

• Pavlik proposes three different β Parameters

  • Item

  • Item-Type

  • Skill

• Result in different number of parameters

  • And greater or lesser potential concern about over-fitting

• What are the circumstances where you might want item versus skill?

Speaker Notes:

For beta parameters, Pavlik proposes 3 different beta parameters:

1. Item 

2. Item-type 

3. Skill

Is difficulty something that each item has difficulty? Is difficulty something that each skill has difficulty? Or are there kinds of items somewhere in between? These 3 approaches result in different numbers of parameters and greater or lesser potential concerns about overfitting. The more parameters — the more risk of overfitting.

Causes of Degeneracy Maier, Baker, and Stalzer (2021)

• If β is used at the Skill or Item-Type level

• And the learning system moves students from easier to harder items within a “skill”

• Then γ < 0. 

• Also, if items are tagged with multiple skills, shared variance (collinearity) between skills could produce degenerate parameters.

Speaker Notes

What causes degeneracy? 

• Beta is a big part of what causes degeneracy in PFA, and LKT in general

• If the beta is used for the skill or item type level, and the learning system moves students from easier to harder items within a skill, then you’re going to get gamma being less than 0, because the systems quietly assign harder items over time

• Also, if items are tagged with multiple skills, then shared variants(collinearity) between skills could produce degenerate parameters

Fitting PFA

• Typically Expectation Maximization is used

Speaker Notes:

How do we fit PFA?

• Unlike BKT where there are several alternatives that people consider

• Typically for PFA, Expectation Maximization is used

Expectation Maximization

1. Starts with initial values for each parameter

2. Estimates student correctness at each problem step

3. Estimates params using student correctness estimates

4. If goodness is substantially better than last time it was estimated, and max iterations has not been reached, go to step 2

Speaker Notes

Expectation Maximization (EM):

1. Starts with initial values for each parameter, which are typically arbitrary 

2. Estimate students’ correctness at each problem step or item based on those values  

3. Then re-estimate the parameters using the student correctness estimates 

4. If the goodness of the model is substantially better than the last time it was estimated, and max iterations have not been reached, go back to step 2

5. Keep going until you don’t get any better or you get to a point where you’ve gone through your maximum number of tries  

Expectation Maximization

• EM is vulnerable to local minima

• Randomized restart typically used

Speaker Notes:

EM is vulnerable to local minima, which means sometimes you get to a parameter base where there’s a much better one somewhere else, but you can’t get to it from where you are. To try to avoid that, the randomized restart is typically used.

Is PFA better than BKT?

• Approximately equal predictive power across a lot of studies Pavlik Jr, Cen, and Koedinger (2009), Gong, Beck, and Heffernan (2010); Baker, Gowda, and Corbett (2011); Pardos et al. (2011); Pardos et al. (2012)

• Different virtues and flaws – choose the one that better fits your goals

Speaker Notes:

Is PFA better than BKT?

• It turns out that they have approximately equal predictive power across a lot of studies

• Different virtues and flaws for the two algorithms, choose the one that better fits your goals

Is PFA used in the real world?

• Yes, but by far fewer learning systems than BKT

• Maier, Baker, and Stalzer (2021) discuss its use in Reveal Math 1

Speaker Notes:

Second key question: Is PFA used in the real world? 

• Yes, but far less often than BKT

• Maier et al., (2021), for example, discuss its use in Reveal Math 1. But compared to BKT, which is used in dozens of systems, PFAs are relatively uncommon algorithms

Using PFA in the real world

• One issue in real-world use is handling rare skills, which can impact model inferences on common skills as well

  • Because PFA is used in cases with items tagged to multiple skills

• Maier, Baker, and Stalzer (2021) handle this by creating a “catch all” skill for rare skills

• Using average parameters from all common skills also works

Speaker Notes

One issue in real-world use:

• How it handles rare skills, which can impact model inferences on common skills as well, because PFA is typically used in cases with items tagged to multiple skills. Ironically, some of the packages for PFA can’t handle that. That’s the main benefit the PFA has over BKT

• When you have an item tagged to multiple skills, a rare skill is going to impact the common skills it intersects with. Maier et al., (2021) handled this by creating a catch-all skill for rare skills, and they found that using average parameters from all the common skills also works 

Final Thoughts on PFA

• PFA is a competitor for measuring student skill, which predicts the probability of correctness rather than latent knowledge
  

• Can handle multiple KCs for the same item, a big virtue

Speaker Notes:

Final thoughts on PFA:

PFA is a competitor for measuring student skill, which predicts the probability of correctness rather than latent knowledge. It can handle multiple cases for the same item in a graceful way, which is a big virtue.

Beyond PFA

•  Can we improve PFA based on what the field knows about learner memory?

Speaker Notes

Please follow the slide

PFA-Decay Gong, Beck, and Heffernan (2011)

• Weights actions further back in order less strongly

• Adds an evidence decay parameter δ

• Substitutes

• For the previous summation

• Very slightly higher AUC (0.003)

Speaker Notes:

An early major extension to PFA was:

• PFA-Decay by Gong et al., (2011) and her colleagues. PFA-Decay weights actions further back in order less strongly, and it adds evidence to the K parameter — sigma, substituting an equation that uses evidence decay for the previous summation

• Is it any better? Not really, historically important people don’t use it today

R-PFA Galyardt and Goldin (2014)

• Weights actions further back in order less strongly

• Looks at proportion of success-failure, weighting by distance in order from current action Adds an evidence decay parameter b Adds “ghost practices” before current practice to make math work

• Substitutes

• For the previous summation

• A little higher AUC (0.003-0.027) Pavlik Jr, Eglington, and Zhang (2021)

Speaker Notes:

The next extension was R-PFA by Galyardt & Goldin (2014): 

• R-PFA also weights actions further back in order less strongly 

• Looks at the proportion of success and failure, weighting by the distance in order from the current action

• It adds an evidence decay parameter b

• Adds “ghost practices” before the current practices to make the math work

• Substitutes a much more complex equation for the previous summation and doesn’t get a big difference over the original PFA

LKT Pavlik, Eglington, and Harrell-Williams (2021)

Creates a general framework for variants of PFA

Speaker Notes:

Building on this and some other work that had been going on in his lab at the time, Pavlik et al., (2021) and his colleagues created a general framework for variants of PFA.

Test

Speaker Notes

Here’s a table. You can see many variants, and there are many things you can do to modify PFA. 

LKT Pavlik, Eglington, and Harrell-Williams (2021)

• Ongoing work on variants to PFA typically frames itself in terms of LKT components (and proposes additional components)

• Examples

  • Fluctuation in response time Chu and Pavlik Jr (2023)

  • Different models of memory decay and spacing effect Maier et al. (2023)

Speaker Notes

The biggest contribution that LKT probably made was:

• New and ongoing work on variants to PFA now typically frame themselves in terms of LKT  components (and propose additional components)

  • Example: instead of getting people in a space where every single little tweak to PFA became a newly named algorithm, now we can just say LKT components 8 and 10

When to use LKT

• Some items have multiple skills

• Learning likely to be gradual rather than sudden

• Relatively small amounts of data

• You want to add new items without refitting the model

Speaker Notes:

Discussing the slides content about when to use LKT

References

Baker, Ryan SJ d, Sujith M Gowda, and Albert T Corbett. 2011. “Towards Predicting Future Transfer of Learning.” In Artificial Intelligence in Education: 15th International Conference, AIED 2011, Auckland, New Zealand, June 28–July 2011 15, 23–30. Springer.

Chu, Wei, and Philip I Pavlik Jr. 2023. “The Predictiveness of PFA Is Improved by Incorporating the Learner’s Correct Response Time Fluctuation.” International Educational Data Mining Society.

Galyardt, April, and Ilya Goldin. 2014. “Recent-Performance Factors Analysis.” In Educational Data Mining 2014.

Gong, Yue, Joseph E Beck, and Neil T Heffernan. 2010. “Comparing Knowledge Tracing and Performance Factor Analysis by Using Multiple Model Fitting Procedures.” In Intelligent Tutoring Systems: 10th International Conference, ITS 2010, Pittsburgh, PA, USA, June 14-18, 2010, Proceedings, Part i 10, 35–44. Springer.

———. 2011. “How to Construct More Accurate Student Models: Comparing and Optimizing Knowledge Tracing and Performance Factor Analysis.” International Journal of Artificial Intelligence in Education 21 (1-2): 27–46.

Maier, Cristina, Ryan S Baker, and Steve Stalzer. 2021. “Challenges to Applying Performance Factor Analysis to Existing Learning Systems.” In Proceedings of the 29th International Conference on Computers in Education.

Maier, Cristina, Isha Slavin, Ryan S. Baker, and Steve Stalzer. 2023. “Studying Memory Decay and Spacing Within Knowledge Tracing.” In. https://api.semanticscholar.org/CorpusID:259935239.

Pardos, Zachary A, Sujith M Gowda, Ryan Shaun Joazeiro de Baker, and Neil T Heffernan. 2011. “Ensembling Predictions of Student Post-Test Scores for an Intelligent Tutoring System.” In EDM, 189–98.

Pardos, Zachary A, Sujith M Gowda, Ryan SJd Baker, and Neil T Heffernan. 2012. “The Sum Is Greater Than the Parts: Ensembling Models of Student Knowledge in Educational Software.” ACM SIGKDD Explorations Newsletter 13 (2): 37–44.

Pavlik Jr, Philip I, Hao Cen, and Kenneth R Koedinger. 2009. “Performance Factors Analysis–a New Alternative to Knowledge Tracing.” Online Submission.

Pavlik Jr, Philip I, Luke G Eglington, and Liang Zhang. 2021. “Automatic Domain Model Creation and Improvement.” Grantee Submission.

Pavlik, Philip I, Luke G Eglington, and Leigh M Harrell-Williams. 2021. “Logistic Knowledge Tracing: A Constrained Framework for Learner Modeling.” IEEE Transactions on Learning Technologies 14 (5): 624–39.