**Truncated Natural Policy Gradient** (TNPG) is the algorithm for computing gradient direction. It only needs to calculate $I(\theta)v$ where:

1. $I (\theta)$ is Fisher Information Matrix
2. v is arbitrary vector.

TNPG improves natural policy gradient but it has a high computational cost. Usually, they are efficient when the space of the parameters is high dimensional. 

San Diego State University

1. Models of Human Memory
2. Spaced Repetition
3. Leitner System
4. Supermemo System
5. Intelligent Tutoring System
6. Relation between Tutoring Systems and Student Learning
7. Reinforcement Learning
8. Trust Region Policy Optimization
9. Truncated Natural Policy Gradient
10. Recurrent Neural Networks

Related Theory (Using deep reinforcement learning for personalizing review sessions on e-learning platforms with spaced repetition)

Here the authors describe three previous probabilistic methods of the human memory:

**Exponential Forgetting Curve**  - one of the oldest human memory models.
The authors use parameterization where user either forgets information completely or retains it and the probability of the recall can be expressed with this function:  

"where θ ∈ R+ is the item difficulty, D ∈ R+ is the time elapsed since the item was last reviewed by the student, and S ∈ R+ is the student’s memory strength for the item." S is set to the amount of trials. 

                                       $$Z ~ Bernoulli(exp(-\theta \frac {D}{S}))$$
**Half-Life Regression** - here unlike **exponential forgetting curve** we don't have the parameter for the difficulty here, instead here we have ~x ∈ X which contains information about student's study history and model parameters θ ∈ Θ . This is how formula looks like:
                                 $$ S = exp(\overrightarrow { \rm \theta} \overrightarrow {\rm x})$$

In order to encode the number of attempts, correct/incorrect answers and the identity  the authors set $X = \N ^{3} \times {0, 1} ^{n}$. By dropping difficulty of the item we are not losing any information as the difficulty "is absorbed into the memory strength via the coefficients of the item identity indicator features."

**Generalized Power Law** (paper describes this part precisely, couldn't be broken down into smaller parts) here we have different likelihood for the recall:


On the last formula from the picture we have a and d  which are parameters for the student ability and the item difficulty.


Background
(Accelerating Human Learning With Deep Reinforcement Learning)

Reviewing or practicing over time can have better results than reviewing in short periods of time.

Spaced Repetition

The method used in flashcards for the spaced repetition. 

**The Goal** focus more on those items with which students have more difficulties to recall and spend less time on the ones that student recalls.

Leitner System

This learning approach trains an agent by letting it interact with an environment and observe the consequences of its actions. After each action, the agent receives feedback in the form of rewards or penalties, then adjusts its behavior to improve future outcomes. The objective is to choose actions that maximize long-term reward, not just the next immediate payoff. This is especially useful for tasks where decisions affect later results, such as game playing, robot control, or scheduling.

Learning from Rewards and Penalties

TRPO is an optimization algorithm in reinforcement learning which uses gradient descent. TRPO builds an algorithm that is stable and guarantees monotonic improvement. "This algorithm is similar to natural policy gradient methods and is effective for optimizing large nonlinear policies such as neural networks"

TRPO has  better performance than the vanilla policy gradients as the length of the step size is easily defined. Additionally, it takes advantage of using old policies sampled distributions for optimizing new ones.



Trust Region Policy Optimization

Recurrent neural networks, or RNNs, are a family of neural networks for processing sequential data. A recurrent neural network is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time add additional layers of comprehension on top of the previous inputs. 


Recurrent Neural Network (RNN)

The SuperMemo 2 (SM-2) algorithm calculates intervals for spaced repetition learning. The steps are:

1. Split knowledge into distinct items.
2. Initialize E-Factor (easiness of memorizing) to 2.5 for all items.
3. Repeat items using the following intervals in days, where $$I(n)$$ is the interval after the $$n$$-th repetition:
- $$I(1) := 1$$
- $$I(2) := 6$$
- For $$n > 2$$: $$I(n) := I(n-1) \times EF$$
4. Assess the quality of the response ($$q$$) on a 0-5 scale: 5 (perfect), 4 (correct after hesitation), 3 (correct but difficult), 2 (incorrect, but correct answer seemed easy to recall), 1 (incorrect, but correct answer remembered), 0 (complete blackout).
5. Update the E-Factor using the formula: $$EF_{new} := EF + (0.1 - (5-q) \times (0.08 + (5-q) \times 0.02))$$. If $$EF_{new} < 1.3$$, set $$EF_{new} = 1.3$$.
6. If $$q < 3$$, repeat the item without changing the E-Factor.
7. Repeat the session until a grade of at least $$q = 4$$ is achieved for all items.

SuperMemo 2 (SM-2) Algorithm

An intelligent tutoring system combines artificial intelligence, cognitive science, and educational theory to support adaptive instruction. Its architecture comprises four modules: (1) the expert-knowledge module, which supplies domain knowledge and supports the generation of questions, answers, and solutions; (2) the student-model module, which tracks the learner's changing knowledge and skills; (3) the tutor module, which uses the student model to select a teaching strategy and determine what to present next; and (4) the user-interface module, through which the system and learner interact.

Learn Before

Related