Technical White Paper

Executive summary

AILAT is an adaptive assessment that measures AI literacy across four dimensions: conceptual knowledge, practical application, critical evaluation, and ethics. The test is 20 items long and completes in roughly 20 minutes.

During the adaptive phase, multiple-choice questions are selected using two-parameter logistic Item Response Theory concepts. The test updates a running proficiency estimate and chooses an item with high information at that estimate. Open-ended responses are scored against dimension-specific language-model rubrics. The final reported score is performance-based and is not an IRT theta estimate.

The result is a dimension-level profile, an AI Literacy Level from 1 to 5, and a set of learning recommendations tied to the user's weakest dimensions and industry context.

AILAT is designed to measure, not sort. The score is a starting point for a learning path — not a gate, not a rank, not a hiring signal.

§ 01

Four dimensions of AI literacy

AI literacy is not one skill. AILAT treats it as four, following the competency structure in Long & Magerko (2020) and later refinements by Ng et al. (2021) and Carolus et al. (2022).

Conceptual knowledge: How AI systems work — terminology, capabilities, limits, and the mechanisms behind modern models.
Use and application: Working with AI tools in practice — prompting, integration, and fitting AI into existing workflows.
Evaluation and creation: Judging AI output for accuracy, fit, and failure modes; building AI-assisted solutions on top of that judgment.
Ethics: Bias, privacy, transparency, and accountability — the responsible-use layer that sits across the other three.

§ 02

Adaptive-delivery model

AILAT uses Item Response Theory (IRT) concepts during adaptive delivery. In the model, each question has authored parameters and each respondent has a running proficiency estimate θ. The model predicts the probability that a respondent will answer a given question correctly.

AILAT uses the two-parameter logistic (2PL) model, assigning each item an authored difficulty (b, ranging −3 to +3) and discrimination (a, ranging 0.5 to 2.5). These values support item selection; they have not yet been estimated through an AILAT-specific empirical calibration study.

P(θ)=11 + e^{−a(θ − b)}

Two-parameter logistic model · probability of correct response

Unlike classical test theory, IRT gives us a natural measure of how much information a question provides at a specific proficiency level. That information function is what makes adaptive testing possible — we can pick the next question to ask by maximizing information at the current θ.

§ 03

Assessment flow

The test runs in three phases. Each phase has a distinct job, and the phase boundaries let us combine breadth, depth, and contextual reasoning without making the whole test adaptive from the first question.

Phase 1 · Calibration

5 questions · fixed, mixed difficulty

One question from each dimension plus a fifth balancer, used to establish an initial proficiency estimate before adaptation begins. “Calibration” is the product’s phase label; it does not mean the item bank has undergone empirical psychometric calibration.

Phase 2 · Adaptive

10 questions · 8 IRT-selected MC + 2 open-ended

Each multiple-choice item is chosen to maximize information at the running proficiency estimate. Two open-ended prompts probe reasoning that multiple-choice cannot capture.

Phase 3 · Scenario

5 questions · industry-contextualized

Four multiple-choice items and one open-ended capstone, framed around a scenario chosen from the respondent's stated industry.

Proficiency is re-estimated after every answer, and respondents are assigned to one of three adaptive tracks. Foundational (θ ≤ −1.0) emphasizes easy items and fuller explanations. Standard (−1.0 < θ < 1.0) mixes difficulty evenly. Advanced (θ ≥ 1.0) prioritizes medium and difficult items to tighten the estimate at the top of the scale.

§ 04

Adaptive algorithms

Two small algorithms do most of the work: selecting the next question, and updating the proficiency estimate after each answer.

Question selection. From the pool of candidate items matching the current track and the weakest dimension, pick the one that maximizes Fisher information at the current θ.

Listing 1 — Maximum-information question selection

function findMaxInfoQuestion(questions, proficiency) {
  return questions.reduce((best, question) => {
    const info = calculateInformation(
      question.IRT_parameters.discrimination,
      question.IRT_parameters.difficulty,
      proficiency,
    );
    return !best || info > best.info ? { question, info } : best;
  }, null).question;
}

function calculateInformation(a, b, theta) {
  const p = calculateProbability(a, b, theta);
  return a ** 2 * p * (1 - p);
}

function calculateProbability(a, b, theta) {
  return 1 / (1 + Math.exp(-a * (theta - b)));
}

Proficiency update. After each response, update θ using a bounded gradient step toward the maximum-likelihood direction. The step size is intentionally small (0.4) so that a single lucky or unlucky answer doesn't swing the estimate.

Listing 2 — Gradient-step proficiency update

function updateProficiency(theta, a, b, isCorrect) {
  const stepSize = 0.4;
  const p = calculateProbability(a, b, theta);
  const gradient = a * ((isCorrect ? 1 : 0) - p);
  return Math.max(-3, Math.min(3, theta + stepSize * gradient));
}

Open-ended evaluation. Each open-ended response is scored by an LLM against dimension-specific guidance combined with per-item criteria — conceptual accuracy for knowledge items, practical application for use-and-apply items, and critical analysis for evaluation items. The per-item rubric is not published, which prevents respondents from optimizing for the scorer rather than demonstrating understanding.

§ 05

Scoring and learning paths

The final profile combines multiple-choice correctness with fractional credit from rubric-scored open-ended responses, including the scenario capstone. These produce dimension percentages. The overall 0–100 score weights Conceptual and Use & Apply at 30% each, Evaluate & Create at 25%, and Ethics at 15%; score bands map to an AI Literacy Level from 1–5.

1
Baseline awareness
Recognizes AI as a category. Can use a tool with step-by-step guidance. Limited ability to judge output.
2
Informed user
Uses AI tools for practical tasks. Identifies obvious errors and hallucinations.
3
Capable practitioner
Integrates AI into workflows. Evaluates output systematically. Understands core concepts like tokens, context, and fine-tuning.
4
Critical evaluator
Reasons about bias, failure modes, and appropriate use. Designs AI solutions with safeguards.
5
AI champion
Shapes strategy around AI capability. Can teach others. Engages with the frontier of the field.

Each level has a matching recommendation template: resource types, depth, and progression strategy tailored to where the respondent starts. The path is further filtered by dimension weakness and stated industry.

§ 06

Ethics and operating limits

The Ethics dimension samples bias, privacy, transparency, and accountability. In the standard enterprise flow, participants are not asked for a name or email address; the assessment records configured setup categories, answers, and scoring data needed for the report. Individually purchased access uses an email address for payment and access delivery.

Session state persists across interruptions so respondents can resume. If automated open-ended evaluation is unavailable, the current fallback assigns neutral partial credit with generic feedback. That keeps the assessment moving, but it reduces interpretability and is not separately identified in the participant report.

The AI Literacy Level is a descriptive classification of current response performance. It is not a hiring signal, a proxy for job fitness, or a ranking of respondents against each other.

§ 07

Limitations

AILAT measures knowledge and reasoning about AI, not hands-on execution with any specific tool. A respondent who scores at Level 4 can reason about model limitations; they may or may not be fluent in any particular API.

The field moves faster than any fixed item bank. Maintaining content relevance therefore requires documented review and versioning as model capabilities and common work practices change.

AILAT has not yet published instrument-specific reliability, model-fit, test–retest, criterion-validity, fairness, or human–model scoring-agreement evidence. It has no population norms. Until those studies exist, results should be used as a formative diagnostic rather than a high-stakes measure.

See the current method and validation status for the evidence ledger and the work still required.

References

01Carolus, A., Koch, M., Straka, S., Latoschik, M. E., & Wienrich, C. (2023). MAILS — Meta AI Literacy Scale: Development and Testing of an AI Literacy Questionnaire.
02Cetindamar, D., Kitto, K., Wu, M., Zhang, Y., Abedin, B., & Knight, S. (2022). Explicating AI literacy of employees at digital workplaces. IEEE Transactions on Engineering Management.
03Ding, L., Kim, S., & Allday, R. A. (2024). Development of an AI literacy assessment for non-technical individuals. Contemporary Educational Technology, 16(3), ep512.
04Hornberger, M., Bewersdorff, A., & Nerdel, C. (2023). What do university students know about Artificial Intelligence? Computers and Education: Artificial Intelligence, 5.
05Long, D., & Magerko, B. (2020). What is AI Literacy? Competencies and Design Considerations. CHI 2020, 1–16.
06Ng, D. T. K., Leung, J. K. L., Chu, S. K. W., & Qiao, M. S. (2021). Conceptualizing AI literacy: An exploratory review. Computers and Education: Artificial Intelligence, 2.
07Wang, B., Rau, P. L. P., & Yuan, T. (2022). Measuring user competence in using artificial intelligence. Behaviour & Information Technology.