Logistic Regression Example Manual Calculation

Comprehensive Guide to Logistic Regression Manual Calculation

Logistic regression is a fundamental statistical method for binary classification problems, where the outcome variable is categorical (typically 0 or 1). Unlike linear regression which predicts continuous values, logistic regression estimates probabilities using the logistic function (sigmoid function). This guide provides a step-by-step explanation of how to perform logistic regression calculations manually, along with practical examples and interpretations.

1. Understanding the Logistic Regression Model

The logistic regression model predicts the probability that an observation belongs to a particular class. The core equation is:

P(Y=1) = 1 / (1 + e^{-(β₀ + β₁X)})

Where:

• P(Y=1): Probability that the dependent variable equals 1
• β₀: Intercept term (constant)
• β₁: Coefficient for the predictor variable
• X: Predictor variable value
• e: Base of natural logarithms (~2.71828)

2. Step-by-Step Calculation Process

1. Calculate the linear combination: Compute β₀ + β₁X (also called log-odds or logit)
   • This represents the log of the odds that Y=1
   • Example: If β₀ = -2.5, β₁ = 1.2, and X = 3, then linear combination = -2.5 + (1.2 × 3) = 1.1
2. Convert log-odds to probability: Apply the logistic function
   • Use the formula: 1 / (1 + e^-z) where z is the linear combination
   • For our example: 1 / (1 + e^-1.1) ≈ 0.7503 or 75.03%
3. Classify the observation: Compare probability to threshold
   • Default threshold is 0.5 (can be adjusted based on problem context)
   • If P(Y=1) ≥ threshold → Class 1
   • If P(Y=1) < threshold → Class 0
4. Calculate odds ratio: e^β₁
   • Represents how the odds change with a one-unit increase in X
   • For β₁ = 1.2: OR = e^1.2 ≈ 3.32
   • Interpretation: Each unit increase in X multiplies the odds by 3.32

3. Practical Example with Real Data

Let’s work through a complete example using medical data where we predict the probability of a patient having a disease (1) or not (0) based on their age.

Parameter	Value	Description
Intercept (β₀)	-4.077	Baseline log-odds when age=0
Coefficient (β₁)	0.111	Change in log-odds per year of age
Predictor (Age)	45	Patient’s age in years
Threshold	0.5	Classification cutoff probability

Step 1: Calculate linear combination

z = β₀ + β₁X = -4.077 + (0.111 × 45) = -4.077 + 4.995 = 0.918

Step 2: Calculate probability

P(Y=1) = 1 / (1 + e^-0.918) ≈ 0.715 or 71.5%

Step 3: Classification

Since 0.715 > 0.5, we classify this patient as having the disease (Class 1)

Step 4: Odds ratio interpretation

OR = e^0.111 ≈ 1.117

Each additional year of age increases the odds of having the disease by about 11.7%

4. Model Evaluation Metrics

After performing calculations, it’s important to evaluate model performance using these key metrics:

Metric	Formula	Interpretation	Good Value
Accuracy	(TP + TN) / (TP + TN + FP + FN)	Overall correctness of predictions	> 0.8 for most problems
Precision	TP / (TP + FP)	Proportion of positive identifications that were correct	> 0.7 for imbalanced data
Recall (Sensitivity)	TP / (TP + FN)	Proportion of actual positives correctly identified	> 0.7 for medical tests
F1 Score	2 × (Precision × Recall) / (Precision + Recall)	Harmonic mean of precision and recall	> 0.7 for balanced metrics
ROC AUC	Area under ROC curve	Model’s ability to distinguish classes	> 0.8 for good discrimination

5. Common Pitfalls and Solutions

1. Complete separation

   Problem: When a predictor perfectly predicts the outcome, coefficients become infinite

   Solution: Use Firth’s penalized likelihood or combine categories

2. Multicollinearity

   Problem: Highly correlated predictors inflate coefficient variances

   Solution: Remove correlated predictors or use regularization

3. Overfitting

   Problem: Model performs well on training data but poorly on new data

   Solution: Use regularization (L1/L2) or cross-validation

4. Imbalanced data

   Problem: Rare class gets ignored (e.g., 95% class 0, 5% class 1)

   Solution: Use class weights, oversampling, or different thresholds

5. Non-linear relationships

   Problem: Linear assumption may not hold for some predictors

   Solution: Add polynomial terms or use splines

6. Advanced Topics in Logistic Regression

6.1 Multinomial Logistic Regression

Extends binary logistic regression to handle outcomes with >2 unordered categories. Uses softmax function instead of sigmoid:

P(Y=k) = e^{(β₀k + β₁kX)} / Σ(e^{(β₀j + β₁jX)}) for j=1 to K

6.2 Ordinal Logistic Regression

For ordered categorical outcomes (e.g., low/medium/high). Uses cumulative logits:

log(P(Y≤k)/P(Y>k)) = αₖ – βX for k=1 to K-1

6.3 Regularized Logistic Regression

Adds penalty terms to prevent overfitting:

• L1 (Lasso): Can shrink coefficients to exactly zero (feature selection)
• L2 (Ridge): Shrinks coefficients but rarely to zero
• Elastic Net: Combination of L1 and L2

7. Real-World Applications

Industry	Application	Predictor Variables	Outcome Variable
Healthcare	Disease risk prediction	Age, BMI, blood pressure, genetic markers	Disease presence (1/0)
Finance	Credit scoring	Income, credit history, loan amount	Default (1/0)
Marketing	Customer churn	Usage frequency, customer service contacts	Churn (1/0)
Manufacturing	Quality control	Production parameters, material properties	Defect (1/0)
Social Sciences	Voter behavior	Demographics, past voting, issue positions	Vote choice (1/0)

8. Software Implementation Comparison

While manual calculations are valuable for understanding, most practical applications use statistical software:

Software	Function/Command	Advantages	Limitations
R	glm(family=binomial)	Extensive statistical capabilities, free, open-source	Steeper learning curve
Python (scikit-learn)	LogisticRegression()	Great for production, integrates with ML pipelines	Less statistical output than R
Stata	logit or logistic	Excellent for social sciences, good documentation	Expensive license
SAS	PROC LOGISTIC	Enterprise-grade, comprehensive output	Very expensive, complex syntax
SPSS	Analyze → Regression → Binary Logistic	User-friendly GUI, good for beginners	Limited customization, expensive

9. Learning Resources

For those interested in deeper study of logistic regression, these authoritative resources provide excellent foundations:

• National Library of Medicine: Logistic Regression Analysis

  Comprehensive guide to logistic regression in medical research with practical examples

• UC Berkeley: Introduction to Logistic Regression

  Academic paper covering theoretical foundations and mathematical derivations

• NCSS: Logistic Regression Handbook

  Practical guide with software implementation examples and interpretation tips

10. Conclusion and Best Practices

Manual calculation of logistic regression provides invaluable insights into how the model works at a fundamental level. While modern software handles the computations effortlessly, understanding the underlying mathematics enables better model interpretation, troubleshooting, and communication of results.

Key takeaways:

• Logistic regression predicts probabilities, not classes directly
• The sigmoid function ensures outputs stay between 0 and 1
• Coefficients represent log-odds changes, not probability changes
• Odds ratios (e^β) are more interpretable than raw coefficients
• Threshold selection should consider the costs of false positives/negatives
• Model evaluation requires multiple metrics beyond just accuracy

Best practices for implementation:

1. Always check for complete separation before modeling
2. Standardize continuous predictors if using regularization
3. Examine coefficient signs for logical consistency
4. Check for influential observations using leverage plots
5. Validate model assumptions (linearity in log-odds, no omitted variables)
6. Use cross-validation for more reliable performance estimates
7. Document all modeling decisions for reproducibility

By mastering these manual calculations and understanding their interpretation, you’ll be better equipped to apply logistic regression effectively in real-world scenarios, critically evaluate model outputs, and communicate results to stakeholders.