Multivariate Bayesian variable selection regression

ROC comparisons

Or rather, precision-recall curve.

In [1]:
%revisions -s -n 10
Revision Author Date Message
b2e364b Gao Wang 2018-06-23 Adjust PIP cutoff for PRC
2cc5551 Gao Wang 2018-06-23 Implement precision-recall curve
e19690c Gao Wang 2018-06-22 Add ROC for other methods
00fb9bd Gao Wang 2018-06-11 Update PIP-based ROC plot
d719ae1 Gao Wang 2018-06-11 Do not use random physical blocks for PIP based ROC
24e1fbc Gao Wang 2018-06-07 Update revision table
27ea530 Gao Wang 2018-06-06 Compute ROC by random blocks
In [2]:
%cd ~/GIT/github/mvarbvs/dsc

/home/gaow/GIT/github/mvarbvs/dsc

Workflow is implemented in the roc section of this notebook. We focus on PIP > 0.3.

In [3]:
colors = c('#348ABD', '#7A68A6', '#A60628', '#467821', '#FF0000', '#188487', '#E2A233',
                  '#A9A9A9', '#000000', '#FF00FF', '#FFD700', '#ADFF2F', '#00FFFF')

Susie vs DAP

For causal from 1 to 5.

In [4]:
d1 = readRDS('susie_comparison/ROC_0622_estvar_true_two.rds')
d2 = readRDS('susie_comparison/ROC_0622_estvar_false_two.rds')
In [5]:
png("susie_comparison/ROC_0622_F1.png", 6, 6, units = 'in', res = 500)
plot(d1$susie$rates$Precision ~ d1$susie$rates$Recall, t="l", 
     col=colors[1], ylab = "Precision", xlab ="Recall", main = "Num. causal 1 to 5 pooled")
lines(d2$susie$rates$Precision ~ d2$susie$rates$Recall, col=colors[2])
lines(d1$dap$rates$Precision ~ d1$dap$rates$Recall, col=colors[3])
legend("bottomleft", legend=c("SuSiE (est_var)", "SuSiE (var(Y)))", "DAP-G"),
       col=colors[1:3], lty=c(1,1,1), cex=0.8)
dev.off()
png: 2
In [6]:
%preview susie_comparison/ROC_0622_F1.png
> susie_comparison/ROC_0622_F1.png (259.5 KiB):
In [7]:
d3 = readRDS('susie_comparison/ROC_0622_estvar_true_all.rds')
d4 = readRDS('susie_comparison/ROC_0622_estvar_false_all.rds')
In [8]:
png("susie_comparison/ROC_0622_F2.png", 6, 6, units = 'in', res = 500)
plot(d3$susie$rates$Precision ~ d3$susie$rates$Recall, t="l", 
     col=colors[1], ylab = "Precision", xlab ="Recall", 
     main = "Num. causal 1 to 3 pooled")
lines(d3$dap$rates$Precision ~ d3$dap$rates$Recall,col=colors[3])
lines(d3$caviar$rates$Precision ~ d3$caviar$rates$Recall,col=colors[4])
lines(d3$finemap$rates$Precision ~ d3$finemap$rates$Recall,col=colors[7])
legend("bottomleft", legend=c("SuSiE", "DAP-G", "CAVIAR", "FINEMAP"),
       col=c(colors[1], colors[3], colors[4], colors[7]), lty=c(1,1,1), cex=0.8)
dev.off()
png: 2
In [9]:
%preview susie_comparison/ROC_0622_F2.png
> susie_comparison/ROC_0622_F2.png (256.2 KiB):
In [10]:
convert +append susie_comparison/ROC_0622_F1.png susie_comparison/ROC_0622_F2.png susie_comparison/ROC_0622.png
In [11]:
%preview susie_comparison/ROC_0622.png
> susie_comparison/ROC_0622.png (471.2 KiB):

Why use precision-recall curve

$Precision=\frac{TP}{TP+FP}$ which is $1-FDR$. $Recall = \frac{TP}{TP+FN}$ aka sensitivity.

  • We do not use conventional ROC because total positives are very small (1 to 5 out of every 1000) so false positive rate FP / (FP + TN) will be very small.
  • It might be better to plot FDR vs Power.
  • Precision-recall curve is similar to FDR vs Power because
    • Precision: TP / (TP + FP) = 1 - FDR
    • Recall: TP / (TP + FN) = sensitivity = power

susie vs DAP (FDR vs Power version)

In [14]:
png("susie_comparison/ROC_0622_F3.png", 6, 6, units = 'in', res = 500)
plot(1 - d1$susie$rates$Precision, d1$susie$rates$Recall, t="l", 
     col=colors[1], ylab = "Power", xlab ="FDR", main = "Num. causal 1 to 5 pooled")
lines(1 - d2$susie$rates$Precision, d2$susie$rates$Recall, col=colors[2])
lines(1 - d1$dap$rates$Precision, d1$dap$rates$Recall, col=colors[3])
legend("bottomright", legend=c("SuSiE (est_var)", "SuSiE (var(Y)))", "DAP-G"),
       col=colors[1:3], lty=c(1,1,1), cex=0.8)
dev.off()
png: 2
In [15]:
%preview susie_comparison/ROC_0622_F3.png
> susie_comparison/ROC_0622_F3.png (237.2 KiB):
In [19]:
png("susie_comparison/ROC_0622_F4.png", 6, 6, units = 'in', res = 500)
plot(1 - d3$susie$rates$Precision, d3$susie$rates$Recall, t="l", 
     col=colors[1], ylab = "Power", xlab ="FDR", 
     main = "Num. causal 1 to 3 pooled")
lines(1 - d3$dap$rates$Precision, d3$dap$rates$Recall,col=colors[3])
lines(1 - d3$caviar$rates$Precision, d3$caviar$rates$Recall,col=colors[4])
lines(1 - d3$finemap$rates$Precision, d3$finemap$rates$Recall,col=colors[7])
legend("bottomright", legend=c("SuSiE", "DAP-G", "CAVIAR", "FINEMAP"),
       col=c(colors[1], colors[3], colors[4], colors[7]), lty=c(1,1,1), cex=0.8)
dev.off()
png: 2
In [20]:
%preview susie_comparison/ROC_0622_F4.png
> susie_comparison/ROC_0622_F4.png (229.9 KiB):
In [21]:
convert +append susie_comparison/ROC_0622_F3.png susie_comparison/ROC_0622_F4.png susie_comparison/ROC_0622_FDR.png
In [23]:
%preview susie_comparison/ROC_0622_FDR.png
> susie_comparison/ROC_0622_FDR.png (432.9 KiB):
In [ ]:
Copyright © 2016-2020 Gao Wang et al at Stephens Lab, University of Chicago