Daily-Blog-Sai

Author

Sai

Blog’s Repo link

utilities_for_plotting_genomic_signal_tracks

This post contains function for plotting genomic signal tracks as well as additional utilities to make working with this kind of data easier.

Aracena_predixcan_workflow

The figure below shows the…

test_plotly_in_quarto

analyze_xwas_results

After running summary predixcan on different epigeneti marks and disease conditions, we can analyze the results to see if these marks help identify associations with disease.

enformer_peak_linearization

This notebook creates inputs for Enformer after creating sets of fine-tuned variants from a GWAS

check_gwas_peak_overlap

This notebook checks the overlap between GWAS fine-tuned loci and epigenetic peaks

predict_for_linearization

Linearize_peak_EnPACT

Reformat summ stats for susieR

compare_elastic_net_EnPACT_models

This notebook is used to compare the number of non-zero coefficients in…

Analyze_personalized_peak_predictions

Having developed a repo…

Determine best enformer track for peak data

When validating the usefulness of EnPACT models, it is helpful to also gain a handle on how much the fine-tuning step (aka the elastic net) is providing useful predictive…

analyze_peak_window_sizes

build_peak_EnPACT

In this notebook, I build linearized Con-EnPACT models for peak data from Aracena et al. The peak data are as follows:

0.1_normalize_peak_data

Having successfully run Con-EnPACT on…

assess_peak_data

Peak data has a number of differences from…

test_window_size

I need to test variable input window size for Con-EnPACT modeling.

write_predict_db_checks

We have been having some difficulties running Predict-DB, so here I am going to debug my latest issues and make a more formal set of pre-checks everyone can use.

0.1_prepare_kallisto_quant

For EnPACT training, it…

0.2_split_data

2.1.2_check_predictDB_standard

I seem to be getting strangely low gene counts past filtering after running PredictDB. Need to investigate this.

1.3.2_predict_for_personalized_predictions

In order to linearize, we first need to predict on the

1.5.2_format_harmonized_GWAS_sumstats_for_SPrediXcan

9980956 rows × 11 columns

3.1.1_analyze_enpact_SPrediXcan_results

3.1.1_analyze_enpact_SPrediXcan_results

Need some unified functions for analyzing SPrediXcan results.

3.1.1_analyze_enpact_SPrediXcan_results

After running Summary PrediXcan using different types of linearized models, we can then compare the results to evaluate effectiveness.

0.1_normalize_expression_GTEx

It is important when running S-PrediXcan type analyses to properly normalize the expression data. Thus far, I have employed simple log normalization, but in…

1.4.3_collect_geuvadis_preds

1.4.2_predict_for_linearization.qmd was run in order…

1.5.1_SPrediXcan_fine-tuned

Having linearized the fine-tuned DL model in 1.4, we can now run SPrediXcan using this linearization to try and identify common genetics with GWAS signals.

1.3_evaluate_fine_tuned_model

Set color palette and output dir

1.4.5_assess_linearization

1.2_train_expression_check_training

1.3.2_personalized_predictions_standard

After training the Con-ENPACT model on variation in gene expression across…

1.4.1_prepare_geuvadis_predictions

Now that I have a working model of gene expression prediction in flu from Enformer predictions, I can continue with the linearization process.

1.1_generate_data

1.4.2_predict_for_linearization

0.1_aracena_predixcan

Having successfully linearized the fine-tuned Aracena predictor, I should also establish a baseline by performing the PrediXcan analysis on the Aracena data. While this was…

2.1.1_prepare_predictDB_standard_inputs

I need to run a standard PrediXcan TWAS analysis…

predict_for_linearization

assess_linearization

linearization

Now that I have a working model of gene expression prediction in flu from Enformer predictions, I can continue with the linearization process.

search_enformer_variant_effects

I have downloaded precomputed Enformer variant effect sizes from the…

QTL_mutagenesis

I wish to perturb the “best” SNP at each gene and observe the effect size on the flu prediction

make_aracena_enformer_preds

Made Enformer predictions on a set of topQTLs, now I can predict using my trained models and try to infer variant effects

log_transform_training_tss_predictors

After evaluating the performance of my trained models, it seems clear that the spearman correlations are very stable genome-wide, but the pearson correlations are not. This…

evaluate_tss_center_mean_pred

Here I trained an elastic net and simple MLP mode to predict TSS RNAseq expression from Flu condition (aracena dataset) from Enformer…

eQTL_variant_effects_aracena_models

Having trained several different models relating predicted Enformer variant effect predictions to ground truth Aracena signal, it is now worth testing these by observing…

eQTL_variant_effects_aracena_models

Having trained several different models relating predicted Enformer variant effect predictions to ground truth Aracena signal, it is now worth testing these by observing…

simulate_summed_phenotype

Simulate true phenotype as sum of individual variant effects.

train_aracena_tss_predictors_evaluation

Here I trained an elastic net and simple MLP mode to predict TSS RNAseq expression from Flu condition (aracena dataset) from Enformer…

train_aracena_tss_predictor_generate_data

I am currently training larger scale enformer prediction -> aracena track models. It’s unclear how long and with what parameters this training will take. In the meantime, I…

harmonizing_training_data_tss_centered

Ok cool, this table contains the indeces for each individual/window combo in their corresponding training dataset. We can subsequently filter out regions, individuals for…

harmonizing_training_data_tss_centered

Poisson loss investigation

Training predicted epigenome to aracena model using full dataset

Having demonstrated that the…

train_pred_to_aracena_full

Evaluating predicted epigenome to aracena model using full dataset

Implementing single node DDP training loop

After successful implementation of a basic training scheme, and demonstration of similarly basic training on subsets of training dataset, it is now time to refocus on the…

consolidate_aracena_training

Regarding the training of the Aracena reference epigenome -> aracena track prediction model, my progress can be summarized as follows:

retraining_pred_to_aracena_models_evaluation

These sites and corresponding TSS sites were extracted in 2023-09-20-diagnose_training_tracks_issue and serve as a good test set to observe model performance.

retraining_pred_to_aracena_models

Load in some example input data

retraining_pred_to_aracena_models

diagnose_training_tracks_issue

I recently have been performing training of the Enformer prediction to Aracena track models. While the training appears to be occuring, with convergence of the loss…

diagnose_training_tracks_issue

harmonizing_training_data

Through my prior training runs, I was able to start training a predicted epigenome-to-aracena track model on data subset. This is promising, but an issue for scaling up is…

harmonizing_training_data

Ok cool, this table contains the indeces for each individual/window combo in their corresponding training dataset. We can subsequently filter out regions, individuals for…

evaluating_aracena_training

Load in some validation datasets for evaluation

evaluating_aracena_training

ref_to_aracena_mlp

Load in some example input data

ref_to_aracena_mlp

For interactive GPU session on beagle:

modeling_aracena_reference_enformer

The goal of creating a training dataset from Aracena et al. data is to use it for actual modeling purposes. There are a variety of ways to do this, with the longer term goal…

prepare_aracena_training_data

As per my updated thesis proposal, I intend to use the Aracena et al. dataset to train personalized, context specific DL models for downstream prediction…

testing_pop_seq

I have verified that the pop-seq…

guide_to_running_pytorch_training

Here I compiled a guide on how to run distributed training using pytorch DDP on ALCF Polaris

creating_population_sequences

Here I will describe in…

test_pop_seq_training

Have been busy…

calculating_genetic_distance_from_ref

Previously, I looked for an association between the local ancestry of genes in the refernce genome and their direction of personalized Enformer expression prediction…

testing_ancestry_differences

Previously I have collected local ancestry information for different genes (03-27). The hypothesis here is that the mismatches observed between local ancestry in the…

training_small_input_enformer

The main questions we wanted to answer during the Hackathon were:

compiling_DL_training_materials

At the ALCF hackathon we had a very productive…

post-BoG-notes

Overall, the conference felt very productive. There are of course certain aspects which felt more valuable than others.

correcting-antisense-gene-prediction

There are 9618 sense (+) genes and 9408 antisense (-) genes in the cohort

population-allele-freq-enformer

With the upcoming ALCF INCITE Hackathon, our goals are two-fold:

quantifying_tss_expression_across_sites

Last week I looked in-depth at the primary TSS quantification for ERAP2 and saw that using the primary TSS alone…

running_Karl_geneset_enrichment

Karl Tayeb in Mathew Stephens’ lab has been working on an improved geneset enrichment software.

updating-epistasis-analysis-primary-TSS

It is now time to start working on…

analyzing-ERAP2-personalized-mutagenesis

ERAP2-personalized-mutagenesis

Last week I was working on implementing personalized mutagenesis updates were a bit infrequent due to lots of meetings and most work being done on the remote cluster.

erap2-fine-mapping

Am going to refocus on ERAP2 again for a few reasons:

reference-genome-local-ancestry

One thing we hadn’t yet taken a look at is the local ancestry of genes in the reference genome. This is a useful metric which should be included in our gene metadata.

title

Single TSS Centering

Thus far, I have been running my gene expression estimates by centering the Enformer window on the gene body as a whole. This is then followed by quantification by summing…

Updating Existing GEUVADIS Analyses part 3

I still need to update my epistasis analysis with the new data. After doing so, the results are roughly as follows (full analysis is in the analysis repo).

Updating Existing GEUVADIS Analyses part 2

Today I am continuing to update my analyses with new data.

Updating Existing GEUVADIS Analyses

After debugging the merged pipeline I reran the GEUVADIS data and replicated the Kircher mutagenesis as a test of the quantification (which I also optimized a bit). I also…

Replicating Kircher Mutagenesis p2

Last week we implemented reverse complement prediction into our core, merged Enformer pipeline after correcting a different bug we had with it. The reverse complement still…

Replicating Kircher Mutagenesis

Last week we implemented reverse complement prediction into our core, merged Enformer pipeline after correcting a different bug we had with it. The reverse complement still…

Implementing Personalized Mutagenesis

Yesterday I restarted the pipeline after Temi debugged it. After it ran for some time, I checked the results. The results are looking much more reasonable now (values are…

Setting up mutagenesis of whole window

Yesterday, the issue with the merged pipeline was identified. The work for doing that can be found here:

debugging-merged-pipeline

Yesterday, I observed that the Enformer CAGE outputs for…

Test blog setup

Note: This is an example centered around setting up a blog post system. You can add other Quarto analyses following these principles as well.