Aims

As part of this experiment, I wanted to:

• get Stata and R working well together; and
• send a notification to my phone when a long running Stata script was complete.

Aim #1. Get Stata and R working well together

My initial approach was to set up a RStudio project, and rearrange the scripts and data I had received into a familiar structure:

.
+-- code
|   +-- .do files
|   +-- .R scripts
+-- data
+-- output
+-- reports
+-- manuscript
+-- supplement
+-- Config - STATA.do  # Configuration file for Stata
+-- Config - R.R       # Configuration file for R

The previous owner of the project had set up a configuration file for both Stata and R, which when run, would source the relevant .do files and .R scripts from the code folder in the correct order.

For example, the Config - STATA.do file set the necessary global paths and then ran a series of .do files to process the raw data, generate the basic cohort, add covariates, calculate summary statistics for the cohort, calculate the frequency of missing data for each important covariate, and then finally perform the analysis of interest:

# Config - STATA.do

  * Set global paths -----------------------------------------------------------
    
  global path "path/to/project/folder"
  global dofiles "$path/code"
  global output "$path/output"
  global data "$path/data"
  
  * Run dictionary -------------------------------------------------------------
  
  run "$dofiles/codedict.do"
  
  * Generate cohort ------------------------------------------------------------

  run "$dofiles/cohort2.do"
  
  * Add covariates (Note: cov.do requires cov_*.do) ----------------------------
  
  run "$dofiles/cov2.do"
  
  * Generate data for Table 1 --------------------------------------------------
  
  run "$dofiles/toc.do"
  
  * Generate data on missing covariate information -----------------------------
  
  run "$dofiles/missingdata.do"
  
  * Run Cox regression analysis ------------------------------------------------
  
  run "$dofiles/analysis2.do"

Similarly , the Config - R.R file loaded the necessary libraries and then sourced the scripts to build the three plots.

# Config- R.R

  # Setup ======================================================================
  library(ggplot2)
  
  # Plot analysis results ======================================================
  
  source("code/plot_cohort2_attrition.R")
  
  source("code/plot_main_forest_plots.R")
  
  source("code/plot_age_forest_plots.R")

To begin with, I would run each of these files seperately, one after the other, which worked perfectly but ran against my aim of being able to run the whole project with a single command.

To begin to address this, I created a new master.R script which would call both the Config - STATA.do and the Config - R.R files. The dependency tree for the new set-up looked like this:

master.R
+-- Config - STATA.do
|   +-- code
|   |   +-- .do files
+-- Config - R.R
|   +-- code
|   |   +-- .R scripts

To call Stata from R, I used the RStata package, built by Luca Braglia, which allows users to execute Stata commands (both inline and from a .do file) from within R itself (the name is a bit of a give-away). You just need to provide the path to your .do file, the path to your Stata executable (stata.path), the version of Stata you have (stata.version) and whether you want to print Stata text output to the console (stata.echo) as arguments.

# master.R - Version 1

  # Analysis ===================================================================
  RStata::stata("Config - Stata.do",
                stata.path = "\"A:\\Stata\\Stata15_MP\\StataMP-64\"",
                stata.version = 15,
                stata.echo = TRUE)


  # Plotting ===================================================================
  source("Config - R.R")

# Fix issue with network drives

  # Replace 
  RStata::stata("Config - Stata.do",
                stata.path = "\"\\ads.bris.ac.uk\\path\\to\\stata\"", #
                stata.version = 15,
                stata.echo = TRUE)
  
  # with
  RStata::stata("Config - Stata.do",
                stata.path = "\"A:\\path\\to\\stata\"", # 
                stata.version = 15,
                stata.echo = TRUE)

Aim #2. Send a notification to my phone when a long running Stata script is complete.

For some of the analysis performed, running the Config - STATA.do file would take >4 hours. In this case, I thought it would be useful to have someway of the script telling me that it was finished, so I could go out and have a walk/coffee or work on other things without feeling the need to constantly check on the progress.

To achieve this, I used the pushoverr package, which allows you to send notifications to the pushover app on your phone via R. I found this blogpost by Brian Connelly very helpful when getting started with pushoverr.

The only downside to Pushover as a system is that after a 7 day trial period, there is a cost of ~£4 to activate the app fully - but personally I think it’s worth it!

So having added pushoverr, my master.R file now looked like this:

# master.R - Version 2

  # Analysis ===================================================================
  RStata::stata("Config - Stata.do",
                stata.path = "\"A:\\Stata\\Stata15_MP\\StataMP-64\"",
                stata.version = 15,
                stata.echo = TRUE)
  
  pushoverr::pushover(message = 'Analysis complete.')
  
  # Plotting ===================================================================
  source("Config - R.R")
  
  pushoverr::pushover(message='Plotting complete.')

By this point, I was cresting the “Peak of inflated expectations” and was feeling very pleased with myself. It didn’t last very long however due to one important limitation of the RStata package . . .

# Add to the very end of 'Config - STATA.do'

  clear
  set obs 0
  gen pushover = ""
  save "$path/pushover.dta"
  

# master.R - Version 3
  
  # Setup ======================================================================
  source("code/pushover.R")
  
  push_clean() # Remove pushoverr indicator file
  
  # Analysis ===================================================================
  RStata::stata("Config - Stata.do",
                stata.path = "\"A:\\Stata\\Stata15_MP\\StataMP-64\"",
                stata.version = 15,
                stata.echo = TRUE)
  
  push_analysis() # Send pushoverr notification to phone on analysis status
  
  # Plot main analysis results =================================================
  source("Config - R.R")
  
  push_plotting() # Send pushoverr notification to phone on plotting status

#pushover.R

  push_clean <- function() {
    try(file.remove("pushover.dta"))
  }
  
  push_analysis <- function() {
    if (file.exists("pushover.dta") == TRUE) {
      pushoverr::pushover(message = 'Analysis complete.')
    } else{
      pushoverr::pushover(message = 'Analysis errored.')
    }
    file.remove("pushover.dta")
  }
  
  push_plotting <- function() {
    pushoverr::pushover(message='Plotting complete.')
  }

Final word

I had a look on Twitter to see if there were any other methods people used to signify that a script was complete and came across this tweet from Anna Kautto:

Today I made my R script say “I have the answer to the great question though I dont think you are going to like it” out loud after finishing calculating. #rstats #deepthought #glmer #phdchat

— Anna Kautto (@annakau) November 11, 2019

While I think the idea of my computer unexpectedly talking to me would likely make me jump and spill tea everywhere, I love the idea and may update the content of my pushoverr notifications to something a bit more fun!