A minimal Project Tree in R

You can also check this post, written in #blogdown, here: minimal-project-tree-r.

Introduction

The last two days arrived at my twitter feed some discussions on how bad are the following sentences at the beginning of your R script/notebook, sparked by @JennyBryan’s slides at the IASC-ARS/NZSA Conference:

setwd()

and

rm(list = ls())

 

Jenny Bryan offered a detailed explanation for this, as well as some fixes, in her tidyverse blog post. The main idea was:

  • To ensure reproducibility within a stable working directory tree. She proposes the very concise here::here() but other methods are available such as the template or the ProjectTemplate packages.
  • To avoid break havoc in other’s computers with rm(list = ls())!.

All of this buzz around project self-containment and reproducibility motivated me to finish a minimal directory tree that (with some variations) I have been using for this year’s data analysis endeavours.

It is a extremely simple tree which separates a /data, a /plot and an /img directory inside the main folder (root)

  • The data folder contains both raw data and processed data files saved by R.
  • The plot folder contains all the plots saved during the workflow.
  • The img folder has every other image (logos, etc) that R takes as an input to build the results.
  • Inside the root folder I store the main .R or .Rmd scripts.

This ensures that every folder has an unidirectional relationship with the root folder (except the data dir in this case). But the important thing is that the paths in the scripts are set relative to the root folder, so the entire tree can be copied elsewhere and still work as expected.

I also added some more features to the tree:

  • An .Rproj file
  • Parametrize the .Rmd file
  • Git repository so the tree can be conveniently cloned or downloaded, with a .gitignore file:

Here is a sketch of how it works:

And here is the actual code of the notebook/script. I have not included regular markdown text outside the R chunks, as this template is intended to be changed and filled with new text each time:

Script code

# Installs missing libraries on render!
list.of.packages <- c("rmarkdown", "dplyr", "ggplot2", "Rcpp", "knitr", "Hmisc", "readxl")
new.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[,"Package"])]
if(length(new.packages)) install.packages(new.packages, repos='https://cran.rstudio.com/')
library(dplyr)
library(knitr)
library(ggplot2)

Working directories

# directory where the notebook is
wdir <- getwd() 
# directory where data are imported from & saved to
datadir <- file.path(wdir, "data") # better than datadir <- paste(wdir, "/data", sep="")
# directory where external images are imported from
imgdir <- file.path(wdir, "img")
# directory where plots are saved to
plotdir <- file.path(wdir, "plot")
# the folder immediately above root
Up <- paste("\\", basename(wdir), sep="") 
wdirUp <- gsub(Up, "", wdir) 

Data import

# Data name (stored as a parameter in the Rmarkdown notebook)
params <- NULL
params$dataname <- "cars"
# MSEXCEL
dataname <- params$dataname # archive name
routexl <- paste(datadir, "/", dataname, ".xlsx", sep="")  # complete route to archive

library(readxl)
mydata <- read_excel(routexl, sheet = 1)  # imports first sheet
# CSV / TSV (separated by tabs in this example)
dataname <- params$dataname # archive name
routecsv <- paste(datadir, "/", dataname, ".csv", sep="")  # complete route to archive

mydata <- read.csv(paste(routecsv, sep=""), 
         header = TRUE, 
         sep = "\t",
         dec = ".")

Data operations

# Hmisc::describe(mydata)
head(mydata)
     speed dist
   1     4    2
   2     4   10
   3     7    4
   4     7   22
   5     8   16
   6     9   10
p1 <- ggplot(mydata, aes(x=speed, y=dist)) + geom_point()
p1

Save plots

# TO PDF 
plotname1 <- "p1.pdf"
# TO PNG 
plotname2 <- "p1.png"

routeplot1 <- file.path(plotdir, plotname1)
routeplot2 <- file.path(plotdir, plotname2)
ggsave(routeplot1)  # (see http://ggplot2.tidyverse.org/reference/ggsave.html)
ggsave(routeplot2) 

Save data

# RDATA
save(mydata, file="data/mydata.RData")
# MSEXCEL # not run
dataname2 <- "mydata"  # name we will give to file
routexl2 <- paste(datadir, "/", dataname2, ".xlsx", sep="")   # complete route to future archive

library(xlsx)
write.xlsx(mydata, routexl2) # creates archive in specified route
# CSV / TSV (separated by tabs in this example)
dataname2 <- "mydata"  # name we will give to file
routecsv2 <- paste(datadir, "/", dataname2, ".csv", sep="")  # complete route to future archive

write.table(mydata, file = routecsv2, append = FALSE, quote = FALSE, sep = "\t ",
            eol = "\n", na = "NA", dec = ".", row.names = FALSE,
            col.names = TRUE)

 

This script -and the dir tree that contains it- is saving me a lot of time and headaches (where I’ve put that data?….), I hope it can be also useful for people out there!.

Future improvements

Anuncios

…start using R, from scratch!

Some time ago, since I was able to use R by myself, have found some fellows and other people who wanted to learn R as well. Then I pointed them to help pages, to CRAN repositories… but in some cases they said that didn’t know how to start using those resources. Obviously, the main self-perceived limitation for non-programmers is the use of “commands” -ok, many of the 80’s kids will remember the use of some command lines to access games such as PacMan, Frogger… :).

At the same time, they also wanted to refresh some basic statistics, acquiring a general knowledge of their data before asking for a statistician’s help. An idea to quickly help them was to make some scripts to guide them through basic commands, seeing results on real-time, and being able to recycle them for their own data.

If you have just started using R, maybe they can be useful for you. However, I will recommend that you use some open “plain text” file(s) to paste your favorite commands and clone/modify them to suit your needs. Remember to store the files where you can access them later!

  • Tip: you can change the extension of your mytext.txt file into mytext.R file, telling Windows to open it with the Notepad again. It will be also a plain text document, but some text editors will recognize it as an “R script” and will highlight the content according to that.
  • Apart from the Notepad in Windows, you also have a bunch of other text/code editors which are more pleasant to use. See for example R-studio and Notepad ++.

Copy the Gists below into your own text files, and begin playing with R!

…learning LaTeX, from scratch!

LaTeX is a high-quality typesetting system; it includes features designed for the production of technical and scientific documentation, and is the de facto standard for the communication and publication of scientific documents.”

It is also… Free and Open. Specially designed for beautiful documents even when they include special language: mathematic formulae, phonetic symbols, musical notation…

The mix of normal text with special languages is indeed much better arranged in LaTeX than in any other text editor, so maybe it can be interesting for you!. If you want to start using LaTeX, first you have to install it. Here is a previous post with something about LaTeX installation and useful links. In particular, I like the LaTeX editor “TeXworks” for having a simpler GUI -graphic interface-, so that’s what I’m gonna use for this tutorial.

Once installed a LaTeX distribution and a TeX editor (you have both with MiKTeX), you can open TeXworks. Here’s what it looks like:

LaTeXtemplate0

To start making documents, I recommend you to choose one of the templates available:

click: File >> New from Template >> choose Article

LaTeXtemplate

Then, a template will appear. You can highlight the code going to:

Format >> Syntax coloring >> LaTeX

It should look like this:

LaTeXtemplate2

Read carefully this “source file”, that is, the original .tex file, for it provides basic instructions and tips for writing.

Then, you can simply press the green, round button and see what happens… then change something in the source file… then press the button again… and so on.

If you wish to learn to insert things like images, tables, etc… then… Wikibooks is for you, take a look. I hope this tutorial has been useful!

…start using Sweave, from scratch!

INTRODUCTION

Sweave is nothing more, nothing less than one of the best ways R can connect with a text editor, in this case LaTeX.

So you don’t know anyting about LaTeX? neither did I 8 months ago… The hyperlinks in this post will take you to some great pages, to learn different things in an organised way. Here’s how I did it (with lots of help from google though!).

It’s important to take it step by step…

Second, you need to install R, from here. Once you have installed it, you will need to install a LaTeX distribution (my preferred LaTeX editor is TeXworks, and I have MiKTeX in MS Windows).

  • If you are a MS Windows user, you will find useful this one: MiKTeX
  • For Linux users, here can be Kile (from KDE), TeXworks, or other alternatives. First of all you should install LaTeX (e.g. TeX Live), and configure it with these tips.
  • For Mac users, you can follow this link which provides instructions to install TeXmaker. You will be able to install TeXworks from here.
  • NOTE: Both, MiKTeX and TeXmaker have portable versions, easy to carry in a pendrive!

What TeX editor did you install? If you have TeXworks, the way to configure Sweave is pretty quick and easy (taken from this great link). If you are a beginner, this may be the best option.

MacTeX installation screen

CONFIGURING TEXWORKS

If you have TeXworks, all you have to do is go to:

Edit >> Preferences >> Typesetting

Then, you will see this screen:

TexworksSweave-1

If you click the “+” button (highlighted in green) to add a new tool, another screen will appear:

TexworksSweaveCommands-1

You only have to fill in these fields with the name of the new tool, “Sweave” and adding its “arguments” clicking in the “+” sign.

The field “Program” is filled with the path to your R executable, which can be similar to that in the picture. Just be careful to type “slashes” ( / ) instead of “backslashes” ( \ ).

The field “Arguments” will contain all the commands that Sweave needs to connect to the R console, to interpret R’s code and to produce a .TEX file and a .PDF.

In (K)Ubuntu is configured in a similar way:

KubuntuTeXWorksSweave

And so in Mac OS X (note that we have the same Sweave commands as in Ubuntu):

SweaveinMacOS

WRITING A .RNW FILE

Let’s write something in a new Sweave file!. To this, you must create a new text file (e.g. myfirstsweave.txt) , and change its extension to .Rnw -see this for more help-:

myfirstsweave.Rnw

This is a blank file, but you can paste the following code to construct a template:

Once you have your template, you must… “compile” it. First, you can highlight the code with nice colours clicking:

Format >> Syntax Coloring >> LaTeX

To compile the LaTeX/Sweave file, you have to click in the pull-down menu, and select:

Sweave

Sweave_compilation

Great! the last step is to click in the green, round “play” button…

Voilà!

sweave_output

EDIT FOR TEXWORKS IN UBUNTU

The installation of TeXworks in my Kubuntu 12.04 has been direct from the console -at least it worked for me-, without following other instructions:

sudo apt-get update
# sudo apt-get upgrade
sudo apt-get install texworks

After the configuration of Sweave typesetting -see above-, I was trying to compile my own template, and this error appeared:

Error in texi2dvi(file = file, pdf = TRUE, clean = clean, quiet = quiet, :

Running ‘texi2dvi’ on ‘untitled-1.tex’ failed.

Messages:

sh: 1: /usr/bin/texi2dvi: not found

Texi2dvi appears to be a script that allows direct compilation of a .pdf from a .Rnw file.

KubuntuTeXWorksIssue

Maybe it’s not the cleanest solution, but I copied the source code of texi2dvi and created a script with it: (sudo gedit if you are running Ubuntu, sudo kate if Kubuntu)

Open up a Terminal and type:

 sudo kate /usr/bin/texi2dvi

Then you can paste the code into this script, and save the changes. To make it executable, type in the Terminal:

 sudo chmod 755 /usr/bin/texi2dvi
Society for Social Medicine

Advancing Knowledge for Population Health

Mad (Data) Scientist

Musings, useful code etc. on R and data science

My Blog

A topnotch WordPress.com site

walkandfish

Just another WordPress.com site

Georeferenced

A blog on all things Geo, Data, Technology & the interconnected world. Occasionally off-piste.

Retraction Watch

Tracking retractions as a window into the scientific process

"R" you ready?

My advances in R - a learner's diary

TRinker's R Blog

Experiments & Experiences in R

What You're Doing Is Rather Desperate

Notes from the life of a [data] scientist

On unicorns and genes

Martin Johnsson's blog

vet epi

Denis Haine

FreshBiostats

Young Researchers in Biostatistics

Nicebread

...messing around with free code

TRinker's R Blog

...messing around with free code

Revolutions

...messing around with free code

Learning R

Finding my way around R

R-bloggers

R news and tutorials contributed by (750) R bloggers