Hour 20. Advanced Package Building
What You’ll Learn in This Hour:
What you can do to extend an R package
Why testing is important and how to use testthat
How to include datasets in a package
How to include a user guide in a package
What you need to do to use C++ code in a package
In the last hour, you saw how to put all of your code together in the form of a package to simplify the sharing and maintenance of code, as well as to aid in the development of high-quality, production-ready code. There are, however, a number of ways you can extend a package to make it more robust to changes and easier for users to get started with. You will see the most common of these extra components in this hour.
Extending R Packages
We have now managed to create a package that contains all the functions we need and even contains the help files for those functions—so why do we need to add more? Surely this is sufficient. In many respects, this is true. We can simply share our package as it is with no need to do anything more, but there are many advantages to the extensions you will see in this hour.
The first additional component we will cover is a test framework. As you have seen throughout this book, once we have code we may want to update it to make it more efficient or simply change the functionality as we find bugs or need new features. A test framework becomes a vital component here for ensuring that we do not introduce more errors into our code or revert back to issues we have already resolved.
There are many instances when we may need to share data with our end users. This may be simply for examples; it may be data relevant to the field that we want to share, or it may be reference data required by functions in the code. This last point is particularly common in the development of code for analytics. Whatever the reason for needing to share the data, we can incorporate it all in our package so there is no need to also send out data separately to the package we have developed.
The next component we’ll implement is the user guide. Whether you are just sharing code with colleagues or you plan to share widely with the R community, the end users of your package are going to need to know how to use it. The individual function help files will help users with questions of “How do I use this function?” and “What are all the options for this function?” However, they will not typically help with the overall workflow of your package. A user guide is aimed at helping to get users started with a general workflow for your package. Just as with data, we have written this anyway and intended to simply email it to people who need it, but incorporating it in the package ensures that it is up to date and always available for the end users.
The final additional component we cover in this hour is C++ code, or more specifically, code we have written with Rcpp. This is not going to be a component that you will include in every package you write, but as you saw in Hour 18, “Code Efficiency,” you may have chosen to incorporate such code into a function for efficiency, so you need to know how to include such code in an R package.
As you can probably see, inclusion of these two components, data and C++ code, will be dependent upon the package itself and its requirements and implementation. When it comes to the user guide and unit tests, they are again optional. However, it is considered to be a best practice to include these components, and we would recommend that you get into the habit of including them as standard in any package you write. As you will see in this hour, they are very simple to add, with devtools functionality available to help you with the package structure, and they don’t take much additional effort once you are familiar with them.
Developing a Test Framework
Whenever we develop code, we test it in some way. As we start out this might just be with an ad-hoc running of a function to ensure it does what we expect. Usually this is with small amounts of data, and typically we test the main functionality we have implemented. As we write more code and begin to change it to handle any issues that arise, we might write a script that can be run regularly where there are known expected outputs we are looking for. This is the beginning of a test framework. For all development, but especially production development, it is recommended that these informal tests are formalized so that they can easily be re-run with specific cases at any point. We can then include these tests within a package so that they are always kept together, and even the end user can run them to ensure the package is still working as it should.
An Introduction to testthat
There are a number of options for providing a test framework in R, but the one introduced here, testthat, is both widely used and easy to get started with. Before we consider how to include tests in an R package, we will simply look at how to write what are known as “unit tests” using testthat.
As an example in this hour, we will implement tests for the function we included in the R package that we developed in the previous hour, sampleFromData. This function is defined in Listing 19.2 and simply randomly samples rows from a dataset we provide. You will also notice that this function includes some error handling by checking that sensible arguments have been provided.
While we write the tests, we will need to consider what we might test. We will return to this topic shortly, but for now we will simply write some tests to ensure that data is returned as expected. If we were to ask you to check that this function worked correctly, you would most likely pick a simple dataset and test the function with argument values that are easy to check the output of. For example, you might try the following:
> library(mangoTraining)
> set.seed(20)
> testData <- sampleFromData(demoData, 3)
> testData
Subject Sex Age Weight Height BMI Smokes
29 29 M 44 81 175 26.4 Yes
26 26 F 25 58 175 18.9 No
10 10 M 23 71 188 20.1 No
In this case, we have used the function set.seed, which allows us to set the value of the random seed to ensure that we can consistently reproduce the random sampling in this function. However many times we run all these lines of code, we will consistently reproduce these same sampled rows. This is a really useful function when it comes to testing. We use testthat to formalize this test and to check for us that the correct data is returned.
We create individual tests using functions named with the pattern expect_. The names are then appended with elements such as equal, named, is, and error, among others. All of the functions follow a similar pattern whereby we provide the object we want to test as the first argument, followed by the value we want to test against as the second argument. In the preceding example, we might ensure that the correct three rows are returned with tests such as these:
> expect_is(testData, "data.frame")
> expect_named(testData, c("Subject", "Sex", "Age", "Weight", "Height", "BMI", "Smokes"))
> expect_equal(testData[,"Subject"], c(29, 26, 10))
So we have checked that the correct structure is returned, that it has the correct columns, and that the elements of the Subject column are correct. We could extensively test the whole returned structure, but in this case, because rows are unique based on the subject number, we can be confident that the same data has been returned if the subject values are the same each time. You will notice that if you run all of these statements, nothing is returned when the output is as expected. Only if the test fails will you see any output.
We can write such statements to test a range of functionality in the sampleFromData function. Typically we want to test that arguments work as expected and change the output in some way, and we want to ensure that errors and warnings are thrown when expected. It is also highly recommended that we write a test for what the correct behavior should be whenever a bug is identified. This will help us to resolve the bug and ensure that we don’t do anything that puts the bug back into our code.
Rather than simply writing a script full of expect_ statements, we use a function called test_that to group expectations together. Therefore, we would typically group the statements we wrote previously as a test for expected default behavior, for instance, which would mean that our test script might look something like the example given in Listing 20.1.
LISTING 20.1 Example of a Test Script for sampleFromData
1: context("sampleFromData must return data frames of the correct format")
2:
3: test_that("Default arguments return correctly", {
4:
5: require(mangoTraining)
6:
7: set.seed(20)
8:
9: testData <- sampleFromData(demoData, 3)
10:
11: expect_is(testData, "data.frame")
12:
13: expect_named(testData,
14: c("Subject", "Sex", "Age", "Weight", "Height", "BMI", "Smokes"))
15:
16: expect_equal(testData[,"Subject"], c(29, 26, 10))
17:
18: })
19:
20: test_that("Throws an error correctly", {
21:
22: expect_error(sampleFromData(airquality, "Subject"),
23: "Size must be a numeric integer value")
24:
25: })
You can see that lines 5 to 16 are the same as we previously ran, but this time they are inside the test_that function. As you can see on line 3, the first argument is a character string to indicate what the purpose of this group of tests is, and the second is the group of code, contained inside curly brackets, that is to be run, including all of the expectations. In this example, we have included a second test_that function call that we are using to test that the function handles errors correctly. We can have as many test_that groups as we want in a single script. It is a best practice to collect test_that statements in a script for a single function or group of functionality so that tests are organized in an easy-to-find way. We will look in the next section at how to structure tests for a package.
You will also notice in this example that on line 1 we have called a function context and that it contains a character string. This is simply a way of grouping together a series of test_that statements. The context indicates that all of the following tests are related to a specific piece of functionality—in this case, the sampleFromData function.
When it comes to running these tests, we can make use of the functions test_file and test_dir. The function test_file will run all of the tests in a single file, whereas the function test_dir will run all of the scripts in a single directory. As an example, suppose that we had saved the code in Listing 20.1 as the file test-sampleFromData.R. We would run all of these tests with the following lines:
> test_file("test-sampleFromData.R")
sampleFromData must return data frames of the correct format : ....
Notice that the context has been used to label all the tests that have been run, and the . in the output indicates that a test has been run and has passed.
Incorporating Tests into a Package
Although we could simply write tests in a script that we can run as we did earlier, if we are writing a package it is much better to include the tests in the package. This way, we always know where to find tests for specific code, we can very easily re-run the tests for the whole package after we have made changes, and we can easily provide the tests to others who may want to re-run them. This final point is quite common in controlled environments where it is necessary to be certain that there have not been changes to the software or environment that impact the results of running specific code.
As you saw in the last hour, components of a package are structured in a specific way, and tests are no exception. Although the devtools functions we have seen so far have not created this for us, we can add a test structure to a package we have already created with the function use_testthat. Thus, to add the test structure to the package we started to develop in Hour 19, “Package Building,” we can run the following line:
> use_testthat("../simTools")
This will create in the package structure a directory called “tests,” which contains a file, testthat.R, that houses the required code to run the tests for the package and doesn’t need to be changed, as well as a directory called “testthat.” It is in this directory that we should store all our test scripts. We can include as many or as few scripts as we want, but all files need to start with “test-”.
When you use devtools to set up the correct package structure for tests, you will also find that it updates the DESCRIPTION file to include testthat as a suggested package. The package is only included as a suggestion because it is not a requirement to have testthat to run your code; however, if someone wants to run your tests, they will need this package.
Once we have included tests inside our package, we no longer need to use the test_file and test_dir functions in testthat to run them. As with all the other components of package building, we can run the tests from RStudio using the Build tab options, or we can use the devtools function test. Running the tests in the simTools package would become
> test("../simTools")
Testing simTools
sampleFromData must return data frames of the correct format : ....
As you can see, the output is just the same as if we had run test_file. When we have structured the tests in this format, they will be run when we run the package checks from Hour 19. However, it is good practice to run your tests before this point if you have made changes to the code so that you don’t get to building your package before you realize that you have introduced an error. Given the ease with which we can run tests inside a package, it won’t take a lot of effort to run test on a regular basis.
Tip: Test-Driven Development
One means of code development that you might find useful is an idea known as test-driven development. In this approach to development, we start by writing tests for what we want our package to do that will initially fail, and then we develop the code. When the test passes, we have completed that component. This is a useful way to develop code if you have a large number of requirements or if you are adding requirements, because you can always see what you have done so far and what is left to do.
Including Data in Packages
As you will know from using other R packages, it is not only code that can be incorporated but also data. This is useful if you have a dataset that you want to be able to use for examples or that you want to make available to others for a specific purpose or even as a reference dataset for functionality in your package. Just as with all other components of a package, we can use devtools to simplify adding data to a package.
Where we add the data will depend on what its purpose is. Data that we want to be available to end users or available for examples or user guides should be stored in the “data” directory. If we haven’t yet added data, this won’t exist in our package structure but will be added when we run the use_data function. This function, in the devtools package, both sets up the correct structure and adds the data we want to include in an appropriately compressed format. The dual purpose of this function means that it is slightly different in usage from other devtools functions for which we simply provide the file path to the package. As an example, let’s create a simple dataset that we will add to our package:
> exampleData <- data.frame(ID = 1:10, Value = rpois(10, lambda = 5))
> use_data(exampleData, pkg = "../simTools")
Saving exampleData to data/exampleData.rda
You will notice here that in the use_data function, we have first listed the data objects we want to have included. Because we can provide any number of data objects, we need to specify the package in which to include the data using the pkg argument. This will create the “data” directory for us as well as compress the data and add it to the package structure.
With the data in this format, we can now load the package and see the data, just as we use data in any other package, by giving the name of the data set. Note that it retains the name we gave the object when it was created (in this case, exampleData).
If you were to run the package checks now, you would find that this creates a warning in the check because any object that can be seen by the user must have a corresponding help file. So the next step we need to take is to provide the documentation. As you saw in Hour 19, we can use roxygen2 to create package documentation, and this extends to help files for data sets. This is very similar to how we document a function, but we use an alternative tag, @format, to describe the structure of the dataset. In addition, rather than giving the function call after the header, we give the name of the dataset. As an example, consider Listing 20.2, where we have created simple documentation for the dataset we just added to the package. This header needs to be saved in an R script in the R directory. As discussed in the previous hour, the naming of these files is up to you but it is generally good practice to name so that it is easy to identify the file.
LISTING 20.2 Roxygen Header for a Dataset
1: #' Simple example of including data
2: #'
3: #' This is a simple example of how we can include data in a package
4: #' and provide the corresponding documentation.
5: #'
6: #' @format A data.frame with 10 rows and two columns:
7: #' describe{
8: #' item{ID}{Unique identity variable}
9: #' item{Value}{Simulated value (g)}
10: #' }
11: #'
12: #' @source Simulated data
13: "exampleData"
You will see that we have documented each column of the data. It is a good idea here to state what the column of data contains as well as any units relevant to that column—for instance, “inches” or “pounds” if you were giving measurements of distance or weight. You might also notice in this example that we have used the tag @source, which is a handy way of detailing where the data came from—obviously, in this case, the data was simply simulated, but this may be details of the location of the original data.
Tip: Adding More Data
We can still use the use_data function to add datasets later in the package development, even if we have already set up the package structure. We use the function in the same way, but the function itself won’t create (or overwrite) a data directory.
If we want to include reference data that is used by a function in our package but is not visible to the end user, we save the data in a file named sysdata.rda in the R directory. Again, we can use the use_data function to incorporate such data, but in this instance we add the argument internal = TRUE. Unlike the user-visible data in the data directory, we do not need to document this data. Including a dataset in this way would look like the following:
> hiddenData <- data.frame(ID = 1:5, Ref = rnorm(5))
> use_data(hiddenData, pkg = "../simTools", internal = TRUE)
Saving hiddenData to R/sysdata.rda
Including a User Guide
In R, a user guide is typically referred to as a vignette and is typically a means of extending the package help files to describe the typical workflow of your package or to give extended details of what you have implemented in your package. If you are sharing your package with others, you will typically need to provide some form of documentation to help them get started. By including this in the package itself, you can be sure that it is always available to the users, that you can easily keep it up to date, and that the code in the vignette actually runs without error because it is checked as part of the package checks.
You can see the vignettes available for a package by using the browseVignettes function. This will allow you to navigate vignettes for all packages or for a specific package. Here is an example:
> #browse all vignettes
> browseVignettes()
> # browse for a specific package
> browseVignettes("roxygen2")
A package can include multiple vignettes, which is useful if you want to include more detailed information about specific components of your package.
Including a Vignette in a Package
When it comes to writing a vignette, we now have multiple options for the tools we use. Traditionally we used Sweave, which requires knowledge of LaTeX, a markup language that allows us to combine text, R code, and mathematical expressions. Since R version 3.0.0, we can use any package to create a vignette that can produce HTML of PDF files. This means that we can now use the package knitr, which allows us to use R Markdown for our vignettes. In this section, we will look at how to incorporate a vignette in a package and get started with creating one.
As with all other aspects of our package, we are going to use devtools to help us get started. It is now a best practice to include package vignettes in a vignettes directory. We can of course create this directory directly; however, the use_vignette function will not only create that directory but it will add all the required components to the DESCRIPTION file, and it will create a template vignette file for us to start working with. To get started on a quick-start guide to using our simTools package, we would run the following line:
> use_vignette("QuickStart", pkg = "../simTools")
The first argument here gives the name of the vignette that we want to create so that the template file takes the correct filename. There will now be a vignette directory containing the file QuickStart.Rmd. You will also find that the package knitr has been added to the list of suggested packages in the DESCRIPTION file and that a new field, VignetteBuilder, will also have been added with knitr listed as the required package to build the vignette.
The vignette file incorporated in your package will be checked when you run the usual package checks, and it will be built into an HTML file when you build the package. During development of the vignette itself, the easiest way to preview the file you are creating is to simply use the Knit button in RStudio. First of all, open the file that was created for you. This is a “.Rmd,” or RMarkdown, file. We will return to how to write this in the next section, but you will initially find that the file has been populated with some sample text. In RStudio, opening this file will have given you some alternative options across the top of the file viewer, one of which being “Knit.” Selecting this option, you will build the file into the corresponding HTML file and a preview will be opened in the Viewer tab.
Tip: Building Vignettes Without RStudio
If you don’t want to use the built-in options in RStudio, you can build your vignettes by running the function build_vignettes in the devtools package. This is used the same as other devtools functions, passing the package as the main argument. This will create the directory inst/docs, which will contain the .Rmd file, an R script, and the built HTML vignette.
Writing a Vignette
R Markdown is simple to read and write markup language that allows us to incorporate text, R code, and output in a single file. In this section, we introduce the basics of markdown. For more details on creating documentation and reports in R, see Hour 23, “Dynamic Reporting.”
Because the step we took in the previous section created a sample file for us, we will start with this. All R Markdown documents use a header at the top of the file to give details such as the title, author, and date, as well as details on the type of file to generate. For a vignette we also have some extra components. Listing 20.3 shows what this template header looks like. As you can see, we have the title and author components that we can update as well as the date (which in this case updates dynamically). We can optionally remove these components, if we don’t want the date to appear, for instance. The remainder of the header gives instructions relating to building the vignette and creating an index of vignettes, as we saw when we ran browseVignettes. The only thing that we need to change here is on line 7, where we need to update the Vignette Title text to match the title on line 2.
1: ---
2: title: "Vignette Title"
3: author: "Vignette Author"
4: date: "`r Sys.Date()`"
5: output: rmarkdown::html_vignette
6: vignette: >
7: %VignetteIndexEntry{Vignette Title}
8: %VignetteEngine{knitr::rmarkdown}
9: %VignetteEncoding{UTF-8}
10: ---
The actual content of the guide is up to you to determine, but a useful guide to produce would walk the user through the main workflow. How do you get started using your package? What are the main functions in your package that a user should look at? There is no need to go into all of the details about all the function arguments, but this type of guide will point a user in the right direction, and they can then use your function help files for more details. As an example, we might produce a guide for our simTools package that guides the user to the sampleFromData function as a starting point for their simulation.
When it comes to starting to write the document, we need to know the basics of markdown. It is quite a limited markup language, but that shouldn’t prevent you from being able to create a functional user guide to your package. Some examples of markdown syntax can be seen in Table 20.1.
An example of how a user guide for the simTools package might look can be seen in Listing 20.4. You will notice that the file created for us by devtools contains text, which can be deleted, and that also includes examples of many of these features.
LISTING 20.4 Example of User Guide Content
1: This guide is intended as a means of quickly getting started with the package
2: **simTools**. It will introduce the main workflow of the package.
3:
4: ## Getting Started
5:
6: The main function in the **simTools** package is `sampleFromData`. This function will
7: allow you to generate random samples from a given data set. It is useful for
8: simulation experiments.
9:
10: ### Loading the package
11:
12: Before starting you will need to load the package in the usual way using either
13: `library` or `require`.
14:
15: ### Running the main function
16:
17: Once the package is loaded we can run the function as follows:
One of the main components of interest to the reader of your vignette will be examples of code and how to run the functions in your package. We include code in vignettes in special code blocks. An example of a code block is shown in Listing 20.5. We use the triple back ticks to mark the start and end of the code block, as you can see on lines 1 and 5. You will also notice the {r} after the back ticks on line 1. This indicates that the code in this block should be executed as R code. We can also include options for the code block inside these curly brackets. We will return to this in Hour 23.
LISTING 20.5 Including a Code Block
1: ```{r}
2: library(mangoTraining)
3: example1 <- sampleFromData(demoData, size = 5)
4: example1
5: ```
Inside the code block we can include any executable code we want, including code that generates graphics. Note that the code will be checked during the standard package checks as well as the build, and any packages used to run examples in the vignette need to be included in the suggests field in your DESCRIPTION file as a minimum. When this code block is included in our vignette, it will include not only the code run but also the output generated. An example of how the code block in Listing 20.5 would be rendered is shown in Figure 20.1.
We can include as much text and as many code blocks as we want into a vignette, but it is worth remembering the reader. If you find your vignette is quite long, you may want to split it into multiple files so it does not seem as long and difficult to read. However, this is entirely up to you.
Code Using Rcpp
You saw in Hour 18 that we could easily incorporate code written in C++ using the package Rcpp. If we have done this and we then wanted to put that code into our packages, we would need to know how to include the code in our packages. As you have seen in the previous section, devtools has simplified all aspects of incorporating additional package components, and the function use_rcpp will help us at this point.
Any source code that is not R code is included in a directory called src. The use_rcpp function will create this directory for us, along with handling the updating of the DESCRIPTION file. As an example, in our simTools package we would run the following:
> use_rcpp("../simTools")
Adding Rcpp to LinkingTo and Imports
Creating src/ and src/.gitignore
Next, include the following roxygen tags somewhere in your package:
#' @useDynLib simTools
#' @importFrom Rcpp sourceCpp
You will notice that this also tells us to add some roxygen tags in the package. You can include this anywhere in the package, but the most sensible place would be in the overall package help file. These two tags will ensure that the C++ code is loaded when the package is loaded.
At this point, we can include the .cpp files, which we discussed in Hour 18, in the source directory. As an example, suppose that we included the sampleInC function that we wrote in Listing 18.5 of Hour 18 in our package. Including this in a .cpp file in the src directory with the same structure that we saw previously, we cause the check and build process for the R package to create the appropriate additional files in both the src and R directories for us. If we are simply using this function in other R functions and we do not intend the end user to see the function, this is all we need to do and we can start to use the function in our code. The function will not be exported but will be available to any code that requires it.
If we want to export this function to be visible to the end user, we will need to include an equivalent roxygen header in the .cpp file. This will be identical to the headers for R functions as we saw in Hour 19, but we use the C++ comment character to indicate the header rather than the R comment character. An example of what the file header would look like can be seen in Listing 20.6.
LISTING 20.6 Including a Code Block
1: #include <Rcpp.h>
2: using namespace Rcpp;
3:
4: //' Sample a series of 0s and 1s
5: //'
6: //' @param len A single integer giving the final length.
7: //' @export
8: // [[Rcpp::export]]
After you have updated the file, you will need to update the package documentation in the usual manner before building your package. You will then have the function sampleInC available to the end user and a corresponding help file for the user to reference. Of course, just like R functions, it is beneficial to include this header for all functions but simply omit the @export tag if you do not want the function to be available to the end user.
Summary
In this hour, you saw how to improve packages, making them more robust, user friendly, and easier to manage. Although these components are not a requirement of a package, they are considered to be best practices, and we would recommend that you get into the habit of structuring your packages in this way, in particular with tests and user guides. In the next hour, we will introduce classes and how to develop our own classes to make code more robust and user friendly.
Q&A
Q. Do I really need to include tests? Isn’t it going to take a long time?
A. You do not need to include any of the package components mentioned in this hour; however, it is good practice to include tests and a vignette. Tests will help you to ensure the quality of your code and make it much easier to make changes to the code in the future knowing that they will not impact the code adversely. The first time you write tests it may take you longer as you get used to the structure, but this will quickly become second nature, and if you do it as you write the code rather than all at the end, it won’t add much to the development time.
Q. Can I include data in a .csv file in my package?
A. Yes, you can include any raw data file that you like in your package, but this is done in a slightly different way. In this case, you should create a directory in the inst directory to contain the data (for instance, inst/extdata). You can then access this data using the system.file function and pointing to the rawdata directory of the package, like so:
system.file("extdata", "myFile.csv", package = "simTools")
Q. I know LaTeX. Can I use this for my vignette instead of markdown?
A. Yes, you can. You simply create your vignette in an .Rnw file rather than an .Rmd file. You will need to include lines 7–9 in Listing 20.3 in your document header.
Workshop
The workshop contains quiz questions and exercises to help you solidify your understanding of the material covered. Try to answer all questions before looking at the “Answers” section that follows.
Quiz
1. Why should you include tests in a package?
2. How would you include data in your package that is not intended to be seen by the end user?
3. What are user guides known as in R?
4. What is the simple markup language that you can use for vignettes?
5. In which directory do you put C++ code?
Answers
1. Tests help you to ensure that your code does what it is meant to do. If you make changes to the code, you can re-run the tests to ensure that the code still runs as expected. You can also write tests for any bugs you identify so that you can continually check that they don’t end up back in your code due to changes that you make.
2. You can include data in your package using the use_data function. You can ensure that this is only available to the package by using the argument internal = TRUE, which will store the data in the R directory rather than the data directory.
3. Longer user guides in R are referred to as vignettes. You can see all of the package vignettes by using the browseVignettes function.
4. You use the markup language markdown. We can also use LaTeX for writing vignettes.
5. Any C++ code, or other compiled code, is included in the src directory.
Activities
1. In the activities for the last hour we developed a package called summaryTools and we wrote two functions for this package. Using the methods introduced in this hour, add a test framework and tests for each of the functions you created.
2. Update both functions to include some simple error checking of the arguments. Ensure that the tests you have written still pass, and add further tests to test the error handling.
3. Create a simple dataset, summaryData, that contains three columns: ID, which should be a numeric factor that is unique for each row; Group, which is a random sample of the values “A” and “B” to identify the group each value is in; and finally “Observed,” which is a sample from a random normal distribution.
4. Include this data in your package and ensure that it is well documented.
5. Create a simple vignette for your package that explains how the user should run your functions.
6. Rebuild and check your package, ensure that all tests pass, and that you can access the data and vignette once your package is loaded.