Development

First, 🎉THANK YOU🎉.

Grouparoo relies on the contributions of many people just like you to create the best marketing infrastructure & automation tools. Thank you for taking the time to join this open source community, and create something we can all use! By contributing to Grouparoo, you agree to our Community Guidelines and Code of Conduct

Getting Started

When developing Grouparoo, you'll be running code within the grouparoo monorepo. This git repository contains a number of related bu separate projects. There are Apps, CLI, Core, Plugins, along with a number of tools and utilities. We use pnpm to install packages to allow us to run and test all of these packages together without needing to publish them to NPM. We use lerna to publish packages.

• Apps: These are example applications like a Grouparoo customer would run - they require @grouparoo/core and any number of plugins and may have toggled certain settings
• Core: The @grouparoo/core project is the main part of any Grouparoo application and includes the API (both web and workers).
• CLI: The Grouparoo CLI tool, ie: grouparoo init .
• Plugins: There are opt-in add-ons for Grouparoo customers, and contain the logic to connect to new sources & destinations, or otherwise add new functionality to Grouparoo.
• UI: The grouparoo UI plugins, ui-community and ui-enterprise and shared component libraries.

Running the Monorepo

See below for the installation guide for your operating system.

• MacOS
• Linux/Ubuntu
• Windows

# Install NVM to manage node versions
# From: https://github.com/nvm-sh/nvm
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash

# Install the homebrew package manager
# From: https://brew.sh/
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

# Install Node.js v16
nvm install v16

# Depending on your shell (bash, zsh, etc) you may need to ensure that nvm is loaded into your environment via .bashrc, .bash_profile, etc
# On OSX/ZSH, we recommend installing Oh My ZSH (https://github.com/ohmyzsh/ohmyzsh) and enabling the nvm plugin

# Clone this repository
git clone git@github.com:grouparoo/grouparoo.git
cd grouparoo

# Install with pnpm (If you don't have pnpm, install it via "npm i -g pnpm")
pnpm install

# Configure your local environment variables in .env to be able to run the "staging-enterprise" app
cp apps/staging-enterprise/.env.example apps/staging-enterprise/.env
open apps/staging-enterprise/.env

# Install & run Redis
brew install redis
brew services start redis

# Install & run postgresql (v9.x)
brew install postgresql
brew services start postgresql

# Create your local development database (Postgres)
createdb "grouparoo_development"

# or in `psql`
CREATE DATABASE grouparoo_development;
GRANT ALL PRIVILEGES ON DATABASE grouparoo_development TO postgres;

# Run the staging-enterprise app example project
cd apps/staging-enterprise # If you aren't already here
npm run dev

# use the redis official Docker image:
docker run -d --rm --name grouparoo-redis -p 6379:6379 redis

# use the postgres official Docker image:
docker run -d --rm --name grouparoo-postgres -p 5432:5432 -e POSTGRES_HOST_AUTH_METHOD=trust -e POSTGRES_DB=grouparoo_development -e POSTGRES_USER=$USER postgres

# Run the staging-enterprise app example project
cd apps/staging-enterprise # If you aren't already here
npm run dev

# Install NVM to manage node versions
# From: https://github.com/nvm-sh/nvm
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash

# Install Node.js 16
nvm install v16

# Clone this repository
git clone git@github.com:grouparoo/grouparoo.git
cd grouparoo

# Install dependencies with pnpm (If you don't have pnpm, install it via "npm i -g pnpm")
pnpm install

# Configure your local environment variables in .env to be able to run the "staging-enterprise" app
cp apps/staging-enterprise/.env.example apps/staging-enterprise/.env
nano apps/staging-enterprise/.env

# Install & run Redis
sudo apt update
sudo apt install redis-server

# Open this file with your preferred text editor:
sudo nano /etc/redis/redis.conf

# comment the line:
supervised no

# add the following line bellow the commented line:
supervised systemd

# Install & run postgresql (v9.x)
sudo apt update
sudo apt install postgresql postgresql-contrib

# Create your local development database (Postgres)
createdb "grouparoo_development"

# or in `psql`
CREATE DATABASE grouparoo_development;
GRANT ALL PRIVILEGES ON DATABASE grouparoo_development TO postgres;

# use the redis official Docker image:
docker run -d --rm --name grouparoo-redis -p 7489:6379 redis

# use the postgres official Docker image:
docker run -d --rm --name grouparoo-postgres -p 5432:5432 -e POSTGRES_HOST_AUTH_METHOD=trust -e POSTGRES_DB=grouparoo_development -e POSTGRES_USER=$USER postgres

# Run the staging-enterprise app example project
cd apps/staging-enterprise # If you aren't already here
npm run dev

# TODO

Typescript

Grouparoo is written in Typescript, which means that we need to compile our Typescript to Javascript before we can run it. This is true for Grouparoo plugins, but not necessarily true for Grouparoo Apps or Core, if you are using npm run dev (development mode), which will handle the compilation on the fly. Every sub-project in the Grouparoo Monorepo has an npm run prepare command which handles the compilation process. npm run prepare is also an NPM lifecycle hook, which will automatically run after every npm install as well.

If you are developing a Plugin, you will need to npm run prepare it before seeing your changes. In many cases, the prepare step only calls out to tsc, and in these cases, you may be able to tsc --watch to automatically source changes and rebuild your Plugin.

Grouparoo self-documents its Typescript interface to docs.grouparoo.com via Typedoc. The docs from the main branch are automatically published to this site. If you want to read the docs for an older version of Grouparoo or a different branch, you can generate the docs locally by running cd core && npm run docs in this repository.

Architecture

Learn about the Grouparoo architecture and models from the other pages in this documentation section.

Dependencies

Grouparoo is built on top of many wonderful open source technologies. An exhaustive list of our dependencies can be found within our package.json files. This below list provides links to the core packages we rely on.

Grouparoo Core:

• General
  • pnpm
  • Lerna
  • Jest
• API
  • Actionhero
  • Sequelzie-Typescript
  • Sequelzie
  • Node Resque
• Web
  • Next.js
  • React
  • React-Bootstrap
  • fontawesome

All dependencies Grouparoo uses must be licensed under a non-viral open-source licence that allows for commercial use. We audit every dependency in every sub-project of this monorepo via license-checker. The list of licenses we accept can be found here.

Populating Demo Data

The @grouparoo/demo Plugin makes it easy to generate sample data! You can use the staging-community App to try things out:

cd apps/staging-community
# populate the system with two models and create config.
# users: 1000 user records, properties, 1000 purchases, and some groups.
# admins: 25 admin records, properties
grouparoo demo -c

# including the --scale param allows you to control how many records you make. e.g. --scale 10 makes 10,000 extra users. create config.
grouparoo demo -c --users --scale 10

Note: These commands will clear all existing data in your Grouparoo database, replacing it with the sample data.

You can then use the following credentials to log in:

• Username: demo@grouparoo.com
• Password: password

Here is the full set of options available:

➜ grouparoo demo -h
Usage: grouparoo demo [options]

Load data into a source database, create properties, destinations, and other config.
A demo@grouparoo.com Team Member is created. Password: "password"

Options:
  --scale [scale]              make the number more than 1 to multiple amount of data. (default: "1")
  --junkPercent [junkPercent]  what percent of the data should have problems? (default: "0")
  -c, --config                 add flag to write to config directory and not populate configuration into Grouparoo database
  --reset                      only clear Grouparoo database and don't load config
  --seed                       add flag to only write (or output) demo source data and not touch Grouparoo database
  --setup                      only create the login Team Member
  --b2c                        (default) loads users and admins models
  -b, --b2b                    loads account model
  -u, --users                  includes users model with purchases
  -a, --accounts               includes accounts model with with payments
  --admins                     includes admins model
  --postgres                   (default) load source data into local Postgres database
  --mongo                      load specified source data into local MongoDB database
  --mysql                      load specified source data into local MySQL database
  --snowflake                  assumes a Snowflake instance with data already present
  --bigquery                   assumes a BigQuery instance with data already present
  -m, --mailchimp              create Mailchimp Destination for data
  -s, --salesforce             create Salesforce Destination for data
  -h, --help                   display help for command

Testing

Grouparoo uses Jest as our testing framework. To run tests, first make sure you've installed all dependencies in the root of the project.

pnpm install # we use pnpm to manage the monorepo

You will need chromedriver installed to run the automated browser tests as well:

• OSX: brew install --cask chromedriver
• Windows: TODO
• Linux: TODO

Next, create the test databases:

cd core
./bin/create_test_databases

Running npm test from the top-level of this monorepo will run the npm test command in every sub-package of the monorepo via pnpm. You can also test an individual package by entering that directory and running npm test. If you want to run a single test file you can ./node_modules/.bin/jest /path/to/test and you can watch files with jest's --watch flag.

Jest will automatically test the latest version of your typescript files, and you do not need to compile your code to test. However, the test suite should test the compile step as well. We usually run npm run prepare as a pretest NPM hook.

A note on chromedriver for OSX Users: As Chrome updates, you may need to brew reinstall --cask chromedriver to keep up to date as well. Every time you update chromedriver, you will need to re-authorize the application to run in system preferences, as it is unsigned.

@grouparoo/spec-helper

The Plugin @grouparoo/spec-helper exists to make testing easier, especially within plugins. It contains factories, startup/shutdown helpers for the application, and reasonable testing defaults. Unlike a traditional node module, including this package into a test suite will have side-effects - specifically your working directly will be changed into @grouparoo/core so you can run the application like normal in your tests. You can see what happens here.

import path from "path";
// First, set an environment variable so that @grouparoo/core will load this Plugin if you are doing an integration test
process.env.GROUPAROO_INJECTED_PLUGINS = JSON.stringify({
  "@grouparoo/hubspot": { path: path.join(__dirname, "..", "..") },
});

// Then, include the spec helper
import { helper } from "@grouparoo/spec-helper";
// Note! Because of tree-shaking, if you don't actually *use* what you've imported form @grouparoo/spec-helper in your test file, the import statement will be ignored.  In that case, do a raw import of the file:
// import "@grouparoo/spec-helper";

// import other things, including @grouparoo/core if you need it
import { GrouparooRecord } from "@grouparoo/core";

describe("my plugin", () => {
  // This command injects a `beforeAll` and `afterAll` to startup and shutdown the server.  There are lots of options to help you with testing.  If you need access to Models, you need to turn on the server so it can connect to Postgres
  helper.grouparooTestServer({ truncate: true, enableTestPlugin: true });

  // Now the server will be booted up, connected to the database, and you can use factories and models like normal
  test("works", async () => {
    const factory = await helper.factories.record();
    const record = await GrouparooRecord.findOne({ where: { id: factory.id } });
    expect(record.id).toBe(record.id);
  });
});

Notes on Example Apps

• If you create a new example App, you will need to pnpm install again to link up the dependencies in the monorepo.
• Be sure that your pacakge.json's name field matches the folder name.
• Be sure that package.json's grouparoo.grouparoo_monorepo_app matches the folder name.

Deploying the Example Server

On the server:

1. Ensure you have the proper environment variables set
2. Run pnpm install at the root of this project, which will run the build steps too
3. Run cd apps/staging-public && npm run start

See the Procfile for an example of how to run a web and worker process independently. If you are deploying to Heroku or a similar PaaS, they might prune node_modules that aren't explicitly in "dependencies". However, due to the way lerna works, this might delete all of your modules. Check with your provider for how to disable this (ex: https://devcenter.heroku.com/articles/nodejs-support#build-behavior)