Skip to content

Latest commit

 

History

History
300 lines (232 loc) · 17.7 KB

README.md

File metadata and controls

300 lines (232 loc) · 17.7 KB
Raw

Release CI License Registry Source

terraform-google-enrich-pubsub-ce

A Terraform module which deploys the Snowplow Enrich PubSub service on Compute Engine. If you want to use a custom image for this deployment you will need to ensure it is based on top of Ubuntu 20.04.

Telemetry

This module by default collects and forwards telemetry information to Snowplow to understand how our applications are being used. No identifying information about your sub-account or account fingerprints are ever forwarded to us - it is very simple information about what modules and applications are deployed and active.

If you wish to subscribe to our mailing list for updates to these modules or security advisories please set the user_provided_id variable to include a valid email address which we can reach you at.

How do I disable it?

To disable telemetry simply set variable telemetry_enabled = false.

What are you collecting?

For details on what information is collected please see this module: https://github.com/snowplow-devops/terraform-snowplow-telemetry

Usage

Standard usage

Enrich PubSub takes data from a raw input topic and pushes validated data to the enriched topic and failed data to the bad topic. As part of this validation process we leverage Iglu which is Snowplow's schema repository - the home for event and entity definitions. If you are using custom events that you have defined yourself you will need to ensure that you link in your own Iglu Registries to this module so that they can be discovered correctly.

By default this module enables 5 enrichments which you can find in the templates/enrichments directory of this module.

module "raw_topic" {
  source  = "snowplow-devops/pubsub-topic/google"
  version = "0.3.0"

  name = "raw-topic"
}

module "bad_1_topic" {
  source  = "snowplow-devops/pubsub-topic/google"
  version = "0.3.0"

  name = "bad-1-topic"
}

module "enriched_topic" {
  source  = "snowplow-devops/pubsub-topic/google"
  version = "0.3.0"

  name = "enriched-topic"
}

module "enrich_pubsub" {
  source = "snowplow-devops/enrich-pubsub-ce/google"

  accept_limited_use_license = true

  name = "enrich-server"

  network    = var.network
  subnetwork = var.subnetwork
  region     = var.region

  raw_topic_name = module.raw_topic.name
  good_topic_id  = module.enriched_topic.id
  bad_topic_id   = module.bad_1_topic.id

  ssh_key_pairs    = []
  ssh_ip_allowlist = ["0.0.0.0/0"]

  # Linking in the custom Iglu Server here
  custom_iglu_resolvers = [
    {
      name            = "Iglu Server"
      priority        = 0
      uri             = "http://your-iglu-server-endpoint/api"
      api_key         = var.iglu_super_api_key
      vendor_prefixes = []
    }
  ]
}

Inserting custom enrichments

To define your own enrichment configurations you will need to provide the enrichment in the appropriate placeholder.

locals {
  enrichment_anon_ip = <<EOF
{
  "schema": "iglu:com.snowplowanalytics.snowplow/anon_ip/jsonschema/1-0-1",
  "data": {
    "name": "anon_ip",
    "vendor": "com.snowplowanalytics.snowplow",
    "enabled": true,
    "parameters": {
      "anonOctets": 1,
      "anonSegments": 1
    }
  }
}
EOF
}

module "enrich_pubsub" {
  source = "snowplow-devops/enrich-pubsub-ce/google"

  accept_limited_use_license = true

  name = "enrich-server"

  network    = var.network
  subnetwork = var.subnetwork
  region     = var.region

  raw_topic_name = module.raw_topic.name
  good_topic_id  = module.enriched_topic.id
  bad_topic_id   = module.bad_1_topic.id

  ssh_key_pairs    = []
  ssh_ip_allowlist = ["0.0.0.0/0"]

  # Linking in the custom Iglu Server here
  custom_iglu_resolvers = [
    {
      name            = "Iglu Server"
      priority        = 0
      uri             = "http://your-iglu-server-endpoint/api"
      api_key         = var.iglu_super_api_key
      vendor_prefixes = []
    }
  ]

  # Enable this enrichment
  enrichment_anon_ip = local.enrichment_anon_ip
}

Disabling default enrichments

As with inserting custom enrichments to disable the default enrichments a similar strategy must be employed. For example to disable YAUAA you would do the following.

locals {
  enrichment_yauaa = <<EOF
{
  "schema": "iglu:com.snowplowanalytics.snowplow.enrichments/yauaa_enrichment_config/jsonschema/1-0-0",
  "data": {
    "enabled": false,
    "vendor": "com.snowplowanalytics.snowplow.enrichments",
    "name": "yauaa_enrichment_config"
  }
}
EOF
}

module "enrich_pubsub" {
  source = "snowplow-devops/enrich-pubsub-ce/google"

  accept_limited_use_license = true

  name = "enrich-server"

  network    = var.network
  subnetwork = var.subnetwork
  region     = var.region

  raw_topic_name = module.raw_topic.name
  good_topic_id  = module.enriched_topic.id
  bad_topic_id   = module.bad_1_topic.id

  ssh_key_pairs    = []
  ssh_ip_allowlist = ["0.0.0.0/0"]

  # Linking in the custom Iglu Server here
  custom_iglu_resolvers = [
    {
      name            = "Iglu Server"
      priority        = 0
      uri             = "http://your-iglu-server-endpoint/api"
      api_key         = var.iglu_super_api_key
      vendor_prefixes = []
    }
  ]

  # Disable this enrichment
  enrichment_yauaa_enrichment_config = local.enrichment_yauaa
}

Requirements

Name Version
terraform >= 1.0.0
google >= 3.44.0

Providers

Name Version
google >= 3.44.0

Modules

Name Source Version
service snowplow-devops/service-ce/google 0.1.0
telemetry snowplow-devops/telemetry/snowplow 0.5.0

Resources

Name Type
google_compute_firewall.egress resource
google_compute_firewall.ingress_ssh resource
google_project_iam_member.sa_logging_log_writer resource
google_project_iam_member.sa_pubsub_publisher resource
google_project_iam_member.sa_pubsub_subscriber resource
google_project_iam_member.sa_pubsub_viewer resource
google_project_iam_member.sa_storage_object_viewer resource
google_pubsub_subscription.in resource
google_service_account.sa resource

Inputs

Name Description Type Default Required
bad_topic_id The id of the bad pubsub topic that enrichment will insert data into string n/a yes
good_topic_id The id of the good pubsub topic that enrichment will insert data into string n/a yes
name A name which will be pre-pended to the resources created string n/a yes
network The name of the network to deploy within string n/a yes
project_id The project ID in which the stack is being deployed string n/a yes
raw_topic_name The name of the raw pubsub topic that enrichment will pull data from string n/a yes
region The name of the region to deploy within string n/a yes
accept_limited_use_license Acceptance of the SLULA terms (https://docs.snowplow.io/limited-use-license-1.0/) bool false no
app_version App version to use. This variable facilitates dev flow, the modules may not work with anything other than the default value. string "3.9.0" no
assets_update_period Period after which enrich assets should be checked for updates (e.g. MaxMind DB) string "7 days" no
associate_public_ip_address Whether to assign a public ip address to this instance; if false this instance must be behind a Cloud NAT to connect to the internet bool true no
custom_iglu_resolvers The custom Iglu Resolvers that will be used by Enrichment to resolve and validate events 
list(object({
    name            = string
    priority        = number
    uri             = string
    api_key         = string
    vendor_prefixes = list(string)
  }))
 [] no
custom_tcp_egress_port_list For opening up TCP ports to access other destinations not served over HTTP(s) (e.g. for SQL / API enrichments) list(string) [] no
default_iglu_resolvers The default Iglu Resolvers that will be used by Enrichment to resolve and validate events 
list(object({
    name            = string
    priority        = number
    uri             = string
    api_key         = string
    vendor_prefixes = list(string)
  }))
 
[
  {
    "api_key": "",
    "name": "Iglu Central",
    "priority": 10,
    "uri": "http://iglucentral.com",
    "vendor_prefixes": []
  },
  {
    "api_key": "",
    "name": "Iglu Central - Mirror 01",
    "priority": 20,
    "uri": "http://mirror01.iglucentral.com",
    "vendor_prefixes": []
  }
]
 no
enrichment_anon_ip n/a string "" no
enrichment_api_request_enrichment_config n/a string "" no
enrichment_campaign_attribution n/a string "" no
enrichment_cookie_extractor_config n/a string "" no
enrichment_currency_conversion_config n/a string "" no
enrichment_event_fingerprint_config n/a string "" no
enrichment_http_header_extractor_config n/a string "" no
enrichment_iab_spiders_and_bots_enrichment Note: Requires paid database to function string "" no
enrichment_ip_lookups Note: Requires free or paid subscription to database to function string "" no
enrichment_javascript_script_config n/a string "" no
enrichment_pii_enrichment_config n/a string "" no
enrichment_referer_parser n/a string "" no
enrichment_sql_query_enrichment_config n/a string "" no
enrichment_ua_parser_config n/a string "" no
enrichment_weather_enrichment_config n/a string "" no
enrichment_yauaa_enrichment_config n/a string "" no
gcp_logs_enabled Whether application logs should be reported to GCP Logging bool true no
java_opts Custom JAVA Options string "-XX:InitialRAMPercentage=75 -XX:MaxRAMPercentage=75" no
labels The labels to append to this resource map(string) {} no
machine_type The machine type to use string "e2-small" no
network_project_id The project ID of the shared VPC in which the stack is being deployed string "" no
ssh_block_project_keys Whether to block project wide SSH keys bool true no
ssh_ip_allowlist The list of CIDR ranges to allow SSH traffic from list(any) 
[
  "0.0.0.0/0"
]
 no
ssh_key_pairs The list of SSH key-pairs to add to the servers 
list(object({
    user_name  = string
    public_key = string
  }))
 [] no
subnetwork The name of the sub-network to deploy within; if populated will override the 'network' setting string "" no
target_size The number of servers to deploy number 1 no
telemetry_enabled Whether or not to send telemetry information back to Snowplow Analytics Ltd bool true no
ubuntu_20_04_source_image The source image to use which must be based of of Ubuntu 20.04; by default the latest community version is used string "" no
user_provided_id An optional unique identifier to identify the telemetry events emitted by this stack string "" no

Outputs

Name Description
instance_group_url The full URL of the instance group created by the manager
manager_id Identifier for the instance group manager
manager_self_link The URL for the instance group manager

Copyright and license

Copyright 2021-present Snowplow Analytics Ltd.

Licensed under the Snowplow Limited Use License Agreement. (If you are uncertain how it applies to your use case, check our answers to frequently asked questions.)