cloudfreexiao
/
zeus
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

🐳chore(工具):去掉 tidb

Browse Source
develop
xiaojin 5 years ago
parent 62e5c0215d
commit e42df0197b
64 changed files with 1 additions and 102319 deletions
Show Stats Download Patch File Download Diff File

14
docker/tidb/.gitignore vendored
Unescape Escape View File

@ -1,14 +0,0 @@
/values.yaml
/generated-docker-compose.yml
/data
/logs
/pd/bin
/tikv/bin
/tidb/bin
/tidb-vision/tidb-vision
/tmp
/docker/dashboard_installer/dashboard/overview.json
/docker/dashboard_installer/dashboard/pd.json
/docker/dashboard_installer/dashboard/tidb.json
/docker/dashboard_installer/dashboard/tikv.json
/.idea

24
docker/tidb/.travis.yml
Unescape Escape View File

@ -1,24 +0,0 @@
sudo: required
services:
- docker
before_install:
- curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
- sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
- sudo apt-get update
- sudo apt-get -y install docker-ce # update docker version
- sudo curl -L https://github.com/docker/compose/releases/download/1.21.2/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
- docker -v
- docker-compose -v
- docker-compose up -d
- sleep 10 # wait all components get ready
- docker-compose ps
- docker images
- docker network ls
- docker-compose logs
script:
- docker ps -a --format="{{.Names}} {{.Image}} {{.Status}}" | grep -v 'Up' | grep -v 'Exited (0)' | awk '{print} END {if (NR>0) {exit 1;}}'
- docker-compose -f docker-compose-test.yml run --rm tispark-tests bash /opt/tispark-tests/tests/loaddata.sh # add some data for tests
# - docker-compose -f docker-compose-test.yml run --rm tispark-tests /opt/spark/bin/spark-submit /opt/spark/tests/tests.py # run tispark tests

277
docker/tidb/README.md
Unescape Escape View File

@ -1,277 +0,0 @@
# TiDB docker-compose
[![Build Status](https://travis-ci.org/pingcap/tidb-docker-compose.svg?branch=master)](https://travis-ci.org/pingcap/tidb-docker-compose)
**WARNING: This is for testing only, DO NOT USE IN PRODUCTION!**
## Requirements
* Docker >= 17.03
* Docker Compose >= 1.6.0
> **Note:** [Legacy Docker Toolbox](https://docs.docker.com/toolbox/toolbox_install_mac/) users must migrate to [Docker for Mac](https://store.docker.com/editions/community/docker-ce-desktop-mac), since it is tested that tidb-docker-compose cannot be started on Docker Toolbox and Docker Machine.
## Quick start
```bash
$ git clone https://github.com/pingcap/tidb-docker-compose.git
$ cd tidb-docker-compose && docker-compose pull # Get the latest Docker images
$ docker-compose up -d
$ mysql -h 127.0.0.1 -P 4000 -u root
```
* Access monitor at http://localhost:3000 (login with admin/admin if you want to modify grafana)
* Access [tidb-vision](https://github.com/pingcap/tidb-vision) at http://localhost:8010
* Access Spark Web UI at http://localhost:8080
and access [TiSpark](https://github.com/pingcap/tispark) through spark://127.0.0.1:7077
## Docker Swarm
You can also use Docker Swarm to deploy a TiDB Platform cluster, and then you can scale the service using `docker stack` commands.
```bash
$ docker swarm init # if your docker daemon is not already part of a swarm
$ mkdir -p data logs
$ docker stack deploy tidb -c docker-swarm.yml
$ mysql -h 127.0.0.1 -P 4000 -u root
```
After deploying the stack, you can scale the number of TiDB Server instances in the cluster like this:
```bash
$ docker service scale tidb_tidb=2
```
Docker Swarm automatically load-balances across the containers that implement a scaled service, which you can see if you execute `select @@hostname` several times:
```bash
$ mysql -h 127.0.0.1 -P 4000 -u root -te 'select @@hostname'
+--------------+
| @@hostname |
+--------------+
| 340092e0ec9e |
+--------------+
$ mysql -h 127.0.0.1 -P 4000 -u root -te 'select @@hostname'
+--------------+
| @@hostname |
+--------------+
| e6f05ffe6274 |
+--------------+
$ mysql -h 127.0.0.1 -P 4000 -u root -te 'select @@hostname'
+--------------+
| @@hostname |
+--------------+
| 340092e0ec9e |
+--------------+
```
If you want to connect to specific backend instances, for example to test concurrency by ensuring that you are connecting to distinct instances of tidb-server, you can use the `docker service ps` command to assemble a hostname for each container:
```bash
$ docker service ps --no-trunc --format '{{.Name}}.{{.ID}}' tidb_tidb
tidb_tidb.1.x3sc2sd66a88phsj103ohr6qq
tidb_tidb.2.lk53apndq394cega46at853zw
```
To be able to resolve those hostnames, it's easiest to run the MySQL client in a container that has access to the swarm network:
```bash
$ docker run --rm --network=tidb_default arey/mysql-client -h tidb_tidb.1.x3sc2sd66a88phsj103ohr6qq -P 4000 -u root -t -e 'select @@version'
+-----------------------------------------+
| @@version |
+-----------------------------------------+
| 5.7.25-TiDB-v3.0.0-beta.1-40-g873d9514b |
+-----------------------------------------+
```
To loop through all instances of TiDB Server, you can use a bash loop like this:
```bash
for host in $(docker service ps --no-trunc --format '{{.Name}}.{{.ID}}' tidb_tidb)
do docker run --rm --network tidb_default arey/mysql-client \
-h "$host" -P 4000 -u root -te "select @@hostname"
done
```
To stop all services and remove all containers in the TiDB stack, execute `docker stack rm tidb`.
## Customize TiDB Cluster
### Configuration
* config/pd.toml is copied from [PD repo](https://github.com/pingcap/pd/tree/master/conf)
* config/tikv.toml is copied from [TiKV repo](https://github.com/pingcap/tikv/tree/master/etc)
* config/tidb.toml is copied from [TiDB repo](https://github.com/pingcap/tidb/tree/master/config)
* config/pump.toml is copied from [TiDB-Binlog repo](https://github.com/pingcap/tidb-binlog/tree/master/cmd/pump)
* config/drainer.toml is copied from [TiDB-Binlog repo](https://github.com/pingcap/tidb-binlog/tree/master/cmd/drainer)
If you find these configuration files outdated or mismatch with TiDB version, you can copy these files from their upstream repos and change their metrics addr with `pushgateway:9091`. Also `max-open-files` are configured to `1024` in tikv.toml to simplify quick start on Linux, because setting up ulimit on Linux with docker is quite tedious.
And config/\*-dashboard.json are copied from [TiDB-Ansible repo](https://github.com/pingcap/tidb-ansible/tree/master/scripts)
You can customize TiDB cluster configuration by editing docker-compose.yml and the above config files if you know what you're doing.
But edit these files manually is tedious and error-prone, a template engine is strongly recommended. See the following steps
### Install Helm
[Helm](https://helm.sh) is used as a template render engine
```
curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash
```
Or if you use Mac, you can use homebrew to install Helm by `brew install kubernetes-helm`
### Bring up TiDB cluster
```bash
$ git clone https://github.com/pingcap/tidb-docker-compose.git
$ cd tidb-docker-compose
$ vi compose/values.yaml # custom cluster size, docker image, port mapping etc
$ helm template compose > generated-docker-compose.yaml
$ docker-compose -f generated-docker-compose.yaml pull # Get the latest Docker images
$ docker-compose -f generated-docker-compose.yaml up -d
# If you want to Bring up TiDB cluster with Binlog support
$ vi compose/values.yaml # set tidb.enableBinlog to true
$ helm template compose > generated-docker-compose-binlog.yaml
$ docker-compose -f generated-docker-compose-binlog.yaml up -d # or you can use 'docker-compose-binlog.yml' file directly
# Note: If the value of drainer.destDBType is "kafka" and
# you want to consume the kafka messages outside the docker containers,
# please update the kafka.advertisedHostName with your docker host IP in compose/values.yaml and
# regenerate the 'generated-docker-compose-binlog.yaml' file
```
You can build docker image yourself for development test.
* Build from binary
For pd, tikv, tidb, pump and drainer comment their `image` and `buildPath` fields out. And then copy their binary files to pd/bin/pd-server, tikv/bin/tikv-server, tidb/bin/tidb-server, tidb-binlog/bin/pump and tidb-binlog/bin/drainer.
These binary files can be built locally or downloaded from https://download.pingcap.org/tidb-latest-linux-amd64.tar.gz
For tidbVision, comment its `image` and `buildPath` fields out. And then copy tidb-vision repo to tidb-vision/tidb-vision.
* Build from source
Leave pd, tikv, tidb and tidbVision `image` field empty and set their `buildPath` field to their source directory.
For example, if your local tikv source directory is $GOPATH/src/github.com/pingcap/tikv, just set tikv `buildPath` to `$GOPATH/src/github.com/pingcap/tikv`
*Note:* Compiling tikv from source consumes lots of memory, memory of Docker for Mac needs to be adjusted to greater than 6GB
[tidb-vision](https://github.com/pingcap/tidb-vision) is a visiualization page of TiDB Cluster, it's WIP project and can be disabled by commenting `tidbVision` out.
[TiSpark](https://github.com/pingcap/tispark) is a thin layer built for running Apache Spark on top of TiDB/TiKV to answer the complex OLAP queries.
#### Host network mode (Linux)
*Note:* Docker for Mac uses a Linux virtual machine, host network mode will not expose any services to host machine. So it's useless to use this mode.
When using TiKV directly without TiDB, host network mode must be enabled. This way all services use host network without isolation. So you can access all services on the host machine.
You can enable this mode by setting `networkMode: host` in compose/values.yaml and regenerate docker-compose.yml. When in this mode, prometheus address in configuration files should be changed from `prometheus:9090` to `127.0.0.1:9090`, and pushgateway address should be changed from `pushgateway:9091` to `127.0.0.1:9091`.
These modification can be done by:
```bash
# Note: this only needed when networkMode is `host`
sed -i 's/pushgateway:9091/127.0.0.1:9091/g' config/*
sed -i 's/prometheus:9090/127.0.0.1:9090/g' config/*
```
After all the above is done, you can start tidb-cluster as usual by `docker-compose -f generated-docker-compose.yml up -d`
### Debug TiDB/TiKV/PD instances
Prerequisites:
Pprof: This is a tool for visualization and analysis of profiling data. Follow [these instructions](https://github.com/google/pprof#building-pprof) to install pprof.
Graphviz: [http://www.graphviz.org/](http://www.graphviz.org/), used to generate graphic visualizations of profiles.
* debug TiDB or PD instances
```bash
### Use the following command to starts a web server for graphic visualizations of golang program profiles
$ ./tool/container_debug -s pd0 -p /pd-server -w
```
The above command will produce graphic visualizations of profiles of `pd0` that can be accessed through the browser.
* debug TiKV instances
```bash
### step 1: select a tikv instance(here is tikv0) and specify the binary path in container to enter debug container
$ ./tool/container_debug -s tikv0 -p /tikv-server
### after step 1, we can generate flame graph for tikv0 in debug container
$ ./run_flamegraph.sh 1 # 1 is the tikv0's process id
### also can fetch tikv0's stack informations with GDB in debug container
$ gdb /tikv-server 1 -batch -ex "thread apply all bt" -ex "info threads"
```
### Access TiDB cluster
TiDB uses ports: 4000(mysql) and 10080(status) by default
```bash
$ mysql -h 127.0.0.1 -P 4000 -u root --comments
```
And Grafana uses port 3000 by default, so open your browser at http://localhost:3000 to view monitor dashboard
If you enabled tidb-vision, you can view it at http://localhost:8010
### Access Spark shell and load TiSpark
Insert some sample data to the TiDB cluster:
```bash
$ docker-compose exec tispark-master bash
$ cd /opt/spark/data/tispark-sample-data
$ mysql --local-infile=1 -h tidb -P 4000 -u root --comments < dss.ddl
```
After the sample data is loaded into the TiDB cluster, you can access Spark Shell by `docker-compose exec tispark-master /opt/spark/bin/spark-shell`.
```bash
$ docker-compose exec tispark-master /opt/spark/bin/spark-shell
...
Spark context available as 'sc' (master = local[*], app id = local-1527045927617).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.1.1
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_172)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import org.apache.spark.sql.TiContext
...
scala> val ti = new TiContext(spark)
...
scala> ti.tidbMapDatabase("TPCH_001")
...
scala> spark.sql("select count(*) from lineitem").show
+--------+
|count(1)|
+--------+
| 60175|
+--------+
```
You can also access Spark with Python or R using the following commands:
```
docker-compose exec tispark-master /opt/spark/bin/pyspark
docker-compose exec tispark-master /opt/spark/bin/sparkR
```
More documents about TiSpark can be found [here](https://github.com/pingcap/tispark).

13
docker/tidb/compose/Chart.yaml
Unescape Escape View File

@ -1,13 +0,0 @@
apiVersion: v1
description: tidb-docker-compose
name: tidb-docker-compose
version: 0.1.0
home: https://github.com/pingcap/tidb-docker-compose
sources:
- https://github.com/pingcap/tidb-docker-compose
keywords:
- newsql
- htap
- database
- mysql
- raft

62
docker/tidb/compose/templates/_helpers.tpl
Unescape Escape View File

@ -1,62 +0,0 @@
{{- define "initial_cluster" }}
{{- range until (.Values.pd.size | int) }}
{{- if . -}}
,
{{- end -}}
pd{{ . }}=http://
{{- if eq $.Values.networkMode "host" -}}
127.0.0.1:{{add (add ($.Values.pd.port | int) 10000) . }}
{{- else -}}
pd{{ . }}:2380
{{- end -}}
{{- end -}}
{{- end -}}
{{- define "pd_list" }}
{{- range until (.Values.pd.size | int) }}
{{- if . -}}
,
{{- end -}}
{{- if eq $.Values.networkMode "host" -}}
127.0.0.1:{{ add ($.Values.pd.port | int) . }}
{{- else -}}
pd{{ . }}:2379
{{- end -}}
{{- end -}}
{{- end -}}
{{- define "pd_urls" }}
{{- range until (.Values.pd.size | int) }}
{{- if . -}}
,
{{- end -}}
{{- if eq $.Values.networkMode "host" -}}
http://127.0.0.1:{{ add ($.Values.pd.port | int) . }}
{{- else -}}
http://pd{{ . }}:2379
{{- end -}}
{{- end -}}
{{- end -}}
{{- define "zoo_servers" }}
{{- range until (.Values.zookeeper.size | int) }}
{{- if eq $.Values.networkMode "host" -}}
{{- if . }} {{ end }}server.{{ add . 1 }}=127.0.0.1:{{ add . 2888 }}:{{ add . 3888 }}
{{- else -}}
{{- if . }} {{ end }}server.{{ add . 1 }}=zoo{{ . }}:2888:3888
{{- end -}}
{{- end -}}
{{- end -}}
{{- define "zoo_connect" }}
{{- range until (.Values.zookeeper.size | int) }}
{{- if . -}}
,
{{- end -}}
{{- if eq $.Values.networkMode "host" -}}
127.0.0.1:{{ add $.Values.zookeeper.port . }}
{{- else -}}
zoo{{ add . }}:{{ add $.Values.zookeeper.port . }}
{{- end -}}
{{- end -}}
{{- end -}}

431
docker/tidb/compose/templates/docker-compose.yml
Unescape Escape View File

@ -1,431 +0,0 @@
{{- $pdSize := .Values.pd.size | int }}
{{- $tikvSize := .Values.tikv.size | int }}
{{- $pdPort := .Values.pd.port | int }}
{{- $pdPeerPort := add $pdPort 10000 }}
{{- $tikvPort := .Values.tikv.port | int -}}
{{- $pumpSize := .Values.pump.size | int }}
{{- $pumpPort := .Values.pump.port | int }}
{{- $zooSize := .Values.zookeeper.size | int }}
{{- $zooPort := .Values.zookeeper.port | int }}
{{- $kafkaSize := .Values.kafka.size | int }}
{{- $kafkaPort := .Values.kafka.port | int }}
version: '2.1'
services:
{{- range until $pdSize }}
pd{{ . }}:
{{- if $.Values.pd.image }}
image: {{ $.Values.pd.image }}
{{- else }}
image: pd:latest
build:
context: {{ $.Values.pd.buildPath | default "./pd" }}
dockerfile: {{ $.Values.pd.dockerfile | default "Dockerfile" }}
{{- end }}
{{- if eq $.Values.networkMode "host" }}
network_mode: host
{{- else }}
ports:
- "2379"
{{- end }}
volumes:
- ./config/pd.toml:/pd.toml:ro
- {{ $.Values.dataDir }}:/data
- {{ $.Values.logsDir }}:/logs
command:
- --name=pd{{ . }}
{{- if eq $.Values.networkMode "host" }}
- --client-urls=http://0.0.0.0:{{ add $pdPort . }}
- --peer-urls=http://0.0.0.0:{{ add $pdPeerPort . }}
- --advertise-client-urls=http://127.0.0.1:{{ add $pdPort . }}
- --advertise-peer-urls=http://127.0.0.1:{{ add $pdPeerPort . }}
{{- else }}
- --client-urls=http://0.0.0.0:2379
- --peer-urls=http://0.0.0.0:2380
- --advertise-client-urls=http://pd{{ . }}:2379
- --advertise-peer-urls=http://pd{{ . }}:2380
{{- end }}
- --initial-cluster={{- template "initial_cluster" $ }}
- --data-dir=/data/pd{{ . }}
- --config=/pd.toml
- --log-file=/logs/pd{{ . }}.log
# sysctls:
# net.core.somaxconn: 32768
# ulimits:
# nofile:
# soft: 1000000
# hard: 1000000
restart: on-failure
{{ end }}
{{- range until $tikvSize }}
tikv{{ . }}:
{{- if $.Values.tikv.image }}
image: {{ $.Values.tikv.image }}
{{- else }}
image: tikv:latest
build:
context: {{ $.Values.tikv.buildPath | default "./tikv" }}
dockerfile: {{ $.Values.tikv.dockerfile | default "Dockerfile" }}
{{- end }}
{{- if eq $.Values.networkMode "host" }}
network_mode: host
{{- end }}
volumes:
- ./config/tikv.toml:/tikv.toml:ro
- {{ $.Values.dataDir }}:/data
- {{ $.Values.logsDir }}:/logs
command:
{{- if eq $.Values.networkMode "host" }}
- --addr=0.0.0.0:{{ add $tikvPort . }}
- --advertise-addr=127.0.0.1:{{ add $tikvPort . }}
{{- else }}
- --addr=0.0.0.0:20160
- --advertise-addr=tikv{{ . }}:20160
{{- end }}
- --data-dir=/data/tikv{{ . }}
- --pd={{- template "pd_list" $ }}
- --config=/tikv.toml
- --log-file=/logs/tikv{{ . }}.log
depends_on:
{{- range until $pdSize }}
- "pd{{.}}"
{{- end }}
# sysctls:
# net.core.somaxconn: 32768
# ulimits:
# nofile:
# soft: 1000000
# hard: 1000000
restart: on-failure
{{ end }}
{{- if .Values.tidb }}
{{- if .Values.tidb.enableBinlog }}
{{- range until $pumpSize }}
pump{{ . }}:
{{- if $.Values.pump.image }}
image: {{ $.Values.pump.image }}
{{- else }}
image: tidb-binlog:latest
build:
context: {{ $.Values.pump.buildPath | default "./tidb-binlog" }}
dockerfile: {{ $.Values.pump.dockerfile | default "Dockerfile" }}
{{- end }}
{{- if eq $.Values.networkMode "host" }}
network_mode: host
{{- end }}
volumes:
- ./config/pump.toml:/pump.toml:ro
- {{ $.Values.dataDir }}:/data
- {{ $.Values.logsDir }}:/logs
command:
- /pump
{{- if eq $.Values.networkMode "host" }}
- --addr=0.0.0.0:{{ add $pumpPort . }}
- --advertise-addr=127.0.0.1:{{ add $pumpPort . }}
{{- else }}
- --addr=0.0.0.0:8250
- --advertise-addr=pump{{ . }}:8250
{{- end }}
- --data-dir=/data/pump{{ . }}
- --log-file=/logs/pump{{ . }}.log
- --node-id=pump{{ . }}
- --pd-urls={{- template "pd_urls" $ }}
- --config=/pump.toml
depends_on:
{{- range until $pdSize }}
- "pd{{.}}"
{{- end }}
restart: on-failure
{{ end }}
drainer:
{{- if $.Values.drainer.image }}
image: {{ $.Values.drainer.image }}
{{- else }}
image: tidb-binlog:latest
build:
context: {{ $.Values.drainer.buildPath | default "./tidb-binlog" }}
dockerfile: {{ $.Values.drainer.dockerfile | default "Dockerfile" }}
{{- end }}
{{- if eq $.Values.networkMode "host" }}
network_mode: host
{{- end }}
volumes:
- ./config/drainer.toml:/drainer.toml:ro
- {{ $.Values.dataDir }}:/data
- {{ $.Values.logsDir }}:/logs
command:
- /drainer
- --addr=0.0.0.0:8249
- --data-dir=/data/data.drainer
- --log-file=/logs/drainer.log
- --pd-urls={{- template "pd_urls" $ }}
- --config=/drainer.toml
- --initial-commit-ts=0
{{- if eq $.Values.drainer.destDBType "kafka" }}
- --dest-db-type=kafka
{{- end }}
depends_on:
{{- range until $pdSize }}
- "pd{{ . }}"
{{- end }}
{{- if eq $.Values.drainer.destDBType "kafka" }}
{{- range until $kafkaSize }}
- "kafka{{ . }}"
{{- end }}
{{- end }}
restart: on-failure
{{- if eq $.Values.drainer.destDBType "kafka" }}
{{ range until $zooSize }}
zoo{{ . }}:
image: zookeeper:latest
{{- if eq $.Values.networkMode "host" }}
network_mode: host
{{- else }}
ports:
- "{{ add $zooPort . }}:{{ add $zooPort . }}"
{{- end }}
environment:
ZOO_MY_ID: {{ add . 1 }}
ZOO_PORT: {{ add $zooPort . }}
ZOO_SERVERS: {{ template "zoo_servers" $ }}
volumes:
- {{ $.Values.dataDir }}/zoo{{ . }}/data:/data
- {{ $.Values.dataDir }}/zoo{{ . }}/datalog:/datalog
restart: on-failure
{{ end }}
{{- range until $kafkaSize }}
kafka{{ . }}:
image: {{ $.Values.kafka.image }}
{{- if eq $.Values.networkMode "host" }}
network_mode: host
{{- else }}
ports:
- "{{ add . $kafkaPort }}:{{ add . $kafkaPort }}"
{{- end }}
environment:
KAFKA_BROKER_ID: {{ add . 1 }}
KAFKA_LOG_DIRS: /data/kafka-logs
{{- if $.Values.kafka.advertisedHostName }}
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://{{ $.Values.kafka.advertisedHostName }}:{{ add . $kafkaPort }}
{{- else }}
{{- if eq $.Values.networkMode "host" }}
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:{{ add . $kafkaPort }}
{{- else }}
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka{{ . }}:{{ add . $kafkaPort }}
{{- end }}
{{- end }}
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:{{ add . $kafkaPort }}
KAFKA_ZOOKEEPER_CONNECT: {{ template "zoo_connect" $ }}
volumes:
- {{ $.Values.dataDir }}/kafka-logs/kafka{{ . }}:/data/kafka-logs
- {{ $.Values.logsDir }}/kafka{{ . }}:/opt/kafka/logs
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
{{- range until $zooSize }}
- "zoo{{ . }}"
{{- end }}
restart: on-failure
{{- end }}
{{- end }}
{{- end }}
{{ end }}
{{- if .Values.tidb }}
tidb:
{{- if .Values.tidb.image }}
image: {{ .Values.tidb.image }}
{{- else }}
image: tidb:latest
build:
context: {{ .Values.tidb.buildPath | default "./tidb" }}
dockerfile: {{ .Values.tidb.dockerfile | default "Dockerfile" }}
{{- end }}
{{- if eq .Values.networkMode "host" }}
network_mode: host
{{- else }}
ports:
- "{{ .Values.tidb.mysqlPort }}:4000"
- "{{ .Values.tidb.statusPort }}:10080"
{{- end }}
volumes:
- ./config/tidb.toml:/tidb.toml:ro
- {{ .Values.logsDir }}:/logs
command:
- --store=tikv
- --path={{- template "pd_list" $ }}
- --config=/tidb.toml
- --log-file=/logs/tidb.log
- --advertise-address=tidb
{{- if .Values.tidb.enableBinlog }}
- --enable-binlog=true
{{- end }}
depends_on:
{{- range until $tikvSize }}
- "tikv{{.}}"
{{- end }}
{{- if .Values.tidb.enableBinlog }}
{{- range until $pumpSize }}
- "pump{{.}}"
{{- end }}
{{- end }}
# sysctls:
# net.core.somaxconn: 32768
# ulimits:
# nofile:
# soft: 1000000
# hard: 1000000
restart: on-failure
{{ end }}
{{- if .Values.tispark }}
tispark-master:
{{- if .Values.tispark.image }}
image: {{ .Values.tispark.image }}
{{- else }}
image: tispark:latest
build:
context: {{ .Values.tispark.buildPath | default "./tispark" }}
dockerfile: {{ .Values.tispark.dockerfile | default "Dockerfile" }}
{{- end }}
command:
- /opt/spark/sbin/start-master.sh
volumes:
- ./config/spark-defaults.conf:/opt/spark/conf/spark-defaults.conf:ro
environment:
SPARK_MASTER_PORT: {{ .Values.tispark.masterPort }}
SPARK_MASTER_WEBUI_PORT: {{ .Values.tispark.webuiPort }}
ports:
- "{{ .Values.tispark.masterPort }}:7077"
- "{{ .Values.tispark.webuiPort }}:8080"
depends_on:
{{- range until $tikvSize }}
- "tikv{{.}}"
{{- end }}
restart: on-failure
{{- range until ( .Values.tispark.workerCount | int ) }}
tispark-slave{{ . }}:
{{- if $.Values.tispark.image }}
image: {{ $.Values.tispark.image }}
{{- else }}
image: tispark:latest
build:
context: {{ $.Values.tispark.buildPath | default "./tispark" }}
dockerfile: {{ $.Values.tispark.dockerfile | default "Dockerfile" }}
{{- end }}
command:
- /opt/spark/sbin/start-slave.sh
- spark://tispark-master:7077
volumes:
- ./config/spark-defaults.conf:/opt/spark/conf/spark-defaults.conf:ro
environment:
SPARK_WORKER_WEBUI_PORT: {{ add $.Values.tispark.workerWebUIPort . }}
ports:
- "{{ add $.Values.tispark.workerWebUIPort . }}:{{ add $.Values.tispark.workerWebUIPort . }}"
depends_on:
- tispark-master
restart: on-failure
{{- end }}
{{ end }}
{{- if .Values.tidbVision }}
tidb-vision:
{{- if .Values.tidbVision.image }}
image: {{ .Values.tidbVision.image }}
{{- else }}
image: tidb-vision:latest
build:
context: {{ .Values.tidbVision.buildPath | default "./tidb-vision" }}
dockerfile: {{ .Values.tidbVision.dockerfile | default "Dockerfile" }}
{{- end }}
environment:
PD_ENDPOINT: {{if eq .Values.networkMode "host"}}127.0.0.1:{{.Values.pd.port}}{{else}}pd0:2379{{end}}
{{- if eq .Values.networkMode "host" }}
PORT: {{ .Values.tidbVision.port }}
network_mode: host
{{- else }}
ports:
- "{{ .Values.tidbVision.port }}:8010"
{{- end }}
restart: on-failure
{{- end }}
{{- if .Values.prometheus }}
pushgateway:
image: {{ .Values.pushgateway.image }}
{{- if eq .Values.networkMode "host" }}
command:
- --web.listen-address=0.0.0.0:{{.Values.pushgateway.port}}
- --log.level={{ .Values.pushgateway.logLevel }}
network_mode: host
{{- else }}
command:
- --log.level={{ .Values.pushgateway.logLevel }}
{{- end }}
restart: on-failure
prometheus:
user: root
image: {{ .Values.prometheus.image }}
command:
- --log.level={{ .Values.prometheus.logLevel }}
- --storage.tsdb.path=/data/prometheus
- --config.file=/etc/prometheus/prometheus.yml
{{- if eq .Values.networkMode "host" }}
- --web.listen-address=0.0.0.0:{{.Values.prometheus.port}}
network_mode: host
{{- else }}
ports:
- "{{ .Values.prometheus.port }}:9090"
{{- end }}
volumes:
- ./config/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./config/pd.rules.yml:/etc/prometheus/pd.rules.yml:ro
- ./config/tikv.rules.yml:/etc/prometheus/tikv.rules.yml:ro
- ./config/tidb.rules.yml:/etc/prometheus/tidb.rules.yml:ro
- {{ .Values.dataDir }}:/data
restart: on-failure
{{- end }}
{{- if .Values.grafana }}
grafana:
image: {{ .Values.grafana.image }}
user: "0"
{{- if eq .Values.networkMode "host" }}
network_mode: host
environment:
GF_SERVER_HTTP_PORT: {{ .Values.grafana.port }}
GF_LOG_LEVEL: {{ .Values.grafana.logLevel }}
{{- else }}
environment:
GF_LOG_LEVEL: {{ .Values.grafana.logLevel }}
GF_PATHS_PROVISIONING: /etc/grafana/provisioning
GF_PATHS_CONFIG: /etc/grafana/grafana.ini
ports:
- "{{ .Values.grafana.port }}:3000"
{{- end }}
volumes:
- ./config/grafana:/etc/grafana
- ./config/dashboards:/tmp/dashboards
- ./data/grafana:/var/lib/grafana
restart: on-failure
dashboard-installer:
{{- if .Values.dashboardInstaller.image }}
image: {{ .Values.dashboardInstaller.image }}
{{- else }}
image: tidb-dashboard-installer:latest
build:
context: {{ .Values.dashboardInstaller.buildPath | default "./dashboard-installer" }}
dockerfile: {{ .Values.dashboardInstaller.dockerfile | default "Dockerfile" }}
{{- end }}
{{- if eq .Values.networkMode "host" }}
network_mode: host
command: ["127.0.0.1:{{.Values.grafana.port}}"]
{{- else }}
command: ["grafana:3000"]
{{- end }}
restart: on-failure
{{- end -}}

125
docker/tidb/compose/values.yaml
Unescape Escape View File

@ -1,125 +0,0 @@
dataDir: ./data
logsDir: ./logs
# supported networkMode: bridge | host
# host network mode is useless on Mac
networkMode: bridge
pd:
size: 3
image: pingcap/pd:latest
# If you want to build pd image from source, leave image empty and specify pd source directory
# and its dockerfile name
# buildPath: ./pd
# dockerfile: Dockerfile
# when network_mode is host, pd port ranges [port, port+size)
port: 2379
tikv:
size: 3
image: pingcap/tikv:latest
# If you want to build tikv image from source, leave image empty and specify tikv source directory
# and its dockerfile name
# buildPath: ./tikv
# dockerfile: Dockerfile
# when network mode is host, tikv port ranges [port, port+size)
port: 20160
# comment this section out if you don't need SQL layer and want to use TiKV directly
# when using TiKV directly, networkMode must be set to `host`
tidb:
image: pingcap/tidb:latest
# If you want to build tidb image from source, leave image empty and specify tidb source directory
# and its dockerfile name
# buildPath: ./tidb
# dockerfile: Dockerfile
mysqlPort: "4000"
statusPort: "10080"
enableBinlog: false
pump:
size: 3
image: pingcap/tidb-binlog:latest
# If you want to build pump image from source, leave image empty and specify pump source directory
# and its dockerfile name
# buildPath: ./pump
# dockerfile: Dockerfile
# when network_mode is host, pump port ranges [port, port+size)
port: 8250
drainer:
image: pingcap/tidb-binlog:latest
# If you want to build drainer image from source, leave image empty and specify drainer source directory
# and its dockerfile name
# buildPath: ./drainer
# dockerfile: Dockerfile
destDBType: "kafka"
zookeeper:
size: 3
image: zookeeper:latest
port: 2181
kafka:
size: 3
image: wurstmeister/kafka:2.12-2.1.1
# If you want to consume the kafka messages outside the docker containers,
# Please update the advertisedHostName with your docker host IP
advertisedHostName:
port: 9092
tispark:
image: pingcap/tispark:latest
# If you want to build tidb image from source, leave image empty and specify tidb source directory
# and its dockerfile name
# buildPath: ./tidb
# dockerfile: Dockerfile
buildPath: ./tispark
dockerfile: Dockerfile
masterPort: 7077
webuiPort: 8080
workerCount: 1
# slave web ui port will be workerWebUIPort ~ workerWebUIPort+workerCount-1
workerWebUIPort: 38081
# comment this out to disable tidb-vision
tidbVision:
image: pingcap/tidb-vision:latest
# If you want to build tidb-vision image from source, leave image empty and specify tidb-vision source directory
# and its dockerfile name
# buildPath: ./tidb-vision
# dockerfile: Dockerfile
port: "8010"
# comment following monitor components sections out to disable monitor
grafana:
image: grafana/grafana:5.3.0
port: "3000"
logLevel: error
pushgateway:
image: prom/pushgateway:v0.3.1
port: "9091"
logLevel: error
prometheus:
image: prom/prometheus:v2.2.1
port: "9090"
logLevel: error
# This is used to import tidb monitor dashboard templates to grafana
# this container runs only once and keep running until templates imported successfully
dashboardInstaller:
image: pingcap/tidb-dashboard-installer:v2.0.0
# If you want to build tidb-dashboard-installer image from source, leave image empty and specify tidb-dashboard-installer source directory
# and its dockerfile name
# buildPath: ./dashboard-installer
# dockerfile: Dockerfile

9
docker/tidb/config/dashboards/README.md
Unescape Escape View File

@ -1,9 +0,0 @@
# TiDB dashboard
With Grafana v5.x or later, we can use provisioning feature to statically provision datasources and dashboards. No need to use scripts to configure Grafana.
The JSON files in dashboards are copied from [tidb-ansible](https://github.com/pingcap/tidb-ansible/tree/master/scripts), and need to replace variables in the json file(It was did by python file before).
It is used in [tidb-docker-compose](https://github.com/pingcap/tidb-docker-compose) and [tidb-operator](https://github.com/pingcap/tidb-operator).

3867
docker/tidb/config/dashboards/overview.json
View File

File diff suppressed because it is too large Load Diff

5999
docker/tidb/config/dashboards/pd.json
View File

File diff suppressed because it is too large Load Diff

6996
docker/tidb/config/dashboards/tidb.json
View File

File diff suppressed because it is too large Load Diff

15500
docker/tidb/config/dashboards/tikv_pull.json
View File

File diff suppressed because it is too large Load Diff

101
docker/tidb/config/drainer.toml
Unescape Escape View File

@ -1,101 +0,0 @@
# drainer Configuration.
# addr (i.e. 'host:port') to listen on for drainer connections
# will register this addr into etcd
# addr = "127.0.0.1:8249"
# the interval time (in seconds) of detect pumps' status
detect-interval = 10
# drainer meta data directory path
data-dir = "data.drainer"
# a comma separated list of PD endpoints
pd-urls = "http://127.0.0.1:2379"
# Use the specified compressor to compress payload between pump and drainer
compressor = ""
#[security]
# Path of file that contains list of trusted SSL CAs for connection with cluster components.
# ssl-ca = "/path/to/ca.pem"
# Path of file that contains X509 certificate in PEM format for connection with cluster components.
# ssl-cert = "/path/to/pump.pem"
# Path of file that contains X509 key in PEM format for connection with cluster components.
# ssl-key = "/path/to/pump-key.pem"
# syncer Configuration.
[syncer]
# Assume the upstream sql-mode.
# If this is setted , will use the same sql-mode to parse DDL statment, and set the same sql-mode at downstream when db-type is mysql.
# If this is not setted, it will not set any sql-mode.
# sql-mode = "STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION"
# number of binlog events in a transaction batch
txn-batch = 20
# work count to execute binlogs
# if the latency between drainer and downstream(mysql or tidb) are too high, you might want to increase this
# to get higher throughput by higher concurrent write to the downstream
worker-count = 16
disable-dispatch = false
# safe mode will split update to delete and insert
safe-mode = false
# downstream storage, equal to --dest-db-type
# valid values are "mysql", "file", "tidb", "flash", "kafka"
db-type = "kafka"
# disable sync these schema
ignore-schemas = "INFORMATION_SCHEMA,PERFORMANCE_SCHEMA,mysql"
##replicate-do-db priority over replicate-do-table if have same db name
##and we support regex expression , start with '~' declare use regex expression.
#
#replicate-do-db = ["~^b.*","s1"]
#[[syncer.replicate-do-table]]
#db-name ="test"
#tbl-name = "log"
#[[syncer.replicate-do-table]]
#db-name ="test"
#tbl-name = "~^a.*"
# disable sync these table
#[[syncer.ignore-table]]
#db-name = "test"
#tbl-name = "log"
# the downstream mysql protocol database
#[syncer.to]
#host = "127.0.0.1"
#user = "root"
#password = ""
#port = 3306
[syncer.to.checkpoint]
# you can uncomment this to change the database to save checkpoint when the downstream is mysql or tidb
#schema = "tidb_binlog"
# Uncomment this if you want to use file as db-type.
#[syncer.to]
# directory to save binlog file, default same as data-dir(save checkpoint file) if this is not configured.
# dir = "data.drainer"
# when db-type is kafka, you can uncomment this to config the down stream kafka, it will be the globle config kafka default
[syncer.to]
# only need config one of zookeeper-addrs and kafka-addrs, will get kafka address if zookeeper-addrs is configed.
# zookeeper-addrs = "127.0.0.1:2181"
kafka-addrs = "kafka0:9092,kafka1:9093,kafka2:9094"
kafka-version = "2.1.1"
kafka-max-messages = 1024
#
#
# the topic name drainer will push msg, the default name is <cluster-id>_obinlog
# be careful don't use the same name if run multi drainer instances
# topic-name = ""

7
docker/tidb/config/grafana-datasource.json
Unescape Escape View File

@ -1,7 +0,0 @@
{
"name": "tidb-cluster",
"type": "prometheus",
"url": "http://prometheus:9090",
"access": "proxy",
"basicAuth": false
}

556
docker/tidb/config/grafana/grafana.ini
Unescape Escape View File

@ -1,556 +0,0 @@
##################### Grafana Configuration Defaults #####################
#
# Do not modify this file in grafana installs
#
# possible values : production, development
app_mode = production
# instance name, defaults to HOSTNAME environment variable value or hostname if HOSTNAME var is empty
instance_name = ${HOSTNAME}
#################################### Paths ###############################
[paths]
# Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used)
data = data
# Temporary files in `data` directory older than given duration will be removed
temp_data_lifetime = 24h
# Directory where grafana can store logs
logs = data/log
# Directory where grafana will automatically scan and look for plugins
plugins = data/plugins
# folder that contains provisioning config files that grafana will apply on startup and while running.
provisioning = conf/provisioning
#################################### Server ##############################
[server]
# Protocol (http, https, socket)
protocol = http
# The ip address to bind to, empty will bind to all interfaces
http_addr =
# The http port to use
http_port = 3000
# The public facing domain name used to access grafana from a browser
domain = localhost
# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
enforce_domain = false
# The full public facing url
root_url = %(protocol)s://%(domain)s:%(http_port)s/
# Log web requests
router_logging = false
# the path relative working path
static_root_path = public
# enable gzip
enable_gzip = false
# https certs & key file
cert_file =
cert_key =
# Unix socket path
socket = /tmp/grafana.sock
#################################### Database ############################
[database]
# You can configure the database connection by specifying type, host, name, user and password
# as separate properties or as on string using the url property.
# Either "mysql", "postgres" or "sqlite3", it's your choice
type = sqlite3
host = 127.0.0.1:3306
name = grafana
user = root
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
password =
# Use either URL or the previous fields to configure the database
# Example: mysql://user:secret@host:port/database
url =
# Max idle conn setting default is 2
max_idle_conn = 2
# Max conn setting default is 0 (mean not set)
max_open_conn =
# Connection Max Lifetime default is 14400 (means 14400 seconds or 4 hours)
conn_max_lifetime = 14400
# Set to true to log the sql calls and execution times.
log_queries =
# For "postgres", use either "disable", "require" or "verify-full"
# For "mysql", use either "true", "false", or "skip-verify".
ssl_mode = disable
ca_cert_path =
client_key_path =
client_cert_path =
server_cert_name =
# For "sqlite3" only, path relative to data_path setting
path = grafana.db
#################################### Session #############################
[session]
# Either "memory", "file", "redis", "mysql", "postgres", "memcache", default is "file"
provider = file
# Provider config options
# memory: not have any config yet
# file: session dir path, is relative to grafana data_path
# redis: config like redis server e.g. `addr=127.0.0.1:6379,pool_size=100,db=grafana`
# postgres: user=a password=b host=localhost port=5432 dbname=c sslmode=disable
# mysql: go-sql-driver/mysql dsn config string, examples:
# `user:password@tcp(127.0.0.1:3306)/database_name`
# `user:password@unix(/var/run/mysqld/mysqld.sock)/database_name`
# memcache: 127.0.0.1:11211
provider_config = sessions
# Session cookie name
cookie_name = grafana_sess
# If you use session in https only, default is false
cookie_secure = false
# Session life time, default is 86400
session_life_time = 86400
gc_interval_time = 86400
# Connection Max Lifetime default is 14400 (means 14400 seconds or 4 hours)
conn_max_lifetime = 14400
#################################### Data proxy ###########################
[dataproxy]
# This enables data proxy logging, default is false
logging = false
#################################### Analytics ###########################
[analytics]
# Server reporting, sends usage counters to stats.grafana.org every 24 hours.
# No ip addresses are being tracked, only simple counters to track
# running instances, dashboard and error counts. It is very helpful to us.
# Change this option to false to disable reporting.
reporting_enabled = true
# Set to false to disable all checks to https://grafana.com
# for new versions (grafana itself and plugins), check is used
# in some UI views to notify that grafana or plugin update exists
# This option does not cause any auto updates, nor send any information
# only a GET request to https://grafana.com to get latest versions
check_for_updates = true
# Google Analytics universal tracking code, only enabled if you specify an id here
google_analytics_ua_id =
# Google Tag Manager ID, only enabled if you specify an id here
google_tag_manager_id =
#################################### Security ############################
[security]
# default admin user, created on startup
admin_user = admin
# default admin password, can be changed before first start of grafana, or in profile settings
admin_password = admin
# used for signing
secret_key = SW2YcwTIb9zpOOhoPsMm
# Auto-login remember days
login_remember_days = 7
cookie_username = grafana_user
cookie_remember_name = grafana_remember
# disable gravatar profile images
disable_gravatar = false
# data source proxy whitelist (ip_or_domain:port separated by spaces)
data_source_proxy_whitelist =
# disable protection against brute force login attempts
disable_brute_force_login_protection = false
#################################### Snapshots ###########################
[snapshots]
# snapshot sharing options
external_enabled = true
external_snapshot_url = https://snapshots-origin.raintank.io
external_snapshot_name = Publish to snapshot.raintank.io
# remove expired snapshot
snapshot_remove_expired = true
#################################### Dashboards ##################
[dashboards]
# Number dashboard versions to keep (per dashboard). Default: 20, Minimum: 1
versions_to_keep = 20
#################################### Users ###############################
[users]
# disable user signup / registration
allow_sign_up = false
# Allow non admin users to create organizations
allow_org_create = false
# Set to true to automatically assign new users to the default organization (id 1)
auto_assign_org = true
# Set this value to automatically add new users to the provided organization (if auto_assign_org above is set to true)
auto_assign_org_id = 1
# Default role new users will be automatically assigned (if auto_assign_org above is set to true)
auto_assign_org_role = Viewer
# Require email validation before sign up completes
verify_email_enabled = false
# Background text for the user field on the login page
login_hint = email or username
# Default UI theme ("dark" or "light")
default_theme = dark
# External user management
external_manage_link_url =
external_manage_link_name =
external_manage_info =
# Viewers can edit/inspect dashboard settings in the browser. But not save the dashboard.
viewers_can_edit = false
[auth]
# Set to true to disable (hide) the login form, useful if you use OAuth
disable_login_form = false
# Set to true to disable the signout link in the side menu. useful if you use auth.proxy
disable_signout_menu = false
# URL to redirect the user to after sign out
signout_redirect_url =
#################################### Anonymous Auth ######################
[auth.anonymous]
# enable anonymous access
enabled = true
# specify organization name that should be used for unauthenticated users
org_name = Main Org.
# specify role for unauthenticated users
org_role = Viewer
#################################### Github Auth #########################
[auth.github]
enabled = false
allow_sign_up = true
client_id = some_id
client_secret = some_secret
scopes = user:email,read:org
auth_url = https://github.com/login/oauth/authorize
token_url = https://github.com/login/oauth/access_token
api_url = https://api.github.com/user
team_ids =
allowed_organizations =
#################################### GitLab Auth #########################
[auth.gitlab]
enabled = false
allow_sign_up = true
client_id = some_id
client_secret = some_secret
scopes = api
auth_url = https://gitlab.com/oauth/authorize
token_url = https://gitlab.com/oauth/token
api_url = https://gitlab.com/api/v4
allowed_groups =
#################################### Google Auth #########################
[auth.google]
enabled = false
allow_sign_up = true
client_id = some_client_id
client_secret = some_client_secret
scopes = https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email
auth_url = https://accounts.google.com/o/oauth2/auth
token_url = https://accounts.google.com/o/oauth2/token
api_url = https://www.googleapis.com/oauth2/v1/userinfo
allowed_domains =
hosted_domain =
#################################### Grafana.com Auth ####################
# legacy key names (so they work in env variables)
[auth.grafananet]
enabled = false
allow_sign_up = true
client_id = some_id
client_secret = some_secret
scopes = user:email
allowed_organizations =
[auth.grafana_com]
enabled = false
allow_sign_up = true
client_id = some_id
client_secret = some_secret
scopes = user:email
allowed_organizations =
#################################### Generic OAuth #######################
[auth.generic_oauth]
name = OAuth
enabled = false
allow_sign_up = true
client_id = some_id
client_secret = some_secret
scopes = user:email
email_attribute_name = email:primary
auth_url =
token_url =
api_url =
team_ids =
allowed_organizations =
tls_skip_verify_insecure = false
tls_client_cert =
tls_client_key =
tls_client_ca =
#################################### Basic Auth ##########################
[auth.basic]
enabled = true
#################################### Auth Proxy ##########################
[auth.proxy]
enabled = false
header_name = X-WEBAUTH-USER
header_property = username
auto_sign_up = true
ldap_sync_ttl = 60
whitelist =
#################################### Auth LDAP ###########################
[auth.ldap]
enabled = false
config_file = /etc/grafana/ldap.toml
allow_sign_up = true
#################################### SMTP / Emailing #####################
[smtp]
enabled = false
host = localhost:25
user =
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
password =
cert_file =
key_file =
skip_verify = false
from_address = admin@grafana.localhost
from_name = Grafana
ehlo_identity =
[emails]
welcome_email_on_sign_up = false
templates_pattern = emails/*.html
#################################### Logging ##########################
[log]
# Either "console", "file", "syslog". Default is console and file
# Use space to separate multiple modes, e.g. "console file"
mode = console file
# Either "debug", "info", "warn", "error", "critical", default is "info"
level = info
# optional settings to set different levels for specific loggers. Ex filters = sqlstore:debug
filters =
# For "console" mode only
[log.console]
level =
# log line format, valid options are text, console and json
format = console
# For "file" mode only
[log.file]
level =
# log line format, valid options are text, console and json
format = text
# This enables automated log rotate(switch of following options), default is true
log_rotate = true
# Max line number of single file, default is 1000000
max_lines = 1000000
# Max size shift of single file, default is 28 means 1 << 28, 256MB
max_size_shift = 28
# Segment log daily, default is true
daily_rotate = true
# Expired days of log file(delete after max days), default is 7
max_days = 7
[log.syslog]
level =
# log line format, valid options are text, console and json
format = text
# Syslog network type and address. This can be udp, tcp, or unix. If left blank, the default unix endpoints will be used.
network =
address =
# Syslog facility. user, daemon and local0 through local7 are valid.
facility =
# Syslog tag. By default, the process' argv[0] is used.
tag =
#################################### Usage Quotas ########################
[quota]
enabled = false
#### set quotas to -1 to make unlimited. ####
# limit number of users per Org.
org_user = 10
# limit number of dashboards per Org.
org_dashboard = 100
# limit number of data_sources per Org.
org_data_source = 10
# limit number of api_keys per Org.
org_api_key = 10
# limit number of orgs a user can create.
user_org = 10
# Global limit of users.
global_user = -1
# global limit of orgs.
global_org = -1
# global limit of dashboards
global_dashboard = -1
# global limit of api_keys
global_api_key = -1
# global limit on number of logged in users.
global_session = -1
#################################### Alerting ############################
[alerting]
# Disable alerting engine & UI features
enabled = true
# Makes it possible to turn off alert rule execution but alerting UI is visible
execute_alerts = true
# Default setting for new alert rules. Defaults to categorize error and timeouts as alerting. (alerting, keep_state)
error_or_timeout = alerting
# Default setting for how Grafana handles nodata or null values in alerting. (alerting, no_data, keep_state, ok)
nodata_or_nullvalues = no_data
# Alert notifications can include images, but rendering many images at the same time can overload the server
# This limit will protect the server from render overloading and make sure notifications are sent out quickly
concurrent_render_limit = 5
#################################### Explore #############################
[explore]
# Enable the Explore section
enabled = false
#################################### Internal Grafana Metrics ############
# Metrics available at HTTP API Url /metrics
[metrics]
enabled = true
interval_seconds = 10
# Send internal Grafana metrics to graphite
[metrics.graphite]
# Enable by setting the address setting (ex localhost:2003)
address =
prefix = prod.grafana.%(instance_name)s.
[grafana_net]
url = https://grafana.com
[grafana_com]
url = https://grafana.com
#################################### Distributed tracing ############
[tracing.jaeger]
# jaeger destination (ex localhost:6831)
address =
# tag that will always be included in when creating new spans. ex (tag1:value1,tag2:value2)
always_included_tag =
# Type specifies the type of the sampler: const, probabilistic, rateLimiting, or remote
sampler_type = const
# jaeger samplerconfig param
# for "const" sampler, 0 or 1 for always false/true respectively
# for "probabilistic" sampler, a probability between 0 and 1
# for "rateLimiting" sampler, the number of spans per second
# for "remote" sampler, param is the same as for "probabilistic"
# and indicates the initial sampling rate before the actual one
# is received from the mothership
sampler_param = 1
#################################### External Image Storage ##############
[external_image_storage]
# You can choose between (s3, webdav, gcs, azure_blob, local)
provider =
[external_image_storage.s3]
bucket_url =
bucket =
region =
path =
access_key =
secret_key =
[external_image_storage.webdav]
url =
username =
password =
public_url =
[external_image_storage.gcs]
key_file =
bucket =
path =
[external_image_storage.azure_blob]
account_name =
account_key =
container_name =
[external_image_storage.local]
# does not require any configuration
[rendering]
# Options to configure external image rendering server like https://github.com/grafana/grafana-image-renderer
server_url =
callback_url =

10
docker/tidb/config/grafana/provisioning/dashboards/dashboards.yaml
Unescape Escape View File

@ -1,10 +0,0 @@
# # config file version
apiVersion: 1
providers:
- name: 'default'
orgId: 1
folder: ''
type: file
options:
path: /tmp/dashboards

53
docker/tidb/config/grafana/provisioning/datasources/datasources.yaml
Unescape Escape View File

@ -1,53 +0,0 @@
# # config file version
apiVersion: 1
# # list of datasources that should be deleted from the database
#deleteDatasources:
# - name: Graphite
# orgId: 1
# # list of datasources to insert/update depending
# # on what's available in the datbase
datasources:
# <string, required> name of the datasource. Required
- name: tidb-cluster
# <string, required> datasource type. Required
type: prometheus
# <string, required> access mode. direct or proxy. Required
access: proxy
# # <int> org id. will default to orgId 1 if not specified
# orgId: 1
# <string> url
url: http://prometheus:9090
# # <string> database password, if used
# password:
# # <string> database user, if used
# user:
# # <string> database name, if used
# database:
# <bool> enable/disable basic auth
basicAuth: false
# # <string> basic auth username
# basicAuthUser:
# # <string> basic auth password
# basicAuthPassword:
# # <bool> enable/disable with credentials headers
# withCredentials:
# # <bool> mark as default datasource. Max one per org
# isDefault:
# # <map> fields that will be converted to json and stored in json_data
# jsonData:
# graphiteVersion: "1.1"
# tlsAuth: true
# tlsAuthWithCACert: true
# httpHeaderName1: "Authorization"
# # <string> json object of data that will be encrypted.
# secureJsonData:
# tlsCACert: "..."
# tlsClientCert: "..."
# tlsClientKey: "..."
# # <openshift\kubernetes token example>
# httpHeaderValue1: "Bearer xf5yhfkpsnmgo"
# version: 1
# # <bool> allow users to edit datasources from the UI.
# editable: false

4892
docker/tidb/config/overview-dashboard.json
View File

File diff suppressed because it is too large Load Diff

5776
docker/tidb/config/pd-dashboard.json
View File

File diff suppressed because it is too large Load Diff

3
docker/tidb/config/pd-nightly-tiflash.toml
Unescape Escape View File

@ -1,3 +0,0 @@
[replication]
enable-placement-rules = true
max-replicas = 1

146
docker/tidb/config/pd.rules.yml
Unescape Escape View File

@ -1,146 +0,0 @@
groups:
- name: alert.rules
rules:
- alert: PD_cluster_offline_tikv_nums
expr: sum ( pd_cluster_status{type="store_down_count"} ) > 0
for: 1m
labels:
env: test-cluster
level: emergency
expr: sum ( pd_cluster_status{type="store_down_count"} ) > 0
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: PD_cluster_offline_tikv_nums
- alert: PD_etcd_write_disk_latency
expr: histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[1m])) by (instance,job,le) ) > 1
for: 1m
labels:
env: test-cluster
level: critical
expr: histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[1m])) by (instance,job,le) ) > 1
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: PD_etcd_write_disk_latency
- alert: PD_miss_peer_region_count
expr: sum( pd_regions_status{type="miss_peer_region_count"} ) > 100
for: 1m
labels:
env: test-cluster
level: critical
expr: sum( pd_regions_status{type="miss_peer_region_count"} ) > 100
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: PD_miss_peer_region_count
- alert: PD_cluster_lost_connect_tikv_nums
expr: sum ( pd_cluster_status{type="store_disconnected_count"} ) > 0
for: 1m
labels:
env: test-cluster
level: warning
expr: sum ( pd_cluster_status{type="store_disconnected_count"} ) > 0
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: PD_cluster_lost_connect_tikv_nums
- alert: PD_cluster_low_space
expr: sum ( pd_cluster_status{type="store_low_space_count"} ) > 0
for: 1m
labels:
env: test-cluster
level: warning
expr: sum ( pd_cluster_status{type="store_low_space_count"} ) > 0
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: PD_cluster_low_space
- alert: PD_etcd_network_peer_latency
expr: histogram_quantile(0.99, sum(rate(etcd_network_peer_round_trip_time_seconds_bucket[1m])) by (To,instance,job,le) ) > 1
for: 1m
labels:
env: test-cluster
level: warning
expr: histogram_quantile(0.99, sum(rate(etcd_network_peer_round_trip_time_seconds_bucket[1m])) by (To,instance,job,le) ) > 1
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: PD_etcd_network_peer_latency
- alert: PD_tidb_handle_requests_duration
expr: histogram_quantile(0.99, sum(rate(pd_client_request_handle_requests_duration_seconds_bucket{type="tso"}[1m])) by (instance,job,le) ) > 0.1
for: 1m
labels:
env: test-cluster
level: warning
expr: histogram_quantile(0.99, sum(rate(pd_client_request_handle_requests_duration_seconds_bucket{type="tso"}[1m])) by (instance,job,le) ) > 0.1
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: PD_tidb_handle_requests_duration
- alert: PD_down_peer_region_nums
expr: sum ( pd_regions_status{type="down_peer_region_count"} ) > 0
for: 1m
labels:
env: test-cluster
level: warning
expr: sum ( pd_regions_status{type="down_peer_region_count"} ) > 0
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: PD_down_peer_region_nums
- alert: PD_incorrect_namespace_region_count
expr: sum ( pd_regions_status{type="incorrect_namespace_region_count"} ) > 100
for: 1m
labels:
env: test-cluster
level: warning
expr: sum ( pd_regions_status{type="incorrect_namespace_region_count"} ) > 0
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: PD_incorrect_namespace_region_count
- alert: PD_pending_peer_region_count
expr: sum( pd_regions_status{type="pending_peer_region_count"} ) > 100
for: 1m
labels:
env: test-cluster
level: warning
expr: sum( pd_regions_status{type="pending_peer_region_count"} ) > 100
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: PD_pending_peer_region_count
- alert: PD_leader_change
expr: count( changes(pd_server_tso{type="save"}[10m]) > 0 ) >= 2
for: 1m
labels:
env: test-cluster
level: warning
expr: count( changes(pd_server_tso{type="save"}[10m]) > 0 ) >= 2
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: PD_leader_change
- alert: TiKV_space_used_more_than_80%
expr: sum(pd_cluster_status{type="storage_size"}) / sum(pd_cluster_status{type="storage_capacity"}) * 100 > 80
for: 1m
labels:
env: test-cluster
level: warning
expr: sum(pd_cluster_status{type="storage_size"}) / sum(pd_cluster_status{type="storage_capacity"}) * 100 > 80
annotations:
description: 'cluster: test-cluster, type: {{ $labels.type }}, instance: {{ $labels.instance }}, values: {{ $value }}'
value: '{{ $value }}'
summary: TiKV_space_used_more_than_80%

86
docker/tidb/config/pd.toml
Unescape Escape View File

@ -1,86 +0,0 @@
# PD Configuration.
name = "pd"
data-dir = "default.pd"
client-urls = "http://127.0.0.1:2379"
# if not set, use ${client-urls}
advertise-client-urls = ""
peer-urls = "http://127.0.0.1:2380"
# if not set, use ${peer-urls}
advertise-peer-urls = ""
initial-cluster = "pd=http://127.0.0.1:2380"
initial-cluster-state = "new"
lease = 3
tso-save-interval = "3s"
[security]
# Path of file that contains list of trusted SSL CAs. if set, following four settings shouldn't be empty
cacert-path = ""
# Path of file that contains X509 certificate in PEM format.
cert-path = ""
# Path of file that contains X509 key in PEM format.
key-path = ""
[log]
level = "error"
# log format, one of json, text, console
#format = "text"
# disable automatic timestamps in output
#disable-timestamp = false
# file logging
[log.file]
#filename = ""
# max log file size in MB
#max-size = 300
# max log file keep days
#max-days = 28
# maximum number of old log files to retain
#max-backups = 7
# rotate log by day
#log-rotate = true
[metric]
# prometheus client push interval, set "0s" to disable prometheus.
interval = "15s"
# prometheus pushgateway address, leaves it empty will disable prometheus.
address = "pushgateway:9091"
[schedule]
max-merge-region-size = 0
split-merge-interval = "1h"
max-snapshot-count = 3
max-pending-peer-count = 16
max-store-down-time = "30m"
leader-schedule-limit = 4
region-schedule-limit = 4
replica-schedule-limit = 8
merge-schedule-limit = 8
tolerant-size-ratio = 5.0
# customized schedulers, the format is as below
# if empty, it will use balance-leader, balance-region, hot-region as default
# [[schedule.schedulers]]
# type = "evict-leader"
# args = ["1"]
[replication]
# The number of replicas for each region.
max-replicas = 3
# The label keys specified the location of a store.
# The placement priorities is implied by the order of label keys.
# For example, ["zone", "rack"] means that we should place replicas to
# different zones first, then to different racks if we don't have enough zones.
location-labels = []
[label-property]
# Do not assign region leaders to stores that have these tags.
# [[label-property.reject-leader]]
# key = "zone"
# value = "cn1

15
docker/tidb/config/prometheus.yml
Unescape Escape View File

@ -1,15 +0,0 @@
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'tidb-cluster'
scrape_interval: 5s
honor_labels: true
static_configs:
- targets: ['pushgateway:9091']
labels:
cluster: 'tidb-cluster'
rule_files:
- 'pd.rules.yml'
- 'tikv.rules.yml'
- 'tidb.rules.yml'

45
docker/tidb/config/pump.toml
Unescape Escape View File

@ -1,45 +0,0 @@
# pump Configuration.
# addr(i.e. 'host:port') to listen on for client traffic
addr = "127.0.0.1:8250"
# addr(i.e. 'host:port') to advertise to the public
advertise-addr = ""
# a integer value to control expiry date of the binlog data, indicates for how long (in days) the binlog data would be stored.
# must bigger than 0
gc = 7
# path to the data directory of pump's data
data-dir = "data.pump"
# number of seconds between heartbeat ticks (in 2 seconds)
heartbeat-interval = 2
# a comma separated list of PD endpoints
pd-urls = "http://127.0.0.1:2379"
#[security]
# Path of file that contains list of trusted SSL CAs for connection with cluster components.
# ssl-ca = "/path/to/ca.pem"
# Path of file that contains X509 certificate in PEM format for connection with cluster components.
# ssl-cert = "/path/to/drainer.pem"
# Path of file that contains X509 key in PEM format for connection with cluster components.
# ssl-key = "/path/to/drainer-key.pem"
#
# [storage]
# Set to `true` (default) for best reliability, which prevents data loss when there is a power failure.
# sync-log = true
#
# we suggest using the default config of the embedded LSM DB now, do not change it useless you know what you are doing
# [storage.kv]
# block-cache-capacity = 8388608
# block-restart-interval = 16
# block-size = 4096
# compaction-L0-trigger = 8
# compaction-table-size = 67108864
# compaction-total-size = 536870912
# compaction-total-size-multiplier = 8.0
# write-buffer = 67108864
# write-L0-pause-trigger = 24
# write-L0-slowdown-trigger = 17

2
docker/tidb/config/spark-defaults.conf
Unescape Escape View File

@ -1,2 +0,0 @@
spark.tispark.pd.addresses pd0:2379
spark.sql.extensions org.apache.spark.sql.TiExtensions

6928
docker/tidb/config/tidb-dashboard.json
View File

File diff suppressed because it is too large Load Diff

134
docker/tidb/config/tidb.rules.yml
Unescape Escape View File

@ -1,134 +0,0 @@
groups:
- name: alert.rules
rules:
- alert: TiDB_schema_error
expr: increase(tidb_session_schema_lease_error_total{type="outdated"}[15m]) > 0
for: 1m
labels:
env: test-cluster
level: emergency
expr: increase(tidb_session_schema_lease_error_total{type="outdated"}[15m]) > 0
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiDB schema error
- alert: TiDB_tikvclient_region_err_total
expr: increase( tidb_tikvclient_region_err_total[10m] ) > 6000
for: 1m
labels:
env: test-cluster
level: emergency
expr: increase( tidb_tikvclient_region_err_total[10m] ) > 6000
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiDB tikvclient_backoff_count error
- alert: TiDB_domain_load_schema_total
expr: increase( tidb_domain_load_schema_total{type="failed"}[10m] ) > 10
for: 1m
labels:
env: test-cluster
level: emergency
expr: increase( tidb_domain_load_schema_total{type="failed"}[10m] ) > 10
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiDB domain_load_schema_total error
- alert: TiDB_monitor_keep_alive
expr: increase(tidb_monitor_keep_alive_total{job="tidb"}[10m]) < 100
for: 1m
labels:
env: test-cluster
level: emergency
expr: increase(tidb_monitor_keep_alive_total{job="tidb"}[10m]) < 100
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiDB monitor_keep_alive error
- alert: TiDB_server_panic_total
expr: increase(tidb_server_panic_total[10m]) > 0
for: 1m
labels:
env: test-cluster
level: critical
expr: increase(tidb_server_panic_total[10m]) > 0
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiDB server panic total
- alert: TiDB_memory_abnormal
expr: go_memstats_heap_inuse_bytes{job="tidb"} > 1e+10
for: 1m
labels:
env: test-cluster
level: warning
expr: go_memstats_heap_inuse_bytes{job="tidb"} > 1e+10
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiDB heap memory usage is over 10 GB
- alert: TiDB_query_duration
expr: histogram_quantile(0.99, sum(rate(tidb_server_handle_query_duration_seconds_bucket[1m])) BY (le, instance)) > 1
for: 1m
labels:
env: test-cluster
level: warning
expr: histogram_quantile(0.99, sum(rate(tidb_server_handle_query_duration_seconds_bucket[1m])) BY (le, instance)) > 1
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiDB query duration 99th percentile is above 1s
- alert: TiDB_server_event_error
expr: increase(tidb_server_server_event{type=~"server_start|server_hang"}[15m]) > 0
for: 1m
labels:
env: test-cluster
level: warning
expr: increase(tidb_server_server_event{type=~"server_start|server_hang"}[15m]) > 0
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiDB server event error
- alert: TiDB_tikvclient_backoff_count
expr: increase( tidb_tikvclient_backoff_count[10m] ) > 10
for: 1m
labels:
env: test-cluster
level: warning
expr: increase( tidb_tikvclient_backoff_count[10m] ) > 10
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiDB tikvclient_backoff_count error
- alert: TiDB_monitor_time_jump_back_error
expr: increase(tidb_monitor_time_jump_back_total[10m]) > 0
for: 1m
labels:
env: test-cluster
level: warning
expr: increase(tidb_monitor_time_jump_back_total[10m]) > 0
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiDB monitor time_jump_back error
- alert: TiDB_ddl_waiting_jobs
expr: sum(tidb_ddl_waiting_jobs) > 5
for: 1m
labels:
env: test-cluster
level: warning
expr: sum(tidb_ddl_waiting_jobs) > 5
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiDB ddl waiting_jobs too much

239
docker/tidb/config/tidb.toml
Unescape Escape View File

@ -1,239 +0,0 @@
# TiDB Configuration.
# TiDB server host.
host = "0.0.0.0"
# TiDB server port.
port = 4000
# Registered store name, [tikv, mocktikv]
store = "mocktikv"
# TiDB storage path.
path = "/tmp/tidb"
# The socket file to use for connection.
socket = ""
# Run ddl worker on this tidb-server.
run-ddl = true
# Schema lease duration, very dangerous to change only if you know what you do.
lease = "0"
# When create table, split a separated region for it. It is recommended to
# turn off this option if there will be a large number of tables created.
split-table = true
# The limit of concurrent executed sessions.
token-limit = 1000
# Only print a log when out of memory quota.
# Valid options: ["log", "cancel"]
oom-action = "log"
# Set the memory quota for a query in bytes. Default: 32GB
mem-quota-query = 34359738368
# Enable coprocessor streaming.
enable-streaming = false
# Set system variable 'lower_case_table_names'
lower-case-table-names = 2
[log]
# Log level: debug, info, warn, error, fatal.
level = "error"
# Log format, one of json, text, console.
format = "text"
# Disable automatic timestamp in output
disable-timestamp = false
# Stores slow query log into separated files.
slow-query-file = ""
# Queries with execution time greater than this value will be logged. (Milliseconds)
slow-threshold = 300
# Queries with internal result greater than this value will be logged.
expensive-threshold = 10000
# Maximum query length recorded in log.
query-log-max-len = 2048
# File logging.
[log.file]
# Log file name.
filename = ""
# Max log file size in MB (upper limit to 4096MB).
max-size = 300
# Max log file keep days. No clean up by default.
max-days = 0
# Maximum number of old log files to retain. No clean up by default.
max-backups = 0
# Rotate log by day
log-rotate = true
[security]
# Path of file that contains list of trusted SSL CAs for connection with mysql client.
ssl-ca = ""
# Path of file that contains X509 certificate in PEM format for connection with mysql client.
ssl-cert = ""
# Path of file that contains X509 key in PEM format for connection with mysql client.
ssl-key = ""
# Path of file that contains list of trusted SSL CAs for connection with cluster components.
cluster-ssl-ca = ""
# Path of file that contains X509 certificate in PEM format for connection with cluster components.
cluster-ssl-cert = ""
# Path of file that contains X509 key in PEM format for connection with cluster components.
cluster-ssl-key = ""
[status]
# If enable status report HTTP service.
report-status = true
# TiDB status port.
status-port = 10080
# Prometheus pushgateway address, leaves it empty will disable prometheus push.
metrics-addr = "pushgateway:9091"
# Prometheus client push interval in second, set \"0\" to disable prometheus push.
metrics-interval = 15
[performance]
# Max CPUs to use, 0 use number of CPUs in the machine.
max-procs = 0
# StmtCountLimit limits the max count of statement inside a transaction.
stmt-count-limit = 5000
# Set keep alive option for tcp connection.
tcp-keep-alive = true
# The maximum number of retries when commit a transaction.
retry-limit = 10
# Whether support cartesian product.
cross-join = true
# Stats lease duration, which influences the time of analyze and stats load.
stats-lease = "3s"
# Run auto analyze worker on this tidb-server.
run-auto-analyze = true
# Probability to use the query feedback to update stats, 0 or 1 for always false/true.
feedback-probability = 0.0
# The max number of query feedback that cache in memory.
query-feedback-limit = 1024
# Pseudo stats will be used if the ratio between the modify count and
# row count in statistics of a table is greater than it.
pseudo-estimate-ratio = 0.7
[proxy-protocol]
# PROXY protocol acceptable client networks.
# Empty string means disable PROXY protocol, * means all networks.
networks = ""
# PROXY protocol header read timeout, unit is second
header-timeout = 5
[plan-cache]
enabled = false
capacity = 2560
shards = 256
[prepared-plan-cache]
enabled = false
capacity = 100
[opentracing]
# Enable opentracing.
enable = false
# Whether to enable the rpc metrics.
rpc-metrics = false
[opentracing.sampler]
# Type specifies the type of the sampler: const, probabilistic, rateLimiting, or remote
type = "const"
# Param is a value passed to the sampler.
# Valid values for Param field are:
# - for "const" sampler, 0 or 1 for always false/true respectively
# - for "probabilistic" sampler, a probability between 0 and 1
# - for "rateLimiting" sampler, the number of spans per second
# - for "remote" sampler, param is the same as for "probabilistic"
# and indicates the initial sampling rate before the actual one
# is received from the mothership
param = 1.0
# SamplingServerURL is the address of jaeger-agent's HTTP sampling server
sampling-server-url = ""
# MaxOperations is the maximum number of operations that the sampler
# will keep track of. If an operation is not tracked, a default probabilistic
# sampler will be used rather than the per operation specific sampler.
max-operations = 0
# SamplingRefreshInterval controls how often the remotely controlled sampler will poll
# jaeger-agent for the appropriate sampling strategy.
sampling-refresh-interval = 0
[opentracing.reporter]
# QueueSize controls how many spans the reporter can keep in memory before it starts dropping
# new spans. The queue is continuously drained by a background go-routine, as fast as spans
# can be sent out of process.
queue-size = 0
# BufferFlushInterval controls how often the buffer is force-flushed, even if it's not full.
# It is generally not useful, as it only matters for very low traffic services.
buffer-flush-interval = 0
# LogSpans, when true, enables LoggingReporter that runs in parallel with the main reporter
# and logs all submitted spans. Main Configuration.Logger must be initialized in the code
# for this option to have any effect.
log-spans = false
# LocalAgentHostPort instructs reporter to send spans to jaeger-agent at this address
local-agent-host-port = ""
[tikv-client]
# Max gRPC connections that will be established with each tikv-server.
grpc-connection-count = 16
# After a duration of this time in seconds if the client doesn't see any activity it pings
# the server to see if the transport is still alive.
grpc-keepalive-time = 10
# After having pinged for keepalive check, the client waits for a duration of Timeout in seconds
# and if no activity is seen even after that the connection is closed.
grpc-keepalive-timeout = 3
# max time for commit command, must be twice bigger than raft election timeout.
commit-timeout = "41s"
[binlog]
# Socket file to write binlog.
binlog-socket = ""
# WriteTimeout specifies how long it will wait for writing binlog to pump.
write-timeout = "15s"
# If IgnoreError is true, when writting binlog meets error, TiDB would stop writting binlog,
# but still provide service.
ignore-error = false

45
docker/tidb/config/tiflash-learner-nightly.toml
Unescape Escape View File

@ -1,45 +0,0 @@
log-file = "/logs/tiflash_tikv.log"
[readpool]
[readpool.coprocessor]
[readpool.storage]
[server]
engine-addr = "tiflash:4030"
addr = "0.0.0.0:20280"
advertise-addr = "tiflash:20280"
#status-addr = "tiflash:20292"
[storage]
data-dir = "/data/flash"
[pd]
[metric]
[raftstore]
capacity = "10GB"
[coprocessor]
[rocksdb]
wal-dir = ""
[rocksdb.defaultcf]
[rocksdb.lockcf]
[rocksdb.writecf]
[raftdb]
[raftdb.defaultcf]
[security]
ca-path = ""
cert-path = ""
key-path = ""
[import]

79
docker/tidb/config/tiflash-nightly.toml
Unescape Escape View File

@ -1,79 +0,0 @@
default_profile = "default"
display_name = "TiFlash"
listen_host = "0.0.0.0"
mark_cache_size = 5368709120
tmp_path = "/data/tmp"
path = "/data"
tcp_port = 9110
http_port = 8223
[flash]
tidb_status_addr = "tidb:10080"
service_addr = "tiflash:4030"
[flash.flash_cluster]
cluster_manager_path = "/tiflash/flash_cluster_manager"
log = "/logs/tiflash_cluster_manager.log"
master_ttl = 60
refresh_interval = 20
update_rule_interval = 5
[flash.proxy]
config = "/tiflash-learner.toml"
[status]
metrics_port = 8234
[logger]
errorlog = "/logs/tiflash_error.log"
log = "/logs/tiflash.log"
count = 20
level = "debug"
size = "1000M"
[application]
runAsDaemon = true
[raft]
pd_addr = "pd0:2379"
storage_engine = "tmt"
[quotas]
[quotas.default]
[quotas.default.interval]
duration = 3600
errors = 0
execution_time = 0
queries = 0
read_rows = 0
result_rows = 0
[users]
[users.default]
password = ""
profile = "default"
quota = "default"
[users.default.networks]
ip = "::/0"
[users.readonly]
password = ""
profile = "readonly"
quota = "default"
[users.readonly.networks]
ip = "::/0"
[profiles]
[profiles.default]
load_balancing = "random"
max_memory_usage = 10000000000
use_uncompressed_cache = 0
[profiles.readonly]
readonly = 1

14099
docker/tidb/config/tikv-dashboard.json
View File

File diff suppressed because it is too large Load Diff

350
docker/tidb/config/tikv.rules.yml
Unescape Escape View File

@ -1,350 +0,0 @@
groups:
- name: alert.rules
rules:
- alert: TiKV_memory_used_too_fast
expr: process_resident_memory_bytes{job=~"tikv.*"} - (process_resident_memory_bytes{job=~"tikv.*"} offset 5m) > 5*1024*1024*1024
for: 5m
labels:
env: test-cluster
level: emergency
expr: process_resident_memory_bytes{job=~"tikv.*"} - (process_resident_memory_bytes{job=~"tikv.*"} offset 5m) > 5*1024*1024*1024
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, job: {{ $labels.job }}, values: {{ $value }}'
value: '{{ $value }}'
summary: TiKV memory used too fast
- alert: TiKV_GC_can_not_work
expr: sum(increase(tidb_tikvclient_gc_action_result{type="success"}[6h])) < 1
for: 1m
labels:
env: test-cluster
level: emergency
expr: sum(increase(tidb_tikvclient_gc_action_result{type="success"}[6h])) < 1
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiKV GC can not work
- alert: TiKV_server_report_failure_msg_total
expr: sum(rate(tikv_server_report_failure_msg_total{type="unreachable"}[10m])) BY (store_id) > 10
for: 1m
labels:
env: test-cluster
level: critical
expr: sum(rate(tikv_server_report_failure_msg_total{type="unreachable"}[10m])) BY (store_id) > 10
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiKV server_report_failure_msg_total error
- alert: TiKV_channel_full_total
expr: sum(rate(tikv_channel_full_total[10m])) BY (type, instance) > 0
for: 1m
labels:
env: test-cluster
level: critical
expr: sum(rate(tikv_channel_full_total[10m])) BY (type, instance) > 0
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiKV channel full
- alert: TiKV_write_stall
expr: delta( tikv_engine_write_stall[10m]) > 0
for: 1m
labels:
env: test-cluster
level: critical
expr: delta( tikv_engine_write_stall[10m]) > 0
annotations:
description: 'cluster: test-cluster, type: {{ $labels.type }}, instance: {{ $labels.instance }}, values: {{ $value }}'
value: '{{ $value }}'
summary: TiKV write stall
- alert: TiKV_raft_log_lag
expr: histogram_quantile(0.99, sum(rate(tikv_raftstore_log_lag_bucket[1m])) by (le, instance, job)) > 5000
for: 1m
labels:
env: test-cluster
level: critical
expr: histogram_quantile(0.99, sum(rate(tikv_raftstore_log_lag_bucket[1m])) by (le, instance, job)) > 5000
annotations:
description: 'cluster: test-cluster, instance {{ $labels.instance }}, values: {{ $value }}'
value: '{{ $value }}'
summary: TiKV raftstore log lag more than 5000
- alert: TiKV_async_request_snapshot_duration_seconds
expr: histogram_quantile(0.99, sum(rate(tikv_storage_engine_async_request_duration_seconds_bucket{type="snapshot"}[1m])) by (le, instance, job,type)) > 1
for: 1m
labels:
env: test-cluster
level: critical
expr: histogram_quantile(0.99, sum(rate(tikv_storage_engine_async_request_duration_seconds_bucket{type="snapshot"}[1m])) by (le, instance, job,type)) > 1
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiKV async request snapshot duration seconds more than 1s
- alert: TiKV_async_request_write_duration_seconds
expr: histogram_quantile(0.99, sum(rate(tikv_storage_engine_async_request_duration_seconds_bucket{type="write"}[1m])) by (le, instance, job,type)) > 1
for: 1m
labels:
env: test-cluster
level: critical
expr: histogram_quantile(0.99, sum(rate(tikv_storage_engine_async_request_duration_seconds_bucket{type="write"}[1m])) by (le, instance, job,type)) > 1
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiKV async request write duration seconds more than 1s
- alert: TiKV_coprocessor_request_wait_seconds
expr: histogram_quantile(0.9999, sum(rate(tikv_coprocessor_request_wait_seconds_bucket[1m])) by (le, instance, job,req)) > 10
for: 1m
labels:
env: test-cluster
level: critical
expr: histogram_quantile(0.9999, sum(rate(tikv_coprocessor_request_wait_seconds_bucket[1m])) by (le, instance, job,req)) > 10
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiKV coprocessor request wait seconds more than 10s
- alert: TiKV_raftstore_thread_cpu_seconds_total
expr: sum(rate(tikv_thread_cpu_seconds_total{name=~"raftstore_.*"}[1m])) by (job, name) > 0.8
for: 1m
labels:
env: test-cluster
level: critical
expr: sum(rate(tikv_thread_cpu_seconds_total{name=~"raftstore_.*"}[1m])) by (job, name) > 0.8
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiKV raftstore thread CPU seconds is high
- alert: TiKV_raft_append_log_duration_secs
expr: histogram_quantile(0.99, sum(rate(tikv_raftstore_append_log_duration_seconds_bucket[1m])) by (le, instance, job)) > 1
for: 1m
labels:
env: test-cluster
level: critical
expr: histogram_quantile(0.99, sum(rate(tikv_raftstore_append_log_duration_seconds_bucket[1m])) by (le, instance, job)) > 1
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiKV_raft_append_log_duration_secs
- alert: TiKV_raft_apply_log_duration_secs
expr: histogram_quantile(0.99, sum(rate(tikv_raftstore_apply_log_duration_seconds_bucket[1m])) by (le, instance, job)) > 1
for: 1m
labels:
env: test-cluster
level: critical
expr: histogram_quantile(0.99, sum(rate(tikv_raftstore_apply_log_duration_seconds_bucket[1m])) by (le, instance, job)) > 1
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiKV_raft_apply_log_duration_secs
- alert: TiKV_scheduler_latch_wait_duration_seconds
expr: histogram_quantile(0.99, sum(rate(tikv_scheduler_latch_wait_duration_seconds_bucket[1m])) by (le, instance, job,type)) > 1
for: 1m
labels:
env: test-cluster
level: critical
expr: histogram_quantile(0.99, sum(rate(tikv_scheduler_latch_wait_duration_seconds_bucket[1m])) by (le, instance, job,type)) > 1
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiKV scheduler latch wait duration seconds more than 1s
- alert: TiKV_thread_apply_worker_cpu_seconds
expr: sum(rate(tikv_thread_cpu_seconds_total{name="apply_worker"}[1m])) by (job) > 0.9
for: 1m
labels:
env: test-cluster
level: critical
expr: sum(rate(tikv_thread_cpu_seconds_total{name="apply_worker"}[1m])) by (job) > 0.9
annotations:
description: 'cluster: test-cluster, type: {{ $labels.type }}, instance: {{ $labels.instance }}, values: {{ $value }}'
value: '{{ $value }}'
summary: TiKV thread apply worker cpu seconds is high
- alert: TiDB_tikvclient_gc_action_fail
expr: sum(increase(tidb_tikvclient_gc_action_result{type="fail"}[1m])) > 10
for: 1m
labels:
env: test-cluster
level: critical
expr: sum(increase(tidb_tikvclient_gc_action_result{type="fail"}[1m])) > 10
annotations:
description: 'cluster: test-cluster, type: {{ $labels.type }}, instance: {{ $labels.instance }}, values: {{ $value }}'
value: '{{ $value }}'
summary: TiDB_tikvclient_gc_action_fail
- alert: TiKV_leader_drops
expr: delta(tikv_pd_heartbeat_tick_total{type="leader"}[30s]) < -10
for: 1m
labels:
env: test-cluster
level: warning
expr: delta(tikv_pd_heartbeat_tick_total{type="leader"}[30s]) < -10
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiKV leader drops
- alert: TiKV_raft_process_ready_duration_secs
expr: histogram_quantile(0.999, sum(rate(tikv_raftstore_raft_process_duration_secs_bucket{type='ready'}[1m])) by (le, instance, job,type)) > 2
for: 1m
labels:
env: test-cluster
level: warning
expr: histogram_quantile(0.999, sum(rate(tikv_raftstore_raft_process_duration_secs_bucket{type='ready'}[1m])) by (le, instance, job,type)) > 2
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values: {{ $value }}'
value: '{{ $value }}'
summary: TiKV_raft_process_ready_duration_secs
- alert: TiKV_raft_process_tick_duration_secs
expr: histogram_quantile(0.999, sum(rate(tikv_raftstore_raft_process_duration_secs_bucket{type='tick'}[1m])) by (le, instance, job,type)) > 2
for: 1m
labels:
env: test-cluster
level: warning
expr: histogram_quantile(0.999, sum(rate(tikv_raftstore_raft_process_duration_secs_bucket{type='tick'}[1m])) by (le, instance, job,type)) > 2
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values: {{ $value }}'
value: '{{ $value }}'
summary: TiKV_raft_process_tick_duration_secs
- alert: TiKV_scheduler_context_total
expr: abs(delta( tikv_scheduler_contex_total[5m])) > 1000
for: 1m
labels:
env: test-cluster
level: warning
expr: abs(delta( tikv_scheduler_contex_total[5m])) > 1000
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiKV scheduler context total
- alert: TiKV_scheduler_command_duration_seconds
expr: histogram_quantile(0.99, sum(rate(tikv_scheduler_command_duration_seconds_bucket[1m])) by (le, instance, job,type) / 1000) > 1
for: 1m
labels:
env: test-cluster
level: warning
expr: histogram_quantile(0.99, sum(rate(tikv_scheduler_command_duration_seconds_bucket[1m])) by (le, instance, job,type) / 1000) > 1
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiKV scheduler command duration seconds more than 1s
- alert: TiKV_thread_storage_scheduler_cpu_seconds
expr: sum(rate(tikv_thread_cpu_seconds_total{name=~"storage_schedul.*"}[1m])) by (job) > 0.8
for: 1m
labels:
env: test-cluster
level: warning
expr: sum(rate(tikv_thread_cpu_seconds_total{name=~"storage_schedul.*"}[1m])) by (job) > 0.8
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values:{{ $value }}'
value: '{{ $value }}'
summary: TiKV storage scheduler cpu seconds more than 80%
- alert: TiKV_coprocessor_outdated_request_wait_seconds
expr: delta( tikv_coprocessor_outdated_request_wait_seconds_count[10m] ) > 0
for: 1m
labels:
env: test-cluster
level: warning
expr: delta( tikv_coprocessor_outdated_request_wait_seconds_count[10m] ) > 0
annotations:
description: 'cluster: test-cluster, instance: {{ $labels.instance }}, values: {{ $value }}'
value: '{{ $value }}'
summary: TiKV coprocessor outdated request wait seconds
- alert: TiKV_coprocessor_request_error
expr: increase(tikv_coprocessor_request_error{reason!="lock"}[10m]) > 100
for: 1m
labels:
env: test-cluster
level: warning
expr: increase(tikv_coprocessor_request_error{reason!="lock"}[10m]) > 100
annotations:
description: 'cluster: test-cluster, reason: {{ $labels.reason }}, instance: {{ $labels.instance }}, values: {{ $value }}'
value: '{{ $value }}'
summary: TiKV coprocessor request error
- alert: TiKV_coprocessor_request_lock_error
expr: increase(tikv_coprocessor_request_error{reason="lock"}[10m]) > 10000
for: 1m
labels:
env: test-cluster
level: warning
expr: increase(tikv_coprocessor_request_error{reason="lock"}[10m]) > 10000
annotations:
description: 'cluster: test-cluster, reason: {{ $labels.reason }}, instance: {{ $labels.instance }}, values: {{ $value }}'
value: '{{ $value }}'
summary: TiKV coprocessor request lock error
- alert: TiKV_coprocessor_pending_request
expr: delta( tikv_coprocessor_pending_request[10m]) > 5000
for: 1m
labels:
env: test-cluster
level: warning
expr: delta( tikv_coprocessor_pending_request[10m]) > 5000
annotations:
description: 'cluster: test-cluster, type: {{ $labels.type }}, instance: {{ $labels.instance }}, values: {{ $value }}'
value: '{{ $value }}'
summary: TiKV pending {{ $labels.type }} request is high
- alert: TiKV_batch_request_snapshot_nums
expr: sum(rate(tikv_thread_cpu_seconds_total{name=~"endpoint.*"}[1m])) by (job) / ( count(tikv_thread_cpu_seconds_total{name=~"endpoint.*"}) * 0.9 ) / count(count(tikv_thread_cpu_seconds_total) by (instance)) > 0
for: 1m
labels:
env: test-cluster
level: warning
expr: sum(rate(tikv_thread_cpu_seconds_total{name=~"endpoint.*"}[1m])) by (job) / ( count(tikv_thread_cpu_seconds_total{name=~"endpoint.*"}) * 0.9 ) / count(count(tikv_thread_cpu_seconds_total) by (instance)) > 0
annotations:
description: 'cluster: test-cluster, type: {{ $labels.type }}, instance: {{ $labels.instance }}, values: {{ $value }}'
value: '{{ $value }}'
summary: TiKV batch request snapshot nums is high
- alert: TiKV_pending_task
expr: sum(tikv_worker_pending_task_total) BY (job,instance,name) > 1000
for: 1m
labels:
env: test-cluster
level: warning
expr: sum(tikv_worker_pending_task_total) BY (job,instance,name) > 1000
annotations:
description: 'cluster: test-cluster, type: {{ $labels.type }}, instance: {{ $labels.instance }}, values: {{ $value }}'
value: '{{ $value }}'
summary: TiKV pending task too much
- alert: TiKV_low_space_and_add_region
expr: count( (sum(tikv_store_size_bytes{type="available"}) by (job) / sum(tikv_store_size_bytes{type="capacity"}) by (job) < 0.2) and (sum(tikv_raftstore_snapshot_traffic_total{type="applying"}) by (job) > 0 ) ) > 0
for: 1m
labels:
env: test-cluster
level: warning
expr: count( (sum(tikv_store_size_bytes{type="available"}) by (job) / sum(tikv_store_size_bytes{type="capacity"}) by (job) < 0.2) and (sum(tikv_raftstore_snapshot_traffic_total{type="applying"}) by (job) > 0 ) ) > 0
annotations:
description: 'cluster: test-cluster, type: {{ $labels.type }}, instance: {{ $labels.instance }}, values: {{ $value }}'
value: '{{ $value }}'
summary: TiKV low_space and add_region
- alert: TiKV_approximate_region_size
expr: histogram_quantile(0.99, sum(rate(tikv_raftstore_region_size_bucket[1m])) by (le)) > 1073741824
for: 1m
labels:
env: test-cluster
level: warning
expr: histogram_quantile(0.99, sum(rate(tikv_raftstore_region_size_bucket[1m])) by (le)) > 1073741824
annotations:
description: 'cluster: test-cluster, type: {{ $labels.type }}, instance: {{ $labels.instance }}, values: {{ $value }}'
value: '{{ $value }}'
summary: TiKV approximate region size is more than 1GB

497
docker/tidb/config/tikv.toml
Unescape Escape View File

@ -1,497 +0,0 @@
# TiKV config template
# Human-readable big numbers:
# File size(based on byte): KB, MB, GB, TB, PB
# e.g.: 1_048_576 = "1MB"
# Time(based on ms): ms, s, m, h
# e.g.: 78_000 = "1.3m"
# log level: trace, debug, info, warn, error, off.
log-level = "error"
# file to store log, write to stderr if it's empty.
# log-file = ""
[readpool.storage]
# size of thread pool for high-priority operations
# high-concurrency = 4
# size of thread pool for normal-priority operations
# normal-concurrency = 4
# size of thread pool for low-priority operations
# low-concurrency = 4
# max running high-priority operations, reject if exceed
# max-tasks-high = 8000
# max running normal-priority operations, reject if exceed
# max-tasks-normal = 8000
# max running low-priority operations, reject if exceed
# max-tasks-low = 8000
# size of stack size for each thread pool
# stack-size = "10MB"
[readpool.coprocessor]
# Notice: if CPU_NUM > 8, default thread pool size for coprocessors
# will be set to CPU_NUM * 0.8.
# high-concurrency = 8
# normal-concurrency = 8
# low-concurrency = 8
# max-tasks-high = 16000
# max-tasks-normal = 16000
# max-tasks-low = 16000
# stack-size = "10MB"
[server]
# set listening address.
# addr = "127.0.0.1:20160"
# set advertise listening address for client communication, if not set, use addr instead.
# advertise-addr = ""
# notify capacity, 40960 is suitable for about 7000 regions.
# notify-capacity = 40960
# maximum number of messages can be processed in one tick.
# messages-per-tick = 4096
# compression type for grpc channel, available values are no, deflate and gzip.
# grpc-compression-type = "no"
# size of thread pool for grpc server.
# grpc-concurrency = 4
# The number of max concurrent streams/requests on a client connection.
# grpc-concurrent-stream = 1024
# The number of connections with each tikv server to send raft messages.
# grpc-raft-conn-num = 10
# Amount to read ahead on individual grpc streams.
# grpc-stream-initial-window-size = "2MB"
# How many snapshots can be sent concurrently.
# concurrent-send-snap-limit = 32
# How many snapshots can be recv concurrently.
# concurrent-recv-snap-limit = 32
# max count of tasks being handled, new tasks will be rejected.
# end-point-max-tasks = 2000
# max recursion level allowed when decoding dag expression
# end-point-recursion-limit = 1000
# max time to handle coprocessor request before timeout
# end-point-request-max-handle-duration = "60s"
# the max bytes that snapshot can be written to disk in one second,
# should be set based on your disk performance
# snap-max-write-bytes-per-sec = "100MB"
# set attributes about this server, e.g. { zone = "us-west-1", disk = "ssd" }.
# labels = {}
[storage]
# set the path to rocksdb directory.
# data-dir = "/tmp/tikv/store"
# notify capacity of scheduler's channel
# scheduler-notify-capacity = 10240
# maximum number of messages can be processed in one tick
# scheduler-messages-per-tick = 1024
# the number of slots in scheduler latches, concurrency control for write.
# scheduler-concurrency = 2048000
# scheduler's worker pool size, should increase it in heavy write cases,
# also should less than total cpu cores.
# scheduler-worker-pool-size = 4
# When the pending write bytes exceeds this threshold,
# the "scheduler too busy" error is displayed.
# scheduler-pending-write-threshold = "100MB"
[pd]
# pd endpoints
# endpoints = []
[metric]
# the Prometheus client push interval. Setting the value to 0s stops Prometheus client from pushing.
# interval = "15s"
# the Prometheus pushgateway address. Leaving it empty stops Prometheus client from pushing.
address = "pushgateway:9091"
# the Prometheus client push job name. Note: A node id will automatically append, e.g., "tikv_1".
# job = "tikv"
[raftstore]
# true (default value) for high reliability, this can prevent data loss when power failure.
# sync-log = true
# set the path to raftdb directory, default value is data-dir/raft
# raftdb-path = ""
# set store capacity, if no set, use disk capacity.
# capacity = 0
# notify capacity, 40960 is suitable for about 7000 regions.
# notify-capacity = 40960
# maximum number of messages can be processed in one tick.
# messages-per-tick = 4096
# Region heartbeat tick interval for reporting to pd.
# pd-heartbeat-tick-interval = "60s"
# Store heartbeat tick interval for reporting to pd.
# pd-store-heartbeat-tick-interval = "10s"
# When region size changes exceeds region-split-check-diff, we should check
# whether the region should be split or not.
# region-split-check-diff = "6MB"
# Interval to check region whether need to be split or not.
# split-region-check-tick-interval = "10s"
# When raft entry exceed the max size, reject to propose the entry.
# raft-entry-max-size = "8MB"
# Interval to gc unnecessary raft log.
# raft-log-gc-tick-interval = "10s"
# A threshold to gc stale raft log, must >= 1.
# raft-log-gc-threshold = 50
# When entry count exceed this value, gc will be forced trigger.
# raft-log-gc-count-limit = 72000
# When the approximate size of raft log entries exceed this value, gc will be forced trigger.
# It's recommanded to set it to 3/4 of region-split-size.
# raft-log-gc-size-limit = "72MB"
# When a peer hasn't been active for max-peer-down-duration,
# we will consider this peer to be down and report it to pd.
# max-peer-down-duration = "5m"
# Interval to check whether start manual compaction for a region,
# region-compact-check-interval = "5m"
# Number of regions for each time to check.
# region-compact-check-step = 100
# The minimum number of delete tombstones to trigger manual compaction.
# region-compact-min-tombstones = 10000
# Interval to check whether should start a manual compaction for lock column family,
# if written bytes reach lock-cf-compact-threshold for lock column family, will fire
# a manual compaction for lock column family.
# lock-cf-compact-interval = "10m"
# lock-cf-compact-bytes-threshold = "256MB"
# Interval (s) to check region whether the data are consistent.
# consistency-check-interval = 0
# Use delete range to drop a large number of continuous keys.
# use-delete-range = false
# delay time before deleting a stale peer
# clean-stale-peer-delay = "10m"
# Interval to cleanup import sst files.
# cleanup-import-sst-interval = "10m"
[coprocessor]
# When it is true, it will try to split a region with table prefix if
# that region crosses tables. It is recommended to turn off this option
# if there will be a large number of tables created.
# split-region-on-table = true
# When the region's size exceeds region-max-size, we will split the region
# into two which the left region's size will be region-split-size or a little
# bit smaller.
# region-max-size = "144MB"
# region-split-size = "96MB"
[rocksdb]
# Maximum number of concurrent background jobs (compactions and flushes)
# max-background-jobs = 8
# This value represents the maximum number of threads that will concurrently perform a
# compaction job by breaking it into multiple, smaller ones that are run simultaneously.
# Default: 1 (i.e. no subcompactions)
# max-sub-compactions = 1
# Number of open files that can be used by the DB. You may need to
# increase this if your database has a large working set. Value -1 means
# files opened are always kept open. You can estimate number of files based
# on target_file_size_base and target_file_size_multiplier for level-based
# compaction.
# If max-open-files = -1, RocksDB will prefetch index and filter blocks into
# block cache at startup, so if your database has a large working set, it will
# take several minutes to open the db.
max-open-files = 1024
# Max size of rocksdb's MANIFEST file.
# For detailed explanation please refer to https://github.com/facebook/rocksdb/wiki/MANIFEST
# max-manifest-file-size = "20MB"
# If true, the database will be created if it is missing.
# create-if-missing = true
# rocksdb wal recovery mode
# 0 : TolerateCorruptedTailRecords, tolerate incomplete record in trailing data on all logs;
# 1 : AbsoluteConsistency, We don't expect to find any corruption in the WAL;
# 2 : PointInTimeRecovery, Recover to point-in-time consistency;
# 3 : SkipAnyCorruptedRecords, Recovery after a disaster;
# wal-recovery-mode = 2
# rocksdb write-ahead logs dir path
# This specifies the absolute dir path for write-ahead logs (WAL).
# If it is empty, the log files will be in the same dir as data.
# When you set the path to rocksdb directory in memory like in /dev/shm, you may want to set
# wal-dir to a directory on a persistent storage.
# See https://github.com/facebook/rocksdb/wiki/How-to-persist-in-memory-RocksDB-database
# wal-dir = "/tmp/tikv/store"
# The following two fields affect how archived write-ahead logs will be deleted.
# 1. If both set to 0, logs will be deleted asap and will not get into the archive.
# 2. If wal-ttl-seconds is 0 and wal-size-limit is not 0,
# WAL files will be checked every 10 min and if total size is greater
# then wal-size-limit, they will be deleted starting with the
# earliest until size_limit is met. All empty files will be deleted.
# 3. If wal-ttl-seconds is not 0 and wal-size-limit is 0, then
# WAL files will be checked every wal-ttl-seconds / 2 and those that
# are older than wal-ttl-seconds will be deleted.
# 4. If both are not 0, WAL files will be checked every 10 min and both
# checks will be performed with ttl being first.
# When you set the path to rocksdb directory in memory like in /dev/shm, you may want to set
# wal-ttl-seconds to a value greater than 0 (like 86400) and backup your db on a regular basis.
# See https://github.com/facebook/rocksdb/wiki/How-to-persist-in-memory-RocksDB-database
# wal-ttl-seconds = 0
# wal-size-limit = 0
# rocksdb max total wal size
# max-total-wal-size = "4GB"
# Rocksdb Statistics provides cumulative stats over time.
# Turn statistics on will introduce about 5%-10% overhead for RocksDB,
# but it is worthy to know the internal status of RocksDB.
# enable-statistics = true
# Dump statistics periodically in information logs.
# Same as rocksdb's default value (10 min).
# stats-dump-period = "10m"
# Due to Rocksdb FAQ: https://github.com/facebook/rocksdb/wiki/RocksDB-FAQ,
# If you want to use rocksdb on multi disks or spinning disks, you should set value at
# least 2MB;
# compaction-readahead-size = 0
# This is the maximum buffer size that is used by WritableFileWrite
# writable-file-max-buffer-size = "1MB"
# Use O_DIRECT for both reads and writes in background flush and compactions
# use-direct-io-for-flush-and-compaction = false
# Limit the disk IO of compaction and flush. Compaction and flush can cause
# terrible spikes if they exceed a certain threshold. Consider setting this to
# 50% ~ 80% of the disk throughput for a more stable result. However, in heavy
# write workload, limiting compaction and flush speed can cause write stalls too.
# rate-bytes-per-sec = 0
# Enable or disable the pipelined write
# enable-pipelined-write = true
# Allows OS to incrementally sync files to disk while they are being
# written, asynchronously, in the background.
# bytes-per-sync = "0MB"
# Allows OS to incrementally sync WAL to disk while it is being written.
# wal-bytes-per-sync = "0KB"
# Specify the maximal size of the Rocksdb info log file. If the log file
# is larger than `max_log_file_size`, a new info log file will be created.
# If max_log_file_size == 0, all logs will be written to one log file.
# Default: 1GB
# info-log-max-size = "1GB"
# Time for the Rocksdb info log file to roll (in seconds).
# If specified with non-zero value, log file will be rolled
# if it has been active longer than `log_file_time_to_roll`.
# Default: 0 (disabled)
# info-log-roll-time = "0"
# Maximal Rocksdb info log files to be kept.
# Default: 10
# info-log-keep-log-file-num = 10
# This specifies the Rocksdb info LOG dir.
# If it is empty, the log files will be in the same dir as data.
# If it is non empty, the log files will be in the specified dir,
# and the db data dir's absolute path will be used as the log file
# name's prefix.
# Default: empty
# info-log-dir = ""
# Column Family default used to store actual data of the database.
[rocksdb.defaultcf]
# compression method (if any) is used to compress a block.
# no: kNoCompression
# snappy: kSnappyCompression
# zlib: kZlibCompression
# bzip2: kBZip2Compression
# lz4: kLZ4Compression
# lz4hc: kLZ4HCCompression
# zstd: kZSTD
# per level compression
# compression-per-level = ["no", "no", "lz4", "lz4", "lz4", "zstd", "zstd"]
# Approximate size of user data packed per block. Note that the
# block size specified here corresponds to uncompressed data.
# block-size = "64KB"
# If you're doing point lookups you definitely want to turn bloom filters on, We use
# bloom filters to avoid unnecessary disk reads. Default bits_per_key is 10, which
# yields ~1% false positive rate. Larger bits_per_key values will reduce false positive
# rate, but increase memory usage and space amplification.
# bloom-filter-bits-per-key = 10
# false means one sst file one bloom filter, true means evry block has a corresponding bloom filter
# block-based-bloom-filter = false
# level0-file-num-compaction-trigger = 4
# Soft limit on number of level-0 files. We start slowing down writes at this point.
# level0-slowdown-writes-trigger = 20
# Maximum number of level-0 files. We stop writes at this point.
# level0-stop-writes-trigger = 36
# Amount of data to build up in memory (backed by an unsorted log
# on disk) before converting to a sorted on-disk file.
# write-buffer-size = "128MB"
# The maximum number of write buffers that are built up in memory.
# max-write-buffer-number = 5
# The minimum number of write buffers that will be merged together
# before writing to storage.
# min-write-buffer-number-to-merge = 1
# Control maximum total data size for base level (level 1).
# max-bytes-for-level-base = "512MB"
# Target file size for compaction.
# target-file-size-base = "8MB"
# Max bytes for compaction.max_compaction_bytes
# max-compaction-bytes = "2GB"
# There are four different algorithms to pick files to compact.
# 0 : ByCompensatedSize
# 1 : OldestLargestSeqFirst
# 2 : OldestSmallestSeqFirst
# 3 : MinOverlappingRatio
# compaction-pri = 3
# block-cache used to cache uncompressed blocks, big block-cache can speed up read.
# in normal cases should tune to 30%-50% system's total memory.
# block-cache-size = "1GB"
# Indicating if we'd put index/filter blocks to the block cache.
# If not specified, each "table reader" object will pre-load index/filter block
# during table initialization.
# cache-index-and-filter-blocks = true
# Pin level0 filter and index blocks in cache.
# pin-l0-filter-and-index-blocks = true
# Enable read amplication statistics.
# value => memory usage (percentage of loaded blocks memory)
# 1 => 12.50 %
# 2 => 06.25 %
# 4 => 03.12 %
# 8 => 01.56 %
# 16 => 00.78 %
# read-amp-bytes-per-bit = 0
# Pick target size of each level dynamically.
# dynamic-level-bytes = true
# Options for Column Family write
# Column Family write used to store commit informations in MVCC model
[rocksdb.writecf]
# compression-per-level = ["no", "no", "lz4", "lz4", "lz4", "zstd", "zstd"]
# block-size = "64KB"
# write-buffer-size = "128MB"
# max-write-buffer-number = 5
# min-write-buffer-number-to-merge = 1
# max-bytes-for-level-base = "512MB"
# target-file-size-base = "8MB"
# in normal cases should tune to 10%-30% system's total memory.
# block-cache-size = "256MB"
# level0-file-num-compaction-trigger = 4
# level0-slowdown-writes-trigger = 20
# level0-stop-writes-trigger = 36
# cache-index-and-filter-blocks = true
# pin-l0-filter-and-index-blocks = true
# compaction-pri = 3
# read-amp-bytes-per-bit = 0
# dynamic-level-bytes = true
[rocksdb.lockcf]
# compression-per-level = ["no", "no", "no", "no", "no", "no", "no"]
# block-size = "16KB"
# write-buffer-size = "128MB"
# max-write-buffer-number = 5
# min-write-buffer-number-to-merge = 1
# max-bytes-for-level-base = "128MB"
# target-file-size-base = "8MB"
# block-cache-size = "256MB"
# level0-file-num-compaction-trigger = 1
# level0-slowdown-writes-trigger = 20
# level0-stop-writes-trigger = 36
# cache-index-and-filter-blocks = true
# pin-l0-filter-and-index-blocks = true
# compaction-pri = 0
# read-amp-bytes-per-bit = 0
# dynamic-level-bytes = true
[raftdb]
# max-sub-compactions = 1
max-open-files = 1024
# max-manifest-file-size = "20MB"
# create-if-missing = true
# enable-statistics = true
# stats-dump-period = "10m"
# compaction-readahead-size = 0
# writable-file-max-buffer-size = "1MB"
# use-direct-io-for-flush-and-compaction = false
# enable-pipelined-write = true
# allow-concurrent-memtable-write = false
# bytes-per-sync = "0MB"
# wal-bytes-per-sync = "0KB"
# info-log-max-size = "1GB"
# info-log-roll-time = "0"
# info-log-keep-log-file-num = 10
# info-log-dir = ""
[raftdb.defaultcf]
# compression-per-level = ["no", "no", "lz4", "lz4", "lz4", "zstd", "zstd"]
# block-size = "64KB"
# write-buffer-size = "128MB"
# max-write-buffer-number = 5
# min-write-buffer-number-to-merge = 1
# max-bytes-for-level-base = "512MB"
# target-file-size-base = "8MB"
# should tune to 256MB~2GB.
# block-cache-size = "256MB"
# level0-file-num-compaction-trigger = 4
# level0-slowdown-writes-trigger = 20
# level0-stop-writes-trigger = 36
# cache-index-and-filter-blocks = true
# pin-l0-filter-and-index-blocks = true
# compaction-pri = 0
# read-amp-bytes-per-bit = 0
# dynamic-level-bytes = true
[security]
# set the path for certificates. Empty string means disabling secure connectoins.
# ca-path = ""
# cert-path = ""
# key-path = ""
[import]
# the directory to store importing kv data.
# import-dir = "/tmp/tikv/import"
# number of threads to handle RPC requests.
# num-threads = 8
# stream channel window size, stream will be blocked on channel full.
# stream-channel-window = 128

7
docker/tidb/dashboard-installer/Dockerfile
Unescape Escape View File

@ -1,7 +0,0 @@
FROM python:2.7-alpine
RUN apk add --no-cache ca-certificates curl
ADD dashboards /
ENTRYPOINT ["/tidb-dashboard-installer.sh"]

13
docker/tidb/dashboard-installer/README.md
Unescape Escape View File

@ -1,13 +0,0 @@
# TiDB dashboard installer
This image is used to configure Grafana datasource and dashboards for TiDB cluster. It is used in [tidb-docker-compose](https://github.com/pingcap/tidb-docker-compose) and [tidb-operator](https://github.com/pingcap/tidb-operator).
The JSON files in dashboards are copied from [tidb-ansible](https://github.com/pingcap/tidb-ansible/tree/master/scripts).
Grafana version prior to v5.0.0 can only use import API to automate datasource and dashboard configuration. So this image is needed to run in docker environment. It runs only once in this environment.
With Grafana v5.x, we can use [provisioning](http://docs.grafana.org/administration/provisioning) feature to statically provision datasources and dashboards. No need to use scripts to configure Grafana.
But currently, the dashboards in [tidb-ansible](https://github.com/pingcap/tidb-ansible/tree/master/scripts) repository are incompatible with Grafana v5.x and cannot be statically provisioned. So this image is still required.
In the future, we can use [grafonnet](https://github.com/grafana/grafonnet-lib) to migrate old dashboards and make dashboard updating reviewable.

201
docker/tidb/dashboard-installer/dashboards/LICENSE
Unescape Escape View File

@ -1,201 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "{}"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright {}
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

7
docker/tidb/dashboard-installer/dashboards/datasource.json
Unescape Escape View File

@ -1,7 +0,0 @@
{
"name": "tidb-cluster",
"type": "prometheus",
"url": "http://127.0.0.1:9090",
"access": "proxy",
"basicAuth": false
}

12
docker/tidb/dashboard-installer/dashboards/dests.json
Unescape Escape View File

@ -1,12 +0,0 @@
[
{
"datasource": "tidb-cluster",
"name": "TiDB-Cluster",
"titles": {
"overview": "TiDB-Cluster-Overview",
"pd": "TiDB-Cluster-PD",
"tidb": "TiDB-Cluster-TiDB",
"tikv": "TiDB-Cluster-TiKV"
}
}
]

110
docker/tidb/dashboard-installer/dashboards/grafana-config-copy.py
Unescape Escape View File

@ -1,110 +0,0 @@
#!/usr/bin/env python
from __future__ import print_function, \
unicode_literals
import urllib
import urllib2
import base64
import json
import sys
from pprint import pprint
try:
input = raw_input
except:
pass
############################################################
################## CONFIGURATION ###########################
############################################################
# use a viewer key
src = dict(
dashboards={
"pd" : 'pd.json',
"tidb": 'tidb.json',
"tikv": 'tikv.json',
"overview": 'overview.json'
})
dests = [
]
if not dests:
with open("./dests.json") as fp:
dests = json.load(fp)
############################################################
################## CONFIGURATION ENDS ######################
############################################################
def export_dashboard(api_url, api_key, dashboard_name):
req = urllib2.Request(api_url + 'api/dashboards/db/' + dashboard_name,
headers={'Authorization': "Bearer {}".format(api_key)})
resp = urllib2.urlopen(req)
data = json.load(resp)
return data['dashboard']
def fill_dashboard_with_dest_config(dashboard, dest, type_='node'):
dashboard['title'] = dest['titles'][type_]
dashboard['id'] = None
# pprint(dashboard)
for row in dashboard['rows']:
for panel in row['panels']:
panel['datasource'] = dest['datasource']
if 'templating' in dashboard:
for templating in dashboard['templating']['list']:
if templating['type'] == 'query':
templating['current'] = {}
templating['options'] = []
templating['datasource'] = dest['datasource']
if 'annotations' in dashboard:
for annotation in dashboard['annotations']['list']:
annotation['datasource'] = dest['datasource']
return dashboard
def import_dashboard_via_user_pass(api_url, user, password, dashboard):
payload = {'dashboard': dashboard,
'overwrite': True}
auth_string = base64.b64encode('%s:%s' % (user, password))
headers = {'Authorization': "Basic {}".format(auth_string),
'Content-Type': 'application/json'}
req = urllib2.Request(api_url + 'api/dashboards/db',
headers=headers,
data=json.dumps(payload))
try:
resp = urllib2.urlopen(req)
data = json.load(resp)
return data
except urllib2.HTTPError, error:
data = json.load(error)
return data
if __name__ == '__main__':
url = sys.argv[1]
user = sys.argv[2]
password = sys.argv[3]
print(url)
for type_ in src['dashboards']:
print("[load] from <{}>:{}".format(
src['dashboards'][type_], type_))
dashboard = json.load(open(src['dashboards'][type_]))
for dest in dests:
dashboard = fill_dashboard_with_dest_config(dashboard, dest, type_)
print("[import] as <{}> to [{}]".format(
dashboard['title'], dest['name']), end='\t............. ')
ret = import_dashboard_via_user_pass(url, user, password, dashboard)
print(ret)
if ret['status'] != 'success':
print(' > ERROR: ', ret)
raise RuntimeError

4892
docker/tidb/dashboard-installer/dashboards/overview.json
View File

File diff suppressed because it is too large Load Diff

5698
docker/tidb/dashboard-installer/dashboards/pd.json
View File

File diff suppressed because it is too large Load Diff

12
docker/tidb/dashboard-installer/dashboards/tidb-dashboard-installer.sh
Unescape Escape View File

@ -1,12 +0,0 @@
#!/bin/sh
url=$1
userName=${GRAFANA_USERNAME:-admin}
password=${GRAFANA_PASSWORD:-admin}
datasource_url="http://${url}/api/datasources"
echo "Adding datasource..."
until curl -s -XPOST -H "Content-Type: application/json" --connect-timeout 1 -u ${userName}:${password} ${datasource_url} -d @/datasource.json >/dev/null; do
sleep 1
done
python grafana-config-copy.py "http://${url}/" ${userName} ${password}

6928
docker/tidb/dashboard-installer/dashboards/tidb.json
View File

File diff suppressed because it is too large Load Diff

14066
docker/tidb/dashboard-installer/dashboards/tikv.json
View File

File diff suppressed because it is too large Load Diff

448
docker/tidb/docker-compose-binlog.yml
Unescape Escape View File

@ -1,448 +0,0 @@
---
# Source: tidb-docker-compose/templates/docker-compose.yml
# dashboard-installer has been deleted, because we don't need it any more in docker-compose model
version: '2.1'
services:
pd0:
image: pingcap/pd:latest
ports:
- "2379"
volumes:
- ./config/pd.toml:/pd.toml:ro
- ./data:/data
- ./logs:/logs
command:
- --name=pd0
- --client-urls=http://0.0.0.0:2379
- --peer-urls=http://0.0.0.0:2380
- --advertise-client-urls=http://pd0:2379
- --advertise-peer-urls=http://pd0:2380
- --initial-cluster=pd0=http://pd0:2380,pd1=http://pd1:2380,pd2=http://pd2:2380
- --data-dir=/data/pd0
- --config=/pd.toml
- --log-file=/logs/pd0.log
# sysctls:
# net.core.somaxconn: 32768
# ulimits:
# nofile:
# soft: 1000000
# hard: 1000000
restart: on-failure
pd1:
image: pingcap/pd:latest
ports:
- "2379"
volumes:
- ./config/pd.toml:/pd.toml:ro
- ./data:/data
- ./logs:/logs
command:
- --name=pd1
- --client-urls=http://0.0.0.0:2379
- --peer-urls=http://0.0.0.0:2380
- --advertise-client-urls=http://pd1:2379
- --advertise-peer-urls=http://pd1:2380
- --initial-cluster=pd0=http://pd0:2380,pd1=http://pd1:2380,pd2=http://pd2:2380
- --data-dir=/data/pd1
- --config=/pd.toml
- --log-file=/logs/pd1.log
# sysctls:
# net.core.somaxconn: 32768
# ulimits:
# nofile:
# soft: 1000000
# hard: 1000000
restart: on-failure
pd2:
image: pingcap/pd:latest
ports:
- "2379"
volumes:
- ./config/pd.toml:/pd.toml:ro
- ./data:/data
- ./logs:/logs
command:
- --name=pd2
- --client-urls=http://0.0.0.0:2379
- --peer-urls=http://0.0.0.0:2380
- --advertise-client-urls=http://pd2:2379
- --advertise-peer-urls=http://pd2:2380
- --initial-cluster=pd0=http://pd0:2380,pd1=http://pd1:2380,pd2=http://pd2:2380
- --data-dir=/data/pd2
- --config=/pd.toml
- --log-file=/logs/pd2.log
# sysctls:
# net.core.somaxconn: 32768
# ulimits:
# nofile:
# soft: 1000000
# hard: 1000000
restart: on-failure
tikv0:
image: pingcap/tikv:latest
volumes:
- ./config/tikv.toml:/tikv.toml:ro
- ./data:/data
- ./logs:/logs
command:
- --addr=0.0.0.0:20160
- --advertise-addr=tikv0:20160
- --data-dir=/data/tikv0
- --pd=pd0:2379,pd1:2379,pd2:2379
- --config=/tikv.toml
- --log-file=/logs/tikv0.log
depends_on:
- "pd0"
- "pd1"
- "pd2"
# sysctls:
# net.core.somaxconn: 32768
# ulimits:
# nofile:
# soft: 1000000
# hard: 1000000
restart: on-failure
tikv1:
image: pingcap/tikv:latest
volumes:
- ./config/tikv.toml:/tikv.toml:ro
- ./data:/data
- ./logs:/logs
command:
- --addr=0.0.0.0:20160
- --advertise-addr=tikv1:20160
- --data-dir=/data/tikv1
- --pd=pd0:2379,pd1:2379,pd2:2379
- --config=/tikv.toml
- --log-file=/logs/tikv1.log
depends_on:
- "pd0"
- "pd1"
- "pd2"
# sysctls:
# net.core.somaxconn: 32768
# ulimits:
# nofile:
# soft: 1000000
# hard: 1000000
restart: on-failure
tikv2:
image: pingcap/tikv:latest
volumes:
- ./config/tikv.toml:/tikv.toml:ro
- ./data:/data
- ./logs:/logs
command:
- --addr=0.0.0.0:20160
- --advertise-addr=tikv2:20160
- --data-dir=/data/tikv2
- --pd=pd0:2379,pd1:2379,pd2:2379
- --config=/tikv.toml
- --log-file=/logs/tikv2.log
depends_on:
- "pd0"
- "pd1"
- "pd2"
# sysctls:
# net.core.somaxconn: 32768
# ulimits:
# nofile:
# soft: 1000000
# hard: 1000000
restart: on-failure
pump0:
image: pingcap/tidb-binlog:latest
volumes:
- ./config/pump.toml:/pump.toml:ro
- ./data:/data
- ./logs:/logs
command:
- /pump
- --addr=0.0.0.0:8250
- --advertise-addr=pump0:8250
- --data-dir=/data/pump0
- --log-file=/logs/pump0.log
- --node-id=pump0
- --pd-urls=http://pd0:2379,http://pd1:2379,http://pd2:2379
- --config=/pump.toml
depends_on:
- "pd0"
- "pd1"
- "pd2"
restart: on-failure
pump1:
image: pingcap/tidb-binlog:latest
volumes:
- ./config/pump.toml:/pump.toml:ro
- ./data:/data
- ./logs:/logs
command:
- /pump
- --addr=0.0.0.0:8250
- --advertise-addr=pump1:8250
- --data-dir=/data/pump1
- --log-file=/logs/pump1.log
- --node-id=pump1
- --pd-urls=http://pd0:2379,http://pd1:2379,http://pd2:2379
- --config=/pump.toml
depends_on:
- "pd0"
- "pd1"
- "pd2"
restart: on-failure
pump2:
image: pingcap/tidb-binlog:latest
volumes:
- ./config/pump.toml:/pump.toml:ro
- ./data:/data
- ./logs:/logs
command:
- /pump
- --addr=0.0.0.0:8250
- --advertise-addr=pump2:8250
- --data-dir=/data/pump2
- --log-file=/logs/pump2.log
- --node-id=pump2
- --pd-urls=http://pd0:2379,http://pd1:2379,http://pd2:2379
- --config=/pump.toml
depends_on:
- "pd0"
- "pd1"
- "pd2"
restart: on-failure
drainer:
image: pingcap/tidb-binlog:latest
volumes:
- ./config/drainer.toml:/drainer.toml:ro
- ./data:/data
- ./logs:/logs
command:
- /drainer
- --addr=0.0.0.0:8249
- --data-dir=/data/data.drainer
- --log-file=/logs/drainer.log
- --pd-urls=http://pd0:2379,http://pd1:2379,http://pd2:2379
- --config=/drainer.toml
- --initial-commit-ts=0
- --dest-db-type=kafka
depends_on:
- "pd0"
- "pd1"
- "pd2"
- "kafka0"
- "kafka1"
- "kafka2"
restart: on-failure
zoo0:
image: zookeeper:latest
ports:
- "2181:2181"
environment:
ZOO_MY_ID: 1
ZOO_PORT: 2181
ZOO_SERVERS: server.1=zoo0:2888:3888 server.2=zoo1:2888:3888 server.3=zoo2:2888:3888
volumes:
- ./data/zoo0/data:/data
- ./data/zoo0/datalog:/datalog
restart: on-failure
zoo1:
image: zookeeper:latest
ports:
- "2182:2182"
environment:
ZOO_MY_ID: 2
ZOO_PORT: 2182
ZOO_SERVERS: server.1=zoo0:2888:3888 server.2=zoo1:2888:3888 server.3=zoo2:2888:3888
volumes:
- ./data/zoo1/data:/data
- ./data/zoo1/datalog:/datalog
restart: on-failure
zoo2:
image: zookeeper:latest
ports:
- "2183:2183"
environment:
ZOO_MY_ID: 3
ZOO_PORT: 2183
ZOO_SERVERS: server.1=zoo0:2888:3888 server.2=zoo1:2888:3888 server.3=zoo2:2888:3888
volumes:
- ./data/zoo2/data:/data
- ./data/zoo2/datalog:/datalog
restart: on-failure
kafka0:
image: wurstmeister/kafka:2.12-2.1.1
ports:
- "9092:9092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_LOG_DIRS: /data/kafka-logs
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka0:9092
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
KAFKA_ZOOKEEPER_CONNECT: zoo0:2181,zoo1:2182,zoo2:2183
volumes:
- ./data/kafka-logs/kafka0:/data/kafka-logs
- ./logs/kafka0:/opt/kafka/logs
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- "zoo0"
- "zoo1"
- "zoo2"
restart: on-failure
kafka1:
image: wurstmeister/kafka:2.12-2.1.1
ports:
- "9093:9093"
environment:
KAFKA_BROKER_ID: 2
KAFKA_LOG_DIRS: /data/kafka-logs
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9093
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9093
KAFKA_ZOOKEEPER_CONNECT: zoo0:2181,zoo1:2182,zoo2:2183
volumes:
- ./data/kafka-logs/kafka1:/data/kafka-logs
- ./logs/kafka1:/opt/kafka/logs
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- "zoo0"
- "zoo1"
- "zoo2"
restart: on-failure
kafka2:
image: wurstmeister/kafka:2.12-2.1.1
ports:
- "9094:9094"
environment:
KAFKA_BROKER_ID: 3
KAFKA_LOG_DIRS: /data/kafka-logs
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9094
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9094
KAFKA_ZOOKEEPER_CONNECT: zoo0:2181,zoo1:2182,zoo2:2183
volumes:
- ./data/kafka-logs/kafka2:/data/kafka-logs
- ./logs/kafka2:/opt/kafka/logs
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- "zoo0"
- "zoo1"
- "zoo2"
restart: on-failure
tidb:
image: pingcap/tidb:latest
ports:
- "4000:4000"
- "10080:10080"
volumes:
- ./config/tidb.toml:/tidb.toml:ro
- ./logs:/logs
command:
- --store=tikv
- --path=pd0:2379,pd1:2379,pd2:2379
- --config=/tidb.toml
- --log-file=/logs/tidb.log
- --advertise-address=tidb
- --enable-binlog=true
depends_on:
- "tikv0"
- "tikv1"
- "tikv2"
- "pump0"
- "pump1"
- "pump2"
# sysctls:
# net.core.somaxconn: 32768
# ulimits:
# nofile:
# soft: 1000000
# hard: 1000000
restart: on-failure
tispark-master:
image: pingcap/tispark:latest
command:
- /opt/spark/sbin/start-master.sh
volumes:
- ./config/spark-defaults.conf:/opt/spark/conf/spark-defaults.conf:ro
environment:
SPARK_MASTER_PORT: 7077
SPARK_MASTER_WEBUI_PORT: 8080
ports:
- "7077:7077"
- "8080:8080"
depends_on:
- "tikv0"
- "tikv1"
- "tikv2"
restart: on-failure
tispark-slave0:
image: pingcap/tispark:latest
command:
- /opt/spark/sbin/start-slave.sh
- spark://tispark-master:7077
volumes:
- ./config/spark-defaults.conf:/opt/spark/conf/spark-defaults.conf:ro
environment:
SPARK_WORKER_WEBUI_PORT: 38081
ports:
- "38081:38081"
depends_on:
- tispark-master
restart: on-failure
tidb-vision:
image: pingcap/tidb-vision:latest
environment:
PD_ENDPOINT: pd0:2379
ports:
- "8010:8010"
restart: on-failure
pushgateway:
image: prom/pushgateway:v0.3.1
command:
- --log.level=error
restart: on-failure
prometheus:
user: root
image: prom/prometheus:v2.2.1
command:
- --log.level=error
- --storage.tsdb.path=/data/prometheus
- --config.file=/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
volumes:
- ./config/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./config/pd.rules.yml:/etc/prometheus/pd.rules.yml:ro
- ./config/tikv.rules.yml:/etc/prometheus/tikv.rules.yml:ro
- ./config/tidb.rules.yml:/etc/prometheus/tidb.rules.yml:ro
- ./data:/data
restart: on-failure
grafana:
image: grafana/grafana:5.3.0
user: "0"
environment:
GF_LOG_LEVEL: error
GF_PATHS_PROVISIONING: /etc/grafana/provisioning
GF_PATHS_CONFIG: /etc/grafana/grafana.ini
ports:
- "3000:3000"
volumes:
- ./config/grafana:/etc/grafana
- ./config/dashboards:/tmp/dashboards
- ./data/grafana:/var/lib/grafana
restart: on-failure

13
docker/tidb/docker-compose-test.yml
Unescape Escape View File

@ -1,13 +0,0 @@
version: '2.1'
services:
tispark-tests:
image: pingcap/tispark:latest
volumes:
- ./config/spark-defaults.conf:/opt/spark/conf/spark-defaults.conf:ro
- ./tispark-tests/tests:/opt/spark/tests:ro
networks:
default:
external:
name: tidb-docker-compose_default

66
docker/tidb/docker-compose-tiflash-nightly.yml
Unescape Escape View File

@ -1,66 +0,0 @@
version: '2.1'
services:
pd0:
image: pingcap/pd:nightly
ports:
- "2379"
volumes:
- ./config/pd-nightly-tiflash.toml:/pd.toml:ro
- ./data:/data
- ./logs:/logs
command:
- --name=pd0
- --client-urls=http://0.0.0.0:2379
- --peer-urls=http://0.0.0.0:2380
- --advertise-client-urls=http://pd0:2379
- --advertise-peer-urls=http://pd0:2380
- --initial-cluster=pd0=http://pd0:2380
- --data-dir=/data/pd
- --config=/pd.toml
- --log-file=/logs/pd.log
restart: on-failure
tikv:
image: pingcap/tikv:nightly
volumes:
- ./data:/data
- ./logs:/logs
command:
- --addr=0.0.0.0:20160
- --advertise-addr=tikv:20160
- --status-addr=tikv:20180
- --data-dir=/data/tikv
- --pd=pd0:2379
- --log-file=/logs/tikv.log
depends_on:
- "pd0"
restart: on-failure
tidb:
image: pingcap/tidb:nightly
ports:
- "4000:4000"
- "10080:10080"
volumes:
- ./logs:/logs
command:
- --status=10080
- --advertise-address=tidb
- --store=tikv
- --path=pd0:2379
- --log-file=/logs/tidb.log
depends_on:
- "tikv"
restart: on-failure
tiflash:
image: pingcap/tiflash:nightly
volumes:
- ./config/tiflash-nightly.toml:/tiflash.toml:ro
- ./config/tiflash-learner-nightly.toml:/tiflash-learner.toml:ro
- ./data:/data
- ./logs:/logs
command:
- --config=/tiflash.toml
depends_on:
- "tikv"
- "tidb"
restart: on-failure

210
docker/tidb/docker-compose.yml
Unescape Escape View File

@ -1,210 +0,0 @@
version: '2.1'
services:
pd0:
image: pingcap/pd:latest
ports:
- "2379"
volumes:
- ./config/pd.toml:/pd.toml:ro
- ./data:/data
- ./logs:/logs
command:
- --name=pd0
- --client-urls=http://0.0.0.0:2379
- --peer-urls=http://0.0.0.0:2380
- --advertise-client-urls=http://pd0:2379
- --advertise-peer-urls=http://pd0:2380
- --initial-cluster=pd0=http://pd0:2380,pd1=http://pd1:2380,pd2=http://pd2:2380
- --data-dir=/data/pd0
- --config=/pd.toml
- --log-file=/logs/pd0.log
restart: on-failure
pd1:
image: pingcap/pd:latest
ports:
- "2379"
volumes:
- ./config/pd.toml:/pd.toml:ro
- ./data:/data
- ./logs:/logs
command:
- --name=pd1
- --client-urls=http://0.0.0.0:2379
- --peer-urls=http://0.0.0.0:2380
- --advertise-client-urls=http://pd1:2379
- --advertise-peer-urls=http://pd1:2380
- --initial-cluster=pd0=http://pd0:2380,pd1=http://pd1:2380,pd2=http://pd2:2380
- --data-dir=/data/pd1
- --config=/pd.toml
- --log-file=/logs/pd1.log
restart: on-failure
pd2:
image: pingcap/pd:latest
ports:
- "2379"
volumes:
- ./config/pd.toml:/pd.toml:ro
- ./data:/data
- ./logs:/logs
command:
- --name=pd2
- --client-urls=http://0.0.0.0:2379
- --peer-urls=http://0.0.0.0:2380
- --advertise-client-urls=http://pd2:2379
- --advertise-peer-urls=http://pd2:2380
- --initial-cluster=pd0=http://pd0:2380,pd1=http://pd1:2380,pd2=http://pd2:2380
- --data-dir=/data/pd2
- --config=/pd.toml
- --log-file=/logs/pd2.log
restart: on-failure
tikv0:
image: pingcap/tikv:latest
volumes:
- ./config/tikv.toml:/tikv.toml:ro
- ./data:/data
- ./logs:/logs
command:
- --addr=0.0.0.0:20160
- --advertise-addr=tikv0:20160
- --data-dir=/data/tikv0
- --pd=pd0:2379,pd1:2379,pd2:2379
- --config=/tikv.toml
- --log-file=/logs/tikv0.log
depends_on:
- "pd0"
- "pd1"
- "pd2"
restart: on-failure
tikv1:
image: pingcap/tikv:latest
volumes:
- ./config/tikv.toml:/tikv.toml:ro
- ./data:/data
- ./logs:/logs
command:
- --addr=0.0.0.0:20160
- --advertise-addr=tikv1:20160
- --data-dir=/data/tikv1
- --pd=pd0:2379,pd1:2379,pd2:2379
- --config=/tikv.toml
- --log-file=/logs/tikv1.log
depends_on:
- "pd0"
- "pd1"
- "pd2"
restart: on-failure
tikv2:
image: pingcap/tikv:latest
volumes:
- ./config/tikv.toml:/tikv.toml:ro
- ./data:/data
- ./logs:/logs
command:
- --addr=0.0.0.0:20160
- --advertise-addr=tikv2:20160
- --data-dir=/data/tikv2
- --pd=pd0:2379,pd1:2379,pd2:2379
- --config=/tikv.toml
- --log-file=/logs/tikv2.log
depends_on:
- "pd0"
- "pd1"
- "pd2"
restart: on-failure
tidb:
image: pingcap/tidb:latest
ports:
- "4000:4000"
- "10080:10080"
volumes:
- ./config/tidb.toml:/tidb.toml:ro
- ./logs:/logs
command:
- --store=tikv
- --path=pd0:2379,pd1:2379,pd2:2379
- --config=/tidb.toml
- --log-file=/logs/tidb.log
- --advertise-address=tidb
depends_on:
- "tikv0"
- "tikv1"
- "tikv2"
restart: on-failure
tispark-master:
image: pingcap/tispark:latest
command:
- /opt/spark/sbin/start-master.sh
volumes:
- ./config/spark-defaults.conf:/opt/spark/conf/spark-defaults.conf:ro
environment:
SPARK_MASTER_PORT: 7077
SPARK_MASTER_WEBUI_PORT: 8080
ports:
- "7077:7077"
- "8080:8080"
depends_on:
- "tikv0"
- "tikv1"
- "tikv2"
restart: on-failure
tispark-slave0:
image: pingcap/tispark:latest
command:
- /opt/spark/sbin/start-slave.sh
- spark://tispark-master:7077
volumes:
- ./config/spark-defaults.conf:/opt/spark/conf/spark-defaults.conf:ro
environment:
SPARK_WORKER_WEBUI_PORT: 38081
ports:
- "38081:38081"
depends_on:
- tispark-master
restart: on-failure
tidb-vision:
image: pingcap/tidb-vision:latest
environment:
PD_ENDPOINT: pd0:2379
ports:
- "8010:8010"
restart: on-failure
# monitors
pushgateway:
image: prom/pushgateway:v0.3.1
command:
- --log.level=error
restart: on-failure
prometheus:
user: root
image: prom/prometheus:v2.2.1
command:
- --log.level=error
- --storage.tsdb.path=/data/prometheus
- --config.file=/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
volumes:
- ./config/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./config/pd.rules.yml:/etc/prometheus/pd.rules.yml:ro
- ./config/tikv.rules.yml:/etc/prometheus/tikv.rules.yml:ro
- ./config/tidb.rules.yml:/etc/prometheus/tidb.rules.yml:ro
- ./data:/data
restart: on-failure
grafana:
image: grafana/grafana:6.0.1
user: "0"
environment:
GF_LOG_LEVEL: error
GF_PATHS_PROVISIONING: /etc/grafana/provisioning
GF_PATHS_CONFIG: /etc/grafana/grafana.ini
volumes:
- ./config/grafana:/etc/grafana
- ./config/dashboards:/tmp/dashboards
- ./data/grafana:/var/lib/grafana
ports:
- "3000:3000"
restart: on-failure

171
docker/tidb/docker-swarm.yml
Unescape Escape View File

@ -1,171 +0,0 @@
version: '3.3'
networks:
default:
driver: overlay
attachable: true
services:
pd0:
image: pingcap/pd:latest
ports:
- "2379"
volumes:
- ./config/pd.toml:/pd.toml:ro
- ./data:/data
- ./logs:/logs
command:
- --name=pd0
- --client-urls=http://0.0.0.0:2379
- --peer-urls=http://0.0.0.0:2380
- --advertise-client-urls=http://pd0:2379
- --advertise-peer-urls=http://pd0:2380
- --initial-cluster=pd0=http://pd0:2380,pd1=http://pd1:2380,pd2=http://pd2:2380
- --data-dir=/data/pd0
- --config=/pd.toml
- --log-file=/logs/pd0.log
pd1:
image: pingcap/pd:latest
ports:
- "2379"
volumes:
- ./config/pd.toml:/pd.toml:ro
- ./data:/data
- ./logs:/logs
command:
- --name=pd1
- --client-urls=http://0.0.0.0:2379
- --peer-urls=http://0.0.0.0:2380
- --advertise-client-urls=http://pd1:2379
- --advertise-peer-urls=http://pd1:2380
- --initial-cluster=pd0=http://pd0:2380,pd1=http://pd1:2380,pd2=http://pd2:2380
- --data-dir=/data/pd1
- --config=/pd.toml
- --log-file=/logs/pd1.log
pd2:
image: pingcap/pd:latest
ports:
- "2379"
volumes:
- ./config/pd.toml:/pd.toml:ro
- ./data:/data
- ./logs:/logs
command:
- --name=pd2
- --client-urls=http://0.0.0.0:2379
- --peer-urls=http://0.0.0.0:2380
- --advertise-client-urls=http://pd2:2379
- --advertise-peer-urls=http://pd2:2380
- --initial-cluster=pd0=http://pd0:2380,pd1=http://pd1:2380,pd2=http://pd2:2380
- --data-dir=/data/pd2
- --config=/pd.toml
- --log-file=/logs/pd2.log
tikv:
image: pingcap/tikv:latest
ports:
- target: 20160
published: 20160
environment:
- TASK_SLOT={{.Task.Slot}}
volumes:
- ./config/tikv.toml:/tikv.toml:ro
- ./data:/data
- ./logs:/logs
entrypoint: [ "/bin/sh", "-c", "/tikv-server --advertise-addr=$$HOSTNAME:20160 --addr=0.0.0.0:20160 --data-dir=/data/tikv$$TASK_SLOT --pd=pd0:2379,pd1:2379,pd2:2379 --config=/tikv.toml --log-file=/logs/tikv$$TASK_SLOT.log --log-level=info" ]
depends_on:
- "pd0"
- "pd1"
- "pd2"
deploy:
replicas: 3
restart_policy:
condition: on-failure
tidb:
image: pingcap/tidb:latest
ports:
- target: 4000
published: 4000
- target: 10080
published: 10080
environment:
- TASK_SLOT={{.Task.Slot}}
volumes:
- ./config/tidb.toml:/tidb.toml:ro
- ./logs:/logs
entrypoint: [ "/bin/sh", "-c", "/tidb-server --advertise-address=$$HOSTNAME --store=tikv --path=pd0:2379,pd1:2379,pd2:2379 --config=/tidb.toml --log-file=/logs/tidb$$TASK_SLOT.log -L info" ]
depends_on:
- "tikv"
deploy:
replicas: 1
tispark-master:
image: pingcap/tispark:latest
command:
- /opt/spark/sbin/start-master.sh
volumes:
- ./config/spark-defaults.conf:/opt/spark/conf/spark-defaults.conf:ro
environment:
SPARK_MASTER_PORT: 7077
SPARK_MASTER_WEBUI_PORT: 8080
ports:
- "7077:7077"
- "8080:8080"
depends_on:
- "tikv"
deploy:
replicas: 1
tispark-slave:
image: pingcap/tispark:latest
command:
- /opt/spark/sbin/start-slave.sh
- spark://tispark-master:7077
volumes:
- ./config/spark-defaults.conf:/opt/spark/conf/spark-defaults.conf:ro
environment:
SPARK_WORKER_WEBUI_PORT: 38081
ports:
- "38081:38081"
depends_on:
- tispark-master
deploy:
replicas: 1
tidb-vision:
image: pingcap/tidb-vision:latest
environment:
PD_ENDPOINT: pd0:2379
ports:
- "8010:8010"
# monitors
pushgateway:
image: prom/pushgateway:v0.3.1
command:
- --log.level=error
prometheus:
user: root
image: prom/prometheus:v2.2.1
command:
- --log.level=error
- --storage.tsdb.path=/data/prometheus
- --config.file=/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
volumes:
- ./config/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./config/pd.rules.yml:/etc/prometheus/pd.rules.yml:ro
- ./config/tikv.rules.yml:/etc/prometheus/tikv.rules.yml:ro
- ./config/tidb.rules.yml:/etc/prometheus/tidb.rules.yml:ro
- ./data:/data
grafana:
image: grafana/grafana:6.0.1
environment:
GF_LOG_LEVEL: error
GF_PATHS_PROVISIONING: /etc/grafana/provisioning
GF_PATHS_CONFIG: /etc/grafana/grafana.ini
volumes:
- ./config/grafana:/etc/grafana
- ./config/dashboards:/var/lib/grafana/dashboards
ports:
- "3000:3000"

50
docker/tidb/docker/debug/Dockerfile
Unescape Escape View File

@ -1,50 +0,0 @@
FROM centos:7
RUN yum update -y && yum install -y \
curl \
file \
gdb \
git \
iotop \
linux-perf \
mysql \
net-tools \
perf \
perl \
procps-ng \
psmisc \
strace \
sysstat \
tree \
tcpdump \
unzip \
vim \
wget \
which \
&& yum clean all \
&& rm -rf /var/cache/yum/*
RUN wget -q http://download.pingcap.org/tidb-latest-linux-amd64.tar.gz \
&& tar xzf tidb-latest-linux-amd64.tar.gz \
&& mv tidb-latest-linux-amd64/bin/* /usr/local/bin/ \
&& rm -rf tidb-latest-linux-amd64.tar.gz tidb-latest-linux-amd64
RUN wget https://github.com/brendangregg/FlameGraph/archive/master.zip \
&& unzip master.zip \
&& mv FlameGraph-master /opt/FlameGraph \
&& rm master.zip
ADD run_flamegraph.sh /run_flamegraph.sh
# used for go pprof
ENV GOLANG_VERSION 1.10
ENV GOLANG_DOWNLOAD_URL https://golang.org/dl/go$GOLANG_VERSION.linux-amd64.tar.gz
ENV GOLANG_DOWNLOAD_SHA256 b5a64335f1490277b585832d1f6c7f8c6c11206cba5cd3f771dcb87b98ad1a33
RUN curl -fsSL "$GOLANG_DOWNLOAD_URL" -o golang.tar.gz \
&& echo "$GOLANG_DOWNLOAD_SHA256 golang.tar.gz" | sha256sum -c - \
&& tar -C /usr/local -xzf golang.tar.gz \
&& rm golang.tar.gz
ENV GOPATH /go
ENV GOROOT /usr/local/go
ENV PATH $GOPATH/bin:$GOROOT/bin:$PATH
ENTRYPOINT ["/bin/bash"]

9
docker/tidb/docker/debug/run_flamegraph.sh
Unescape Escape View File

@ -1,9 +0,0 @@
#!/bin/bash
set -e
perf record -F 99 -p $1 -g -- sleep 60
perf script > out.perf
/opt/FlameGraph/stackcollapse-perf.pl out.perf > out.folded
/opt/FlameGraph/flamegraph.pl out.folded > kernel.svg
curl --upload-file ./kernel.svg https://transfer.sh/kernel.svg

9
docker/tidb/pd/Dockerfile
Unescape Escape View File

@ -1,9 +0,0 @@
FROM alpine:3.5
ADD bin/pd-server /pd-server
WORKDIR /
EXPOSE 2379 2380
ENTRYPOINT ["/pd-server"]

11
docker/tidb/tidb-binlog/Dockerfile
Unescape Escape View File

@ -1,11 +0,0 @@
from alpine:3.5
ADD bin/pump /pump
ADD bin/drainer /drainer
RUN chmod +x /pump /drainer
WORKDIR /
EXPOSE 8249 8250

13
docker/tidb/tidb-vision/Dockerfile
Unescape Escape View File

@ -1,13 +0,0 @@
FROM node:8
ADD tidb-vision /home/node/tidb-vision
WORKDIR /home/node/tidb-vision
RUN npm install
ENV PD_ENDPOINT=localhost:9000
EXPOSE 8010
CMD ["npm", "start"]

11
docker/tidb/tidb/Dockerfile
Unescape Escape View File

@ -1,11 +0,0 @@
from alpine:3.5
ADD bin/tidb-server /tidb-server
RUN chmod +x /tidb-server
WORKDIR /
EXPOSE 4000 10080
ENTRYPOINT ["/tidb-server"]

11
docker/tidb/tikv/Dockerfile
Unescape Escape View File

@ -1,11 +0,0 @@
FROM pingcap/alpine-glibc
ADD bin/tikv-server /tikv-server
RUN chmod +x /tikv-server
WORKDIR /
EXPOSE 20160
ENTRYPOINT ["/tikv-server"]

40
docker/tidb/tispark/Dockerfile
Unescape Escape View File

@ -1,40 +0,0 @@
FROM anapsix/alpine-java:8
ENV SPARK_VERSION=2.4.3 \
HADOOP_VERSION=2.7 \
TISPARK_PYTHON_VERSION=2.0 \
SPARK_HOME=/opt/spark \
SPARK_NO_DAEMONIZE=true \
SPARK_MASTER_PORT=7077 \
SPARK_MASTER_HOST=0.0.0.0 \
SPARK_MASTER_WEBUI_PORT=8080
ADD tispark-tests /opt/tispark-tests
# base image only contains busybox version nohup and ps
# spark scripts needs nohup in coreutils and ps in procps
# and we can use mysql-client to test tidb connection
RUN apk --no-cache add \
coreutils \
mysql-client \
procps \
python \
py-pip \
R
RUN wget -q https://download.pingcap.org/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz \
&& tar zxf spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz -C /opt/ \
&& ln -s /opt/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION} ${SPARK_HOME} \
&& wget -q http://download.pingcap.org/tispark-assembly-latest-linux-amd64.tar.gz \
&& tar zxf ./tispark-assembly-latest-linux-amd64.tar.gz -C /opt/ \
&& cp /opt/assembly/target/tispark-assembly-*.jar ${SPARK_HOME}/jars \
&& wget -q http://download.pingcap.org/tispark-sample-data.tar.gz \
&& tar zxf tispark-sample-data.tar.gz -C ${SPARK_HOME}/data/ \
&& rm -rf /opt/assembly/ spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz tispark-latest-linux-amd64.tar.gz tispark-sample-data.tar.gz
ADD spark-${SPARK_VERSION}/session.py ${SPARK_HOME}/python/pyspark/sql/
ADD conf/log4j.properties /opt/spark/conf/log4j.properties
ENV PYTHONPATH=${SPARK_HOME}/python/lib/py4j-0.10.4-src.zip:${SPARK_HOME}/python:$PYTHONPATH
WORKDIR ${SPARK_HOME}

43
docker/tidb/tispark/conf/log4j.properties
Unescape Escape View File

@ -1,43 +0,0 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Set everything to be logged to the console
log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN
# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR
# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
# tispark disable "WARN ObjectStore:568 - Failed to get database"
log4j.logger.org.apache.hadoop.hive.metastore.ObjectStore=ERROR

811
docker/tidb/tispark/spark-2.3.3/session.py
Unescape Escape View File

@ -1,811 +0,0 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
from __future__ import print_function
import sys
import warnings
from functools import reduce
from threading import RLock
if sys.version >= '3':
basestring = unicode = str
xrange = range
else:
from itertools import izip as zip, imap as map
from pyspark import since
from pyspark.rdd import RDD, ignore_unicode_prefix
from pyspark.sql.conf import RuntimeConfig
from pyspark.sql.dataframe import DataFrame
from pyspark.sql.readwriter import DataFrameReader
from pyspark.sql.streaming import DataStreamReader
from pyspark.sql.types import Row, DataType, StringType, StructType, TimestampType, \
_make_type_verifier, _infer_schema, _has_nulltype, _merge_type, _create_converter, \
_parse_datatype_string
from pyspark.sql.utils import install_exception_handler
__all__ = ["SparkSession"]
def _monkey_patch_RDD(sparkSession):
def toDF(self, schema=None, sampleRatio=None):
"""
Converts current :class:`RDD` into a :class:`DataFrame`
This is a shorthand for ``spark.createDataFrame(rdd, schema, sampleRatio)``
:param schema: a :class:`pyspark.sql.types.StructType` or list of names of columns
:param samplingRatio: the sample ratio of rows used for inferring
:return: a DataFrame
>>> rdd.toDF().collect()
[Row(name=u'Alice', age=1)]
"""
return sparkSession.createDataFrame(self, schema, sampleRatio)
RDD.toDF = toDF
class SparkSession(object):
"""The entry point to programming Spark with the Dataset and DataFrame API.
A SparkSession can be used create :class:`DataFrame`, register :class:`DataFrame` as
tables, execute SQL over tables, cache tables, and read parquet files.
To create a SparkSession, use the following builder pattern:
>>> spark = SparkSession.builder \\
... .master("local") \\
... .appName("Word Count") \\
... .config("spark.some.config.option", "some-value") \\
... .getOrCreate()
.. autoattribute:: builder
:annotation:
"""
class Builder(object):
"""Builder for :class:`SparkSession`.
"""
_lock = RLock()
_options = {}
@since(2.0)
def config(self, key=None, value=None, conf=None):
"""Sets a config option. Options set using this method are automatically propagated to
both :class:`SparkConf` and :class:`SparkSession`'s own configuration.
For an existing SparkConf, use `conf` parameter.
>>> from pyspark.conf import SparkConf
>>> SparkSession.builder.config(conf=SparkConf())
<pyspark.sql.session...
For a (key, value) pair, you can omit parameter names.
>>> SparkSession.builder.config("spark.some.config.option", "some-value")
<pyspark.sql.session...
:param key: a key name string for configuration property
:param value: a value for configuration property
:param conf: an instance of :class:`SparkConf`
"""
with self._lock:
if conf is None:
self._options[key] = str(value)
else:
for (k, v) in conf.getAll():
self._options[k] = v
return self
@since(2.0)
def master(self, master):
"""Sets the Spark master URL to connect to, such as "local" to run locally, "local[4]"
to run locally with 4 cores, or "spark://master:7077" to run on a Spark standalone
cluster.
:param master: a url for spark master
"""
return self.config("spark.master", master)
@since(2.0)
def appName(self, name):
"""Sets a name for the application, which will be shown in the Spark web UI.
If no application name is set, a randomly generated name will be used.
:param name: an application name
"""
return self.config("spark.app.name", name)
@since(2.0)
def enableHiveSupport(self):
"""Enables Hive support, including connectivity to a persistent Hive metastore, support
for Hive serdes, and Hive user-defined functions.
"""
return self.config("spark.sql.catalogImplementation", "hive")
@since(2.0)
def getOrCreate(self):
"""Gets an existing :class:`SparkSession` or, if there is no existing one, creates a
new one based on the options set in this builder.
This method first checks whether there is a valid global default SparkSession, and if
yes, return that one. If no valid global default SparkSession exists, the method
creates a new SparkSession and assigns the newly created SparkSession as the global
default.
>>> s1 = SparkSession.builder.config("k1", "v1").getOrCreate()
>>> s1.conf.get("k1") == s1.sparkContext.getConf().get("k1") == "v1"
True
In case an existing SparkSession is returned, the config options specified
in this builder will be applied to the existing SparkSession.
>>> s2 = SparkSession.builder.config("k2", "v2").getOrCreate()
>>> s1.conf.get("k1") == s2.conf.get("k1")
True
>>> s1.conf.get("k2") == s2.conf.get("k2")
True
"""
with self._lock:
from pyspark.context import SparkContext
from pyspark.conf import SparkConf
session = SparkSession._instantiatedSession
if session is None or session._sc._jsc is None:
sparkConf = SparkConf()
for key, value in self._options.items():
sparkConf.set(key, value)
sc = SparkContext.getOrCreate(sparkConf)
# This SparkContext may be an existing one.
for key, value in self._options.items():
# we need to propagate the confs
# before we create the SparkSession. Otherwise, confs like
# warehouse path and metastore url will not be set correctly (
# these confs cannot be changed once the SparkSession is created).
sc._conf.set(key, value)
session = SparkSession(sc)
for key, value in self._options.items():
session._jsparkSession.sessionState().conf().setConfString(key, value)
for key, value in self._options.items():
session.sparkContext._conf.set(key, value)
return session
builder = Builder()
"""A class attribute having a :class:`Builder` to construct :class:`SparkSession` instances"""
_instantiatedSession = None
@ignore_unicode_prefix
def __init__(self, sparkContext, jsparkSession=None):
"""Creates a new SparkSession.
>>> from datetime import datetime
>>> spark = SparkSession(sc)
>>> allTypes = sc.parallelize([Row(i=1, s="string", d=1.0, l=1,
... b=True, list=[1, 2, 3], dict={"s": 0}, row=Row(a=1),
... time=datetime(2014, 8, 1, 14, 1, 5))])
>>> df = allTypes.toDF()
>>> df.createOrReplaceTempView("allTypes")
>>> spark.sql('select i+1, d+1, not b, list[1], dict["s"], time, row.a '
... 'from allTypes where b and i > 0').collect()
[Row((i + CAST(1 AS BIGINT))=2, (d + CAST(1 AS DOUBLE))=2.0, (NOT b)=False, list[1]=2, \
dict[s]=0, time=datetime.datetime(2014, 8, 1, 14, 1, 5), a=1)]
>>> df.rdd.map(lambda x: (x.i, x.s, x.d, x.l, x.b, x.time, x.row.a, x.list)).collect()
[(1, u'string', 1.0, 1, True, datetime.datetime(2014, 8, 1, 14, 1, 5), 1, [1, 2, 3])]
"""
from pyspark.sql.context import SQLContext
self._sc = sparkContext
self._jsc = self._sc._jsc
self._jvm = self._sc._jvm
if jsparkSession is None:
jsparkSession = self._jvm.SparkSession.builder().getOrCreate()
self._jsparkSession = jsparkSession
self._jwrapped = self._jsparkSession.sqlContext()
self._wrapped = SQLContext(self._sc, self, self._jwrapped)
_monkey_patch_RDD(self)
install_exception_handler()
# If we had an instantiated SparkSession attached with a SparkContext
# which is stopped now, we need to renew the instantiated SparkSession.
# Otherwise, we will use invalid SparkSession when we call Builder.getOrCreate.
if SparkSession._instantiatedSession is None \
or SparkSession._instantiatedSession._sc._jsc is None:
SparkSession._instantiatedSession = self
def _repr_html_(self):
return """
<div>
<p><b>SparkSession - {catalogImplementation}</b></p>
{sc_HTML}
</div>
""".format(
catalogImplementation=self.conf.get("spark.sql.catalogImplementation"),
sc_HTML=self.sparkContext._repr_html_()
)
@since(2.0)
def newSession(self):
"""
Returns a new SparkSession as new session, that has separate SQLConf,
registered temporary views and UDFs, but shared SparkContext and
table cache.
"""
return self.__class__(self._sc, self._jsparkSession.newSession())
@property
@since(2.0)
def sparkContext(self):
"""Returns the underlying :class:`SparkContext`."""
return self._sc
@property
@since(2.0)
def version(self):
"""The version of Spark on which this application is running."""
return self._jsparkSession.version()
@property
@since(2.0)
def conf(self):
"""Runtime configuration interface for Spark.
This is the interface through which the user can get and set all Spark and Hadoop
configurations that are relevant to Spark SQL. When getting the value of a config,
this defaults to the value set in the underlying :class:`SparkContext`, if any.
"""
if not hasattr(self, "_conf"):
self._conf = RuntimeConfig(self._jsparkSession.conf())
return self._conf
@property
@since(2.0)
def catalog(self):
"""Interface through which the user may create, drop, alter or query underlying
databases, tables, functions etc.
:return: :class:`Catalog`
"""
from pyspark.sql.catalog import Catalog
if not hasattr(self, "_catalog"):
self._catalog = Catalog(self)
return self._catalog
@property
@since(2.0)
def udf(self):
"""Returns a :class:`UDFRegistration` for UDF registration.
:return: :class:`UDFRegistration`
"""
from pyspark.sql.udf import UDFRegistration
return UDFRegistration(self)
@since(2.0)
def range(self, start, end=None, step=1, numPartitions=None):
"""
Create a :class:`DataFrame` with single :class:`pyspark.sql.types.LongType` column named
``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
step value ``step``.
:param start: the start value
:param end: the end value (exclusive)
:param step: the incremental step (default: 1)
:param numPartitions: the number of partitions of the DataFrame
:return: :class:`DataFrame`
>>> spark.range(1, 7, 2).collect()
[Row(id=1), Row(id=3), Row(id=5)]
If only one argument is specified, it will be used as the end value.
>>> spark.range(3).collect()
[Row(id=0), Row(id=1), Row(id=2)]
"""
if numPartitions is None:
numPartitions = self._sc.defaultParallelism
if end is None:
jdf = self._jsparkSession.range(0, int(start), int(step), int(numPartitions))
else:
jdf = self._jsparkSession.range(int(start), int(end), int(step), int(numPartitions))
return DataFrame(jdf, self._wrapped)
def _inferSchemaFromList(self, data, names=None):
"""
Infer schema from list of Row or tuple.
:param data: list of Row or tuple
:param names: list of column names
:return: :class:`pyspark.sql.types.StructType`
"""
if not data:
raise ValueError("can not infer schema from empty dataset")
first = data[0]
if type(first) is dict:
warnings.warn("inferring schema from dict is deprecated,"
"please use pyspark.sql.Row instead")
schema = reduce(_merge_type, (_infer_schema(row, names) for row in data))
if _has_nulltype(schema):
raise ValueError("Some of types cannot be determined after inferring")
return schema
def _inferSchema(self, rdd, samplingRatio=None, names=None):
"""
Infer schema from an RDD of Row or tuple.
:param rdd: an RDD of Row or tuple
:param samplingRatio: sampling ratio, or no sampling (default)
:return: :class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
if not first:
raise ValueError("The first row in RDD is empty, "
"can not infer schema")
if type(first) is dict:
warnings.warn("Using RDD of dict to inferSchema is deprecated. "
"Use pyspark.sql.Row instead")
if samplingRatio is None:
schema = _infer_schema(first, names=names)
if _has_nulltype(schema):
for row in rdd.take(100)[1:]:
schema = _merge_type(schema, _infer_schema(row, names=names))
if not _has_nulltype(schema):
break
else:
raise ValueError("Some of types cannot be determined by the "
"first 100 rows, please try again with sampling")
else:
if samplingRatio < 0.99:
rdd = rdd.sample(False, float(samplingRatio))
schema = rdd.map(lambda row: _infer_schema(row, names)).reduce(_merge_type)
return schema
def _createFromRDD(self, rdd, schema, samplingRatio):
"""
Create an RDD for DataFrame from an existing RDD, returns the RDD and schema.
"""
if schema is None or isinstance(schema, (list, tuple)):
struct = self._inferSchema(rdd, samplingRatio, names=schema)
converter = _create_converter(struct)
rdd = rdd.map(converter)
if isinstance(schema, (list, tuple)):
for i, name in enumerate(schema):
struct.fields[i].name = name
struct.names[i] = name
schema = struct
elif not isinstance(schema, StructType):
raise TypeError("schema should be StructType or list or None, but got: %s" % schema)
# convert python objects to sql data
rdd = rdd.map(schema.toInternal)
return rdd, schema
def _createFromLocal(self, data, schema):
"""
Create an RDD for DataFrame from a list or pandas.DataFrame, returns
the RDD and schema.
"""
# make sure data could consumed multiple times
if not isinstance(data, list):
data = list(data)
if schema is None or isinstance(schema, (list, tuple)):
struct = self._inferSchemaFromList(data, names=schema)
converter = _create_converter(struct)
data = map(converter, data)
if isinstance(schema, (list, tuple)):
for i, name in enumerate(schema):
struct.fields[i].name = name
struct.names[i] = name
schema = struct
elif not isinstance(schema, StructType):
raise TypeError("schema should be StructType or list or None, but got: %s" % schema)
# convert python objects to sql data
data = [schema.toInternal(row) for row in data]
return self._sc.parallelize(data), schema
def _get_numpy_record_dtype(self, rec):
"""
Used when converting a pandas.DataFrame to Spark using to_records(), this will correct
the dtypes of fields in a record so they can be properly loaded into Spark.
:param rec: a numpy record to check field dtypes
:return corrected dtype for a numpy.record or None if no correction needed
"""
import numpy as np
cur_dtypes = rec.dtype
col_names = cur_dtypes.names
record_type_list = []
has_rec_fix = False
for i in xrange(len(cur_dtypes)):
curr_type = cur_dtypes[i]
# If type is a datetime64 timestamp, convert to microseconds
# NOTE: if dtype is datetime[ns] then np.record.tolist() will output values as longs,
# conversion from [us] or lower will lead to py datetime objects, see SPARK-22417
if curr_type == np.dtype('datetime64[ns]'):
curr_type = 'datetime64[us]'
has_rec_fix = True
record_type_list.append((str(col_names[i]), curr_type))
return np.dtype(record_type_list) if has_rec_fix else None
def _convert_from_pandas(self, pdf, schema, timezone):
"""
Convert a pandas.DataFrame to list of records that can be used to make a DataFrame
:return list of records
"""
if timezone is not None:
from pyspark.sql.types import _check_series_convert_timestamps_tz_local
copied = False
if isinstance(schema, StructType):
for field in schema:
# TODO: handle nested timestamps, such as ArrayType(TimestampType())?
if isinstance(field.dataType, TimestampType):
s = _check_series_convert_timestamps_tz_local(pdf[field.name], timezone)
if s is not pdf[field.name]:
if not copied:
# Copy once if the series is modified to prevent the original
# Pandas DataFrame from being updated
pdf = pdf.copy()
copied = True
pdf[field.name] = s
else:
for column, series in pdf.iteritems():
s = _check_series_convert_timestamps_tz_local(series, timezone)
if s is not series:
if not copied:
# Copy once if the series is modified to prevent the original
# Pandas DataFrame from being updated
pdf = pdf.copy()
copied = True
pdf[column] = s
# Convert pandas.DataFrame to list of numpy records
np_records = pdf.to_records(index=False)
# Check if any columns need to be fixed for Spark to infer properly
if len(np_records) > 0:
record_dtype = self._get_numpy_record_dtype(np_records[0])
if record_dtype is not None:
return [r.astype(record_dtype).tolist() for r in np_records]
# Convert list of numpy records to python lists
return [r.tolist() for r in np_records]
def _create_from_pandas_with_arrow(self, pdf, schema, timezone):
"""
Create a DataFrame from a given pandas.DataFrame by slicing it into partitions, converting
to Arrow data, then sending to the JVM to parallelize. If a schema is passed in, the
data types will be used to coerce the data in Pandas to Arrow conversion.
"""
from pyspark.serializers import ArrowSerializer, _create_batch
from pyspark.sql.types import from_arrow_schema, to_arrow_type, TimestampType
from pyspark.sql.utils import require_minimum_pandas_version, \
require_minimum_pyarrow_version
require_minimum_pandas_version()
require_minimum_pyarrow_version()
from pandas.api.types import is_datetime64_dtype, is_datetime64tz_dtype
# Determine arrow types to coerce data when creating batches
if isinstance(schema, StructType):
arrow_types = [to_arrow_type(f.dataType) for f in schema.fields]
elif isinstance(schema, DataType):
raise ValueError("Single data type %s is not supported with Arrow" % str(schema))
else:
# Any timestamps must be coerced to be compatible with Spark
arrow_types = [to_arrow_type(TimestampType())
if is_datetime64_dtype(t) or is_datetime64tz_dtype(t) else None
for t in pdf.dtypes]
# Slice the DataFrame to be batched
step = -(-len(pdf) // self.sparkContext.defaultParallelism) # round int up
pdf_slices = (pdf[start:start + step] for start in xrange(0, len(pdf), step))
# Create Arrow record batches
batches = [_create_batch([(c, t) for (_, c), t in zip(pdf_slice.iteritems(), arrow_types)],
timezone)
for pdf_slice in pdf_slices]
# Create the Spark schema from the first Arrow batch (always at least 1 batch after slicing)
if isinstance(schema, (list, tuple)):
struct = from_arrow_schema(batches[0].schema)
for i, name in enumerate(schema):
struct.fields[i].name = name
struct.names[i] = name
schema = struct
# Create the Spark DataFrame directly from the Arrow data and schema
jrdd = self._sc._serialize_to_jvm(batches, len(batches), ArrowSerializer())
jdf = self._jvm.PythonSQLUtils.arrowPayloadToDataFrame(
jrdd, schema.json(), self._wrapped._jsqlContext)
df = DataFrame(jdf, self._wrapped)
df._schema = schema
return df
@since(2.0)
@ignore_unicode_prefix
def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=True):
"""
Creates a :class:`DataFrame` from an :class:`RDD`, a list or a :class:`pandas.DataFrame`.
When ``schema`` is a list of column names, the type of each column
will be inferred from ``data``.
When ``schema`` is ``None``, it will try to infer the schema (column names and types)
from ``data``, which should be an RDD of :class:`Row`,
or :class:`namedtuple`, or :class:`dict`.
When ``schema`` is :class:`pyspark.sql.types.DataType` or a datatype string, it must match
the real data, or an exception will be thrown at runtime. If the given schema is not
:class:`pyspark.sql.types.StructType`, it will be wrapped into a
:class:`pyspark.sql.types.StructType` as its only field, and the field name will be "value",
each record will also be wrapped into a tuple, which can be converted to row later.
If schema inference is needed, ``samplingRatio`` is used to determined the ratio of
rows used for schema inference. The first row will be used if ``samplingRatio`` is ``None``.
:param data: an RDD of any kind of SQL data representation(e.g. row, tuple, int, boolean,
etc.), or :class:`list`, or :class:`pandas.DataFrame`.
:param schema: a :class:`pyspark.sql.types.DataType` or a datatype string or a list of
column names, default is ``None``. The data type string format equals to
:class:`pyspark.sql.types.DataType.simpleString`, except that top level struct type can
omit the ``struct<>`` and atomic types use ``typeName()`` as their format, e.g. use
``byte`` instead of ``tinyint`` for :class:`pyspark.sql.types.ByteType`. We can also use
``int`` as a short name for ``IntegerType``.
:param samplingRatio: the sample ratio of rows used for inferring
:param verifySchema: verify data types of every row against schema.
:return: :class:`DataFrame`
.. versionchanged:: 2.1
Added verifySchema.
>>> l = [('Alice', 1)]
>>> spark.createDataFrame(l).collect()
[Row(_1=u'Alice', _2=1)]
>>> spark.createDataFrame(l, ['name', 'age']).collect()
[Row(name=u'Alice', age=1)]
>>> d = [{'name': 'Alice', 'age': 1}]
>>> spark.createDataFrame(d).collect()
[Row(age=1, name=u'Alice')]
>>> rdd = sc.parallelize(l)
>>> spark.createDataFrame(rdd).collect()
[Row(_1=u'Alice', _2=1)]
>>> df = spark.createDataFrame(rdd, ['name', 'age'])
>>> df.collect()
[Row(name=u'Alice', age=1)]
>>> from pyspark.sql import Row
>>> Person = Row('name', 'age')
>>> person = rdd.map(lambda r: Person(*r))
>>> df2 = spark.createDataFrame(person)
>>> df2.collect()
[Row(name=u'Alice', age=1)]
>>> from pyspark.sql.types import *
>>> schema = StructType([
... StructField("name", StringType(), True),
... StructField("age", IntegerType(), True)])
>>> df3 = spark.createDataFrame(rdd, schema)
>>> df3.collect()
[Row(name=u'Alice', age=1)]
>>> spark.createDataFrame(df.toPandas()).collect() # doctest: +SKIP
[Row(name=u'Alice', age=1)]
>>> spark.createDataFrame(pandas.DataFrame([[1, 2]])).collect() # doctest: +SKIP
[Row(0=1, 1=2)]
>>> spark.createDataFrame(rdd, "a: string, b: int").collect()
[Row(a=u'Alice', b=1)]
>>> rdd = rdd.map(lambda row: row[1])
>>> spark.createDataFrame(rdd, "int").collect()
[Row(value=1)]
>>> spark.createDataFrame(rdd, "boolean").collect() # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
Py4JJavaError: ...
"""
if isinstance(data, DataFrame):
raise TypeError("data is already a DataFrame")
if isinstance(schema, basestring):
schema = _parse_datatype_string(schema)
elif isinstance(schema, (list, tuple)):
# Must re-encode any unicode strings to be consistent with StructField names
schema = [x.encode('utf-8') if not isinstance(x, str) else x for x in schema]
try:
import pandas
has_pandas = True
except Exception:
has_pandas = False
if has_pandas and isinstance(data, pandas.DataFrame):
from pyspark.sql.utils import require_minimum_pandas_version
require_minimum_pandas_version()
if self.conf.get("spark.sql.execution.pandas.respectSessionTimeZone").lower() \
== "true":
timezone = self.conf.get("spark.sql.session.timeZone")
else:
timezone = None
# If no schema supplied by user then get the names of columns only
if schema is None:
schema = [str(x) if not isinstance(x, basestring) else
(x.encode('utf-8') if not isinstance(x, str) else x)
for x in data.columns]
if self.conf.get("spark.sql.execution.arrow.enabled", "false").lower() == "true" \
and len(data) > 0:
try:
return self._create_from_pandas_with_arrow(data, schema, timezone)
except Exception as e:
warnings.warn("Arrow will not be used in createDataFrame: %s" % str(e))
# Fallback to create DataFrame without arrow if raise some exception
data = self._convert_from_pandas(data, schema, timezone)
if isinstance(schema, StructType):
verify_func = _make_type_verifier(schema) if verifySchema else lambda _: True
def prepare(obj):
verify_func(obj)
return obj
elif isinstance(schema, DataType):
dataType = schema
schema = StructType().add("value", schema)
verify_func = _make_type_verifier(
dataType, name="field value") if verifySchema else lambda _: True
def prepare(obj):
verify_func(obj)
return obj,
else:
prepare = lambda obj: obj
if isinstance(data, RDD):
rdd, schema = self._createFromRDD(data.map(prepare), schema, samplingRatio)
else:
rdd, schema = self._createFromLocal(map(prepare, data), schema)
jrdd = self._jvm.SerDeUtil.toJavaArray(rdd._to_java_object_rdd())
jdf = self._jsparkSession.applySchemaToPythonRDD(jrdd.rdd(), schema.json())
df = DataFrame(jdf, self._wrapped)
df._schema = schema
return df
@ignore_unicode_prefix
@since(2.0)
def sql(self, sqlQuery):
"""Returns a :class:`DataFrame` representing the result of the given query.
:return: :class:`DataFrame`
>>> df.createOrReplaceTempView("table1")
>>> df2 = spark.sql("SELECT field1 AS f1, field2 as f2 from table1")
>>> df2.collect()
[Row(f1=1, f2=u'row1'), Row(f1=2, f2=u'row2'), Row(f1=3, f2=u'row3')]
"""
return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
@since(2.0)
def table(self, tableName):
"""Returns the specified table as a :class:`DataFrame`.
:return: :class:`DataFrame`
>>> df.createOrReplaceTempView("table1")
>>> df2 = spark.table("table1")
>>> sorted(df.collect()) == sorted(df2.collect())
True
"""
return DataFrame(self._jsparkSession.table(tableName), self._wrapped)
@property
@since(2.0)
def read(self):
"""
Returns a :class:`DataFrameReader` that can be used to read data
in as a :class:`DataFrame`.
:return: :class:`DataFrameReader`
"""
return DataFrameReader(self._wrapped)
@property
@since(2.0)
def readStream(self):
"""
Returns a :class:`DataStreamReader` that can be used to read data streams
as a streaming :class:`DataFrame`.
.. note:: Evolving.
:return: :class:`DataStreamReader`
"""
return DataStreamReader(self._wrapped)
@property
@since(2.0)
def streams(self):
"""Returns a :class:`StreamingQueryManager` that allows managing all the
:class:`StreamingQuery` StreamingQueries active on `this` context.
.. note:: Evolving.
:return: :class:`StreamingQueryManager`
"""
from pyspark.sql.streaming import StreamingQueryManager
return StreamingQueryManager(self._jsparkSession.streams())
@since(2.0)
def stop(self):
"""Stop the underlying :class:`SparkContext`.
"""
self._sc.stop()
SparkSession._instantiatedSession = None
@since(2.0)
def __enter__(self):
"""
Enable 'with SparkSession.builder.(...).getOrCreate() as session: app' syntax.
"""
return self
@since(2.0)
def __exit__(self, exc_type, exc_val, exc_tb):
"""
Enable 'with SparkSession.builder.(...).getOrCreate() as session: app' syntax.
Specifically stop the SparkSession on exit of the with block.
"""
self.stop()
def _test():
import os
import doctest
from pyspark.context import SparkContext
from pyspark.sql import Row
import pyspark.sql.session
os.chdir(os.environ["SPARK_HOME"])
globs = pyspark.sql.session.__dict__.copy()
sc = SparkContext('local[4]', 'PythonTest')
globs['sc'] = sc
globs['spark'] = SparkSession(sc)
globs['rdd'] = rdd = sc.parallelize(
[Row(field1=1, field2="row1"),
Row(field1=2, field2="row2"),
Row(field1=3, field2="row3")])
globs['df'] = rdd.toDF()
(failure_count, test_count) = doctest.testmod(
pyspark.sql.session, globs=globs,
optionflags=doctest.ELLIPSIS | doctest.NORMALIZE_WHITESPACE)
globs['sc'].stop()
if failure_count:
exit(-1)
if __name__ == "__main__":
_test()

872
docker/tidb/tispark/spark-2.4.3/session.py
Unescape Escape View File

@ -1,872 +0,0 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
from __future__ import print_function
import sys
import warnings
from functools import reduce
from threading import RLock
if sys.version >= '3':
basestring = unicode = str
xrange = range
else:
from itertools import izip as zip, imap as map
from pyspark import since
from pyspark.rdd import RDD, ignore_unicode_prefix
from pyspark.sql.conf import RuntimeConfig
from pyspark.sql.dataframe import DataFrame
from pyspark.sql.readwriter import DataFrameReader
from pyspark.sql.streaming import DataStreamReader
from pyspark.sql.types import Row, DataType, StringType, StructType, TimestampType, \
_make_type_verifier, _infer_schema, _has_nulltype, _merge_type, _create_converter, \
_parse_datatype_string
from pyspark.sql.utils import install_exception_handler
__all__ = ["SparkSession"]
def _monkey_patch_RDD(sparkSession):
def toDF(self, schema=None, sampleRatio=None):
"""
Converts current :class:`RDD` into a :class:`DataFrame`
This is a shorthand for ``spark.createDataFrame(rdd, schema, sampleRatio)``
:param schema: a :class:`pyspark.sql.types.StructType` or list of names of columns
:param samplingRatio: the sample ratio of rows used for inferring
:return: a DataFrame
>>> rdd.toDF().collect()
[Row(name=u'Alice', age=1)]
"""
return sparkSession.createDataFrame(self, schema, sampleRatio)
RDD.toDF = toDF
class SparkSession(object):
"""The entry point to programming Spark with the Dataset and DataFrame API.
A SparkSession can be used create :class:`DataFrame`, register :class:`DataFrame` as
tables, execute SQL over tables, cache tables, and read parquet files.
To create a SparkSession, use the following builder pattern:
>>> spark = SparkSession.builder \\
... .master("local") \\
... .appName("Word Count") \\
... .config("spark.some.config.option", "some-value") \\
... .getOrCreate()
.. autoattribute:: builder
:annotation:
"""
class Builder(object):
"""Builder for :class:`SparkSession`.
"""
_lock = RLock()
_options = {}
@since(2.0)
def config(self, key=None, value=None, conf=None):
"""Sets a config option. Options set using this method are automatically propagated to
both :class:`SparkConf` and :class:`SparkSession`'s own configuration.
For an existing SparkConf, use `conf` parameter.
>>> from pyspark.conf import SparkConf
>>> SparkSession.builder.config(conf=SparkConf())
<pyspark.sql.session...
For a (key, value) pair, you can omit parameter names.
>>> SparkSession.builder.config("spark.some.config.option", "some-value")
<pyspark.sql.session...
:param key: a key name string for configuration property
:param value: a value for configuration property
:param conf: an instance of :class:`SparkConf`
"""
with self._lock:
if conf is None:
self._options[key] = str(value)
else:
for (k, v) in conf.getAll():
self._options[k] = v
return self
@since(2.0)
def master(self, master):
"""Sets the Spark master URL to connect to, such as "local" to run locally, "local[4]"
to run locally with 4 cores, or "spark://master:7077" to run on a Spark standalone
cluster.
:param master: a url for spark master
"""
return self.config("spark.master", master)
@since(2.0)
def appName(self, name):
"""Sets a name for the application, which will be shown in the Spark web UI.
If no application name is set, a randomly generated name will be used.
:param name: an application name
"""
return self.config("spark.app.name", name)
@since(2.0)
def enableHiveSupport(self):
"""Enables Hive support, including connectivity to a persistent Hive metastore, support
for Hive serdes, and Hive user-defined functions.
"""
return self.config("spark.sql.catalogImplementation", "hive")
@since(2.0)
def getOrCreate(self):
"""Gets an existing :class:`SparkSession` or, if there is no existing one, creates a
new one based on the options set in this builder.
This method first checks whether there is a valid global default SparkSession, and if
yes, return that one. If no valid global default SparkSession exists, the method
creates a new SparkSession and assigns the newly created SparkSession as the global
default.
>>> s1 = SparkSession.builder.config("k1", "v1").getOrCreate()
>>> s1.conf.get("k1") == s1.sparkContext.getConf().get("k1") == "v1"
True
In case an existing SparkSession is returned, the config options specified
in this builder will be applied to the existing SparkSession.
>>> s2 = SparkSession.builder.config("k2", "v2").getOrCreate()
>>> s1.conf.get("k1") == s2.conf.get("k1")
True
>>> s1.conf.get("k2") == s2.conf.get("k2")
True
"""
with self._lock:
from pyspark.context import SparkContext
from pyspark.conf import SparkConf
session = SparkSession._instantiatedSession
if session is None or session._sc._jsc is None:
sparkConf = SparkConf()
for key, value in self._options.items():
sparkConf.set(key, value)
sc = SparkContext.getOrCreate(sparkConf)
# This SparkContext may be an existing one.
for key, value in self._options.items():
# we need to propagate the confs
# before we create the SparkSession. Otherwise, confs like
# warehouse path and metastore url will not be set correctly (
# these confs cannot be changed once the SparkSession is created).
sc._conf.set(key, value)
session = SparkSession(sc)
for key, value in self._options.items():
session._jsparkSession.sessionState().conf().setConfString(key, value)
for key, value in self._options.items():
session.sparkContext._conf.set(key, value)
return session
builder = Builder()
"""A class attribute having a :class:`Builder` to construct :class:`SparkSession` instances"""
_instantiatedSession = None
@ignore_unicode_prefix
def __init__(self, sparkContext, jsparkSession=None):
"""Creates a new SparkSession.
>>> from datetime import datetime
>>> spark = SparkSession(sc)
>>> allTypes = sc.parallelize([Row(i=1, s="string", d=1.0, l=1,
... b=True, list=[1, 2, 3], dict={"s": 0}, row=Row(a=1),
... time=datetime(2014, 8, 1, 14, 1, 5))])
>>> df = allTypes.toDF()
>>> df.createOrReplaceTempView("allTypes")
>>> spark.sql('select i+1, d+1, not b, list[1], dict["s"], time, row.a '
... 'from allTypes where b and i > 0').collect()
[Row((i + CAST(1 AS BIGINT))=2, (d + CAST(1 AS DOUBLE))=2.0, (NOT b)=False, list[1]=2, \
dict[s]=0, time=datetime.datetime(2014, 8, 1, 14, 1, 5), a=1)]
>>> df.rdd.map(lambda x: (x.i, x.s, x.d, x.l, x.b, x.time, x.row.a, x.list)).collect()
[(1, u'string', 1.0, 1, True, datetime.datetime(2014, 8, 1, 14, 1, 5), 1, [1, 2, 3])]
"""
from pyspark.sql.context import SQLContext
self._sc = sparkContext
self._jsc = self._sc._jsc
self._jvm = self._sc._jvm
if jsparkSession is None:
if self._jvm.SparkSession.getDefaultSession().isDefined() \
and not self._jvm.SparkSession.getDefaultSession().get() \
.sparkContext().isStopped():
jsparkSession = self._jvm.SparkSession.getDefaultSession().get()
else:
jsparkSession = self._jvm.SparkSession.builder().getOrCreate()
# jsparkSession = self._jvm.SparkSession(self._jsc.sc())
self._jsparkSession = jsparkSession
self._jwrapped = self._jsparkSession.sqlContext()
self._wrapped = SQLContext(self._sc, self, self._jwrapped)
_monkey_patch_RDD(self)
install_exception_handler()
# If we had an instantiated SparkSession attached with a SparkContext
# which is stopped now, we need to renew the instantiated SparkSession.
# Otherwise, we will use invalid SparkSession when we call Builder.getOrCreate.
if SparkSession._instantiatedSession is None \
or SparkSession._instantiatedSession._sc._jsc is None:
SparkSession._instantiatedSession = self
self._jvm.SparkSession.setDefaultSession(self._jsparkSession)
def _repr_html_(self):
return """
<div>
<p><b>SparkSession - {catalogImplementation}</b></p>
{sc_HTML}
</div>
""".format(
catalogImplementation=self.conf.get("spark.sql.catalogImplementation"),
sc_HTML=self.sparkContext._repr_html_()
)
@since(2.0)
def newSession(self):
"""
Returns a new SparkSession as new session, that has separate SQLConf,
registered temporary views and UDFs, but shared SparkContext and
table cache.
"""
return self.__class__(self._sc, self._jsparkSession.newSession())
@property
@since(2.0)
def sparkContext(self):
"""Returns the underlying :class:`SparkContext`."""
return self._sc
@property
@since(2.0)
def version(self):
"""The version of Spark on which this application is running."""
return self._jsparkSession.version()
@property
@since(2.0)
def conf(self):
"""Runtime configuration interface for Spark.
This is the interface through which the user can get and set all Spark and Hadoop
configurations that are relevant to Spark SQL. When getting the value of a config,
this defaults to the value set in the underlying :class:`SparkContext`, if any.
"""
if not hasattr(self, "_conf"):
self._conf = RuntimeConfig(self._jsparkSession.conf())
return self._conf
@property
@since(2.0)
def catalog(self):
"""Interface through which the user may create, drop, alter or query underlying
databases, tables, functions etc.
:return: :class:`Catalog`
"""
from pyspark.sql.catalog import Catalog
if not hasattr(self, "_catalog"):
self._catalog = Catalog(self)
return self._catalog
@property
@since(2.0)
def udf(self):
"""Returns a :class:`UDFRegistration` for UDF registration.
:return: :class:`UDFRegistration`
"""
from pyspark.sql.udf import UDFRegistration
return UDFRegistration(self)
@since(2.0)
def range(self, start, end=None, step=1, numPartitions=None):
"""
Create a :class:`DataFrame` with single :class:`pyspark.sql.types.LongType` column named
``id``, containing elements in a range from ``start`` to ``end`` (exclusive) with
step value ``step``.
:param start: the start value
:param end: the end value (exclusive)
:param step: the incremental step (default: 1)
:param numPartitions: the number of partitions of the DataFrame
:return: :class:`DataFrame`
>>> spark.range(1, 7, 2).collect()
[Row(id=1), Row(id=3), Row(id=5)]
If only one argument is specified, it will be used as the end value.
>>> spark.range(3).collect()
[Row(id=0), Row(id=1), Row(id=2)]
"""
if numPartitions is None:
numPartitions = self._sc.defaultParallelism
if end is None:
jdf = self._jsparkSession.range(0, int(start), int(step), int(numPartitions))
else:
jdf = self._jsparkSession.range(int(start), int(end), int(step), int(numPartitions))
return DataFrame(jdf, self._wrapped)
def _inferSchemaFromList(self, data, names=None):
"""
Infer schema from list of Row or tuple.
:param data: list of Row or tuple
:param names: list of column names
:return: :class:`pyspark.sql.types.StructType`
"""
if not data:
raise ValueError("can not infer schema from empty dataset")
first = data[0]
if type(first) is dict:
warnings.warn("inferring schema from dict is deprecated,"
"please use pyspark.sql.Row instead")
schema = reduce(_merge_type, (_infer_schema(row, names) for row in data))
if _has_nulltype(schema):
raise ValueError("Some of types cannot be determined after inferring")
return schema
def _inferSchema(self, rdd, samplingRatio=None, names=None):
"""
Infer schema from an RDD of Row or tuple.
:param rdd: an RDD of Row or tuple
:param samplingRatio: sampling ratio, or no sampling (default)
:return: :class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
if not first:
raise ValueError("The first row in RDD is empty, "
"can not infer schema")
if type(first) is dict:
warnings.warn("Using RDD of dict to inferSchema is deprecated. "
"Use pyspark.sql.Row instead")
if samplingRatio is None:
schema = _infer_schema(first, names=names)
if _has_nulltype(schema):
for row in rdd.take(100)[1:]:
schema = _merge_type(schema, _infer_schema(row, names=names))
if not _has_nulltype(schema):
break
else:
raise ValueError("Some of types cannot be determined by the "
"first 100 rows, please try again with sampling")
else:
if samplingRatio < 0.99:
rdd = rdd.sample(False, float(samplingRatio))
schema = rdd.map(lambda row: _infer_schema(row, names)).reduce(_merge_type)
return schema
def _createFromRDD(self, rdd, schema, samplingRatio):
"""
Create an RDD for DataFrame from an existing RDD, returns the RDD and schema.
"""
if schema is None or isinstance(schema, (list, tuple)):
struct = self._inferSchema(rdd, samplingRatio, names=schema)
converter = _create_converter(struct)
rdd = rdd.map(converter)
if isinstance(schema, (list, tuple)):
for i, name in enumerate(schema):
struct.fields[i].name = name
struct.names[i] = name
schema = struct
elif not isinstance(schema, StructType):
raise TypeError("schema should be StructType or list or None, but got: %s" % schema)
# convert python objects to sql data
rdd = rdd.map(schema.toInternal)
return rdd, schema
def _createFromLocal(self, data, schema):
"""
Create an RDD for DataFrame from a list or pandas.DataFrame, returns
the RDD and schema.
"""
# make sure data could consumed multiple times
if not isinstance(data, list):
data = list(data)
if schema is None or isinstance(schema, (list, tuple)):
struct = self._inferSchemaFromList(data, names=schema)
converter = _create_converter(struct)
data = map(converter, data)
if isinstance(schema, (list, tuple)):
for i, name in enumerate(schema):
struct.fields[i].name = name
struct.names[i] = name
schema = struct
elif not isinstance(schema, StructType):
raise TypeError("schema should be StructType or list or None, but got: %s" % schema)
# convert python objects to sql data
data = [schema.toInternal(row) for row in data]
return self._sc.parallelize(data), schema
def _get_numpy_record_dtype(self, rec):
"""
Used when converting a pandas.DataFrame to Spark using to_records(), this will correct
the dtypes of fields in a record so they can be properly loaded into Spark.
:param rec: a numpy record to check field dtypes
:return corrected dtype for a numpy.record or None if no correction needed
"""
import numpy as np
cur_dtypes = rec.dtype
col_names = cur_dtypes.names
record_type_list = []
has_rec_fix = False
for i in xrange(len(cur_dtypes)):
curr_type = cur_dtypes[i]
# If type is a datetime64 timestamp, convert to microseconds
# NOTE: if dtype is datetime[ns] then np.record.tolist() will output values as longs,
# conversion from [us] or lower will lead to py datetime objects, see SPARK-22417
if curr_type == np.dtype('datetime64[ns]'):
curr_type = 'datetime64[us]'
has_rec_fix = True
record_type_list.append((str(col_names[i]), curr_type))
return np.dtype(record_type_list) if has_rec_fix else None
def _convert_from_pandas(self, pdf, schema, timezone):
"""
Convert a pandas.DataFrame to list of records that can be used to make a DataFrame
:return list of records
"""
if timezone is not None:
from pyspark.sql.types import _check_series_convert_timestamps_tz_local
copied = False
if isinstance(schema, StructType):
for field in schema:
# TODO: handle nested timestamps, such as ArrayType(TimestampType())?
if isinstance(field.dataType, TimestampType):
s = _check_series_convert_timestamps_tz_local(pdf[field.name], timezone)
if s is not pdf[field.name]:
if not copied:
# Copy once if the series is modified to prevent the original
# Pandas DataFrame from being updated
pdf = pdf.copy()
copied = True
pdf[field.name] = s
else:
for column, series in pdf.iteritems():
s = _check_series_convert_timestamps_tz_local(series, timezone)
if s is not series:
if not copied:
# Copy once if the series is modified to prevent the original
# Pandas DataFrame from being updated
pdf = pdf.copy()
copied = True
pdf[column] = s
# Convert pandas.DataFrame to list of numpy records
np_records = pdf.to_records(index=False)
# Check if any columns need to be fixed for Spark to infer properly
if len(np_records) > 0:
record_dtype = self._get_numpy_record_dtype(np_records[0])
if record_dtype is not None:
return [r.astype(record_dtype).tolist() for r in np_records]
# Convert list of numpy records to python lists
return [r.tolist() for r in np_records]
def _create_from_pandas_with_arrow(self, pdf, schema, timezone):
"""
Create a DataFrame from a given pandas.DataFrame by slicing it into partitions, converting
to Arrow data, then sending to the JVM to parallelize. If a schema is passed in, the
data types will be used to coerce the data in Pandas to Arrow conversion.
"""
from pyspark.serializers import ArrowStreamSerializer, _create_batch
from pyspark.sql.types import from_arrow_schema, to_arrow_type, TimestampType
from pyspark.sql.utils import require_minimum_pandas_version, \
require_minimum_pyarrow_version
require_minimum_pandas_version()
require_minimum_pyarrow_version()
from pandas.api.types import is_datetime64_dtype, is_datetime64tz_dtype
# Determine arrow types to coerce data when creating batches
if isinstance(schema, StructType):
arrow_types = [to_arrow_type(f.dataType) for f in schema.fields]
elif isinstance(schema, DataType):
raise ValueError("Single data type %s is not supported with Arrow" % str(schema))
else:
# Any timestamps must be coerced to be compatible with Spark
arrow_types = [to_arrow_type(TimestampType())
if is_datetime64_dtype(t) or is_datetime64tz_dtype(t) else None
for t in pdf.dtypes]
# Slice the DataFrame to be batched
step = -(-len(pdf) // self.sparkContext.defaultParallelism) # round int up
pdf_slices = (pdf[start:start + step] for start in xrange(0, len(pdf), step))
# Create Arrow record batches
batches = [_create_batch([(c, t) for (_, c), t in zip(pdf_slice.iteritems(), arrow_types)],
timezone)
for pdf_slice in pdf_slices]
# Create the Spark schema from the first Arrow batch (always at least 1 batch after slicing)
if isinstance(schema, (list, tuple)):
struct = from_arrow_schema(batches[0].schema)
for i, name in enumerate(schema):
struct.fields[i].name = name
struct.names[i] = name
schema = struct
jsqlContext = self._wrapped._jsqlContext
def reader_func(temp_filename):
return self._jvm.PythonSQLUtils.readArrowStreamFromFile(jsqlContext, temp_filename)
def create_RDD_server():
return self._jvm.ArrowRDDServer(jsqlContext)
# Create Spark DataFrame from Arrow stream file, using one batch per partition
jrdd = self._sc._serialize_to_jvm(batches, ArrowStreamSerializer(), reader_func,
create_RDD_server)
jdf = self._jvm.PythonSQLUtils.toDataFrame(jrdd, schema.json(), jsqlContext)
df = DataFrame(jdf, self._wrapped)
df._schema = schema
return df
@staticmethod
def _create_shell_session():
"""
Initialize a SparkSession for a pyspark shell session. This is called from shell.py
to make error handling simpler without needing to declare local variables in that
script, which would expose those to users.
"""
import py4j
from pyspark.conf import SparkConf
from pyspark.context import SparkContext
try:
# Try to access HiveConf, it will raise exception if Hive is not added
conf = SparkConf()
if conf.get('spark.sql.catalogImplementation', 'hive').lower() == 'hive':
SparkContext._jvm.org.apache.hadoop.hive.conf.HiveConf()
return SparkSession.builder\
.enableHiveSupport()\
.getOrCreate()
else:
return SparkSession.builder.getOrCreate()
except (py4j.protocol.Py4JError, TypeError):
if conf.get('spark.sql.catalogImplementation', '').lower() == 'hive':
warnings.warn("Fall back to non-hive support because failing to access HiveConf, "
"please make sure you build spark with hive")
return SparkSession.builder.getOrCreate()
@since(2.0)
@ignore_unicode_prefix
def createDataFrame(self, data, schema=None, samplingRatio=None, verifySchema=True):
"""
Creates a :class:`DataFrame` from an :class:`RDD`, a list or a :class:`pandas.DataFrame`.
When ``schema`` is a list of column names, the type of each column
will be inferred from ``data``.
When ``schema`` is ``None``, it will try to infer the schema (column names and types)
from ``data``, which should be an RDD of :class:`Row`,
or :class:`namedtuple`, or :class:`dict`.
When ``schema`` is :class:`pyspark.sql.types.DataType` or a datatype string, it must match
the real data, or an exception will be thrown at runtime. If the given schema is not
:class:`pyspark.sql.types.StructType`, it will be wrapped into a
:class:`pyspark.sql.types.StructType` as its only field, and the field name will be "value",
each record will also be wrapped into a tuple, which can be converted to row later.
If schema inference is needed, ``samplingRatio`` is used to determined the ratio of
rows used for schema inference. The first row will be used if ``samplingRatio`` is ``None``.
:param data: an RDD of any kind of SQL data representation(e.g. row, tuple, int, boolean,
etc.), or :class:`list`, or :class:`pandas.DataFrame`.
:param schema: a :class:`pyspark.sql.types.DataType` or a datatype string or a list of
column names, default is ``None``. The data type string format equals to
:class:`pyspark.sql.types.DataType.simpleString`, except that top level struct type can
omit the ``struct<>`` and atomic types use ``typeName()`` as their format, e.g. use
``byte`` instead of ``tinyint`` for :class:`pyspark.sql.types.ByteType`. We can also use
``int`` as a short name for ``IntegerType``.
:param samplingRatio: the sample ratio of rows used for inferring
:param verifySchema: verify data types of every row against schema.
:return: :class:`DataFrame`
.. versionchanged:: 2.1
Added verifySchema.
.. note:: Usage with spark.sql.execution.arrow.enabled=True is experimental.
>>> l = [('Alice', 1)]
>>> spark.createDataFrame(l).collect()
[Row(_1=u'Alice', _2=1)]
>>> spark.createDataFrame(l, ['name', 'age']).collect()
[Row(name=u'Alice', age=1)]
>>> d = [{'name': 'Alice', 'age': 1}]
>>> spark.createDataFrame(d).collect()
[Row(age=1, name=u'Alice')]
>>> rdd = sc.parallelize(l)
>>> spark.createDataFrame(rdd).collect()
[Row(_1=u'Alice', _2=1)]
>>> df = spark.createDataFrame(rdd, ['name', 'age'])
>>> df.collect()
[Row(name=u'Alice', age=1)]
>>> from pyspark.sql import Row
>>> Person = Row('name', 'age')
>>> person = rdd.map(lambda r: Person(*r))
>>> df2 = spark.createDataFrame(person)
>>> df2.collect()
[Row(name=u'Alice', age=1)]
>>> from pyspark.sql.types import *
>>> schema = StructType([
... StructField("name", StringType(), True),
... StructField("age", IntegerType(), True)])
>>> df3 = spark.createDataFrame(rdd, schema)
>>> df3.collect()
[Row(name=u'Alice', age=1)]
>>> spark.createDataFrame(df.toPandas()).collect() # doctest: +SKIP
[Row(name=u'Alice', age=1)]
>>> spark.createDataFrame(pandas.DataFrame([[1, 2]])).collect() # doctest: +SKIP
[Row(0=1, 1=2)]
>>> spark.createDataFrame(rdd, "a: string, b: int").collect()
[Row(a=u'Alice', b=1)]
>>> rdd = rdd.map(lambda row: row[1])
>>> spark.createDataFrame(rdd, "int").collect()
[Row(value=1)]
>>> spark.createDataFrame(rdd, "boolean").collect() # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
Py4JJavaError: ...
"""
if isinstance(data, DataFrame):
raise TypeError("data is already a DataFrame")
if isinstance(schema, basestring):
schema = _parse_datatype_string(schema)
elif isinstance(schema, (list, tuple)):
# Must re-encode any unicode strings to be consistent with StructField names
schema = [x.encode('utf-8') if not isinstance(x, str) else x for x in schema]
try:
import pandas
has_pandas = True
except Exception:
has_pandas = False
if has_pandas and isinstance(data, pandas.DataFrame):
from pyspark.sql.utils import require_minimum_pandas_version
require_minimum_pandas_version()
if self._wrapped._conf.pandasRespectSessionTimeZone():
timezone = self._wrapped._conf.sessionLocalTimeZone()
else:
timezone = None
# If no schema supplied by user then get the names of columns only
if schema is None:
schema = [str(x) if not isinstance(x, basestring) else
(x.encode('utf-8') if not isinstance(x, str) else x)
for x in data.columns]
if self._wrapped._conf.arrowEnabled() and len(data) > 0:
try:
return self._create_from_pandas_with_arrow(data, schema, timezone)
except Exception as e:
from pyspark.util import _exception_message
if self._wrapped._conf.arrowFallbackEnabled():
msg = (
"createDataFrame attempted Arrow optimization because "
"'spark.sql.execution.arrow.enabled' is set to true; however, "
"failed by the reason below:\n %s\n"
"Attempting non-optimization as "
"'spark.sql.execution.arrow.fallback.enabled' is set to "
"true." % _exception_message(e))
warnings.warn(msg)
else:
msg = (
"createDataFrame attempted Arrow optimization because "
"'spark.sql.execution.arrow.enabled' is set to true, but has reached "
"the error below and will not continue because automatic fallback "
"with 'spark.sql.execution.arrow.fallback.enabled' has been set to "
"false.\n %s" % _exception_message(e))
warnings.warn(msg)
raise
data = self._convert_from_pandas(data, schema, timezone)
if isinstance(schema, StructType):
verify_func = _make_type_verifier(schema) if verifySchema else lambda _: True
def prepare(obj):
verify_func(obj)
return obj
elif isinstance(schema, DataType):
dataType = schema
schema = StructType().add("value", schema)
verify_func = _make_type_verifier(
dataType, name="field value") if verifySchema else lambda _: True
def prepare(obj):
verify_func(obj)
return obj,
else:
prepare = lambda obj: obj
if isinstance(data, RDD):
rdd, schema = self._createFromRDD(data.map(prepare), schema, samplingRatio)
else:
rdd, schema = self._createFromLocal(map(prepare, data), schema)
jrdd = self._jvm.SerDeUtil.toJavaArray(rdd._to_java_object_rdd())
jdf = self._jsparkSession.applySchemaToPythonRDD(jrdd.rdd(), schema.json())
df = DataFrame(jdf, self._wrapped)
df._schema = schema
return df
@ignore_unicode_prefix
@since(2.0)
def sql(self, sqlQuery):
"""Returns a :class:`DataFrame` representing the result of the given query.
:return: :class:`DataFrame`
>>> df.createOrReplaceTempView("table1")
>>> df2 = spark.sql("SELECT field1 AS f1, field2 as f2 from table1")
>>> df2.collect()
[Row(f1=1, f2=u'row1'), Row(f1=2, f2=u'row2'), Row(f1=3, f2=u'row3')]
"""
return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
@since(2.0)
def table(self, tableName):
"""Returns the specified table as a :class:`DataFrame`.
:return: :class:`DataFrame`
>>> df.createOrReplaceTempView("table1")
>>> df2 = spark.table("table1")
>>> sorted(df.collect()) == sorted(df2.collect())
True
"""
return DataFrame(self._jsparkSession.table(tableName), self._wrapped)
@property
@since(2.0)
def read(self):
"""
Returns a :class:`DataFrameReader` that can be used to read data
in as a :class:`DataFrame`.
:return: :class:`DataFrameReader`
"""
return DataFrameReader(self._wrapped)
@property
@since(2.0)
def readStream(self):
"""
Returns a :class:`DataStreamReader` that can be used to read data streams
as a streaming :class:`DataFrame`.
.. note:: Evolving.
:return: :class:`DataStreamReader`
"""
return DataStreamReader(self._wrapped)
@property
@since(2.0)
def streams(self):
"""Returns a :class:`StreamingQueryManager` that allows managing all the
:class:`StreamingQuery` StreamingQueries active on `this` context.
.. note:: Evolving.
:return: :class:`StreamingQueryManager`
"""
from pyspark.sql.streaming import StreamingQueryManager
return StreamingQueryManager(self._jsparkSession.streams())
@since(2.0)
def stop(self):
"""Stop the underlying :class:`SparkContext`.
"""
self._sc.stop()
# We should clean the default session up. See SPARK-23228.
self._jvm.SparkSession.clearDefaultSession()
SparkSession._instantiatedSession = None
@since(2.0)
def __enter__(self):
"""
Enable 'with SparkSession.builder.(...).getOrCreate() as session: app' syntax.
"""
return self
@since(2.0)
def __exit__(self, exc_type, exc_val, exc_tb):
"""
Enable 'with SparkSession.builder.(...).getOrCreate() as session: app' syntax.
Specifically stop the SparkSession on exit of the with block.
"""
self.stop()
def _test():
import os
import doctest
from pyspark.context import SparkContext
from pyspark.sql import Row
import pyspark.sql.session
os.chdir(os.environ["SPARK_HOME"])
globs = pyspark.sql.session.__dict__.copy()
sc = SparkContext('local[4]', 'PythonTest')
globs['sc'] = sc
globs['spark'] = SparkSession(sc)
globs['rdd'] = rdd = sc.parallelize(
[Row(field1=1, field2="row1"),
Row(field1=2, field2="row2"),
Row(field1=3, field2="row3")])
globs['df'] = rdd.toDF()
(failure_count, test_count) = doctest.testmod(
pyspark.sql.session, globs=globs,
optionflags=doctest.ELLIPSIS | doctest.NORMALIZE_WHITESPACE)
globs['sc'].stop()
if failure_count:
sys.exit(-1)
if __name__ == "__main__":
_test()

5
docker/tidb/tispark/tispark-tests/tests/loaddata.sh
Unescape Escape View File

@ -1,5 +0,0 @@
#!/usr/bin/env bash
cd /opt/spark/data/tispark-sample-data
mysql -h tidb -P 4000 -u root < dss.ddl

9
docker/tidb/tispark/tispark-tests/tests/tests.py
Unescape Escape View File

@ -1,9 +0,0 @@
from pyspark.sql import SparkSession
spark = SparkSession.builder.master("spark://tispark-master:7077").appName("TiSpark tests").getOrCreate()
spark.sql("use TPCH_001")
count = spark.sql("select count(*) as c from lineitem").first()['c']
assert 60175 == count

190
docker/tidb/tools/container_debug
Unescape Escape View File

@ -1,190 +0,0 @@
#!/bin/bash
# Copyright 2018 PingCAP, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# See the License for the specific language governing permissions and
# limitations under the License.
#* --------------------------------------------------------------------- */
#* log configure */
#* --------------------------------------------------------------------- */
### Define logging color
COLOR_ORIGIN="\033[0m"
COLOR_GREEN="\033[32m"
COLOR_YELLOW="\033[33m"
COLOR_RED="\033[31m"
### Define logging level
LOGGER_LEVEL="3"
### Define common logger
function logger() {
cur_level=$1
cur_type=$2
cur_color=$3
shift && shift && shift
cur_msg=$*
[[ ${LOGGER_LEVEL} -lt ${cur_level} ]] && return 0
pre_fix="${cur_color}[${cur_type}][$(date +%F)][$(date +%T)]"
pos_fix="${COLOR_ORIGIN}"
echo -e "${pre_fix} ${cur_msg} ${pos_fix}"
}
### Define notice logger
function notice() {
logger 3 "NOTICE" ${COLOR_GREEN} $*
}
### Define warning logger
function warning() {
logger 2 "WARNING" ${COLOR_YELLOW} $*
}
### Define fatal logger
function fatal() {
logger 1 "FATAL" ${COLOR_RED} $*
exit 1
}
##########################################################################
function print_help() {
echo "\
${1:-Debug tool for container.}
Usage:
container_debug [OPTIONS] [ARG]
Options:
-i The container's identity, possible values are 'containerID' or 'containerName'
-s The service name defined in docker-compose
-w Run pprof via a web interface for go program
-p The binary path of the debugged process in its own container
-h Print help infomation
When you enter the debug container, you can find the pid of the debugged process through the ps command,
then you can find the binary of the debugged process through this path /proc/\${pid}/root/\${binary_path}.
\${binary_path} represents the binary path of the debugged process in its own container.
\${pid} represents the process id of the debugged process as seen in the debug container.
" >&2
exit
}
###############################variable define##################################
WORKSPACE=$(cd $(dirname $0)/..; pwd)
DEBUG_IMAGE=${DEBUG_IMAGE:-uhub.service.ucloud.cn/pingcap/tidb-debug:latest}
SUFFIX=$(uuidgen|cut -d'-' -f1|tr '[A-Z]' '[a-z]')
DEBUG_CONTAINER_NAME=debug-${SUFFIX}
TMP_FILE=$(mktemp /tmp/binary.XXXXXX)
################################################################################
if [[ $# -eq 0 ]]
then
print_help
fi
function cleanup() {
notice "start to clean tmp file ${TMP_FILE}"
[[ -f ${TMP_FILE} ]] && rm -f ${TMP_FILE}
}
### register signal processing function
trap cleanup EXIT
### change workspace
cd $WORKSPACE
optstring=":i:s:p:wh"
while getopts "$optstring" opt; do
case $opt in
i)
container_id=${OPTARG}
;;
s)
service_name=${OPTARG}
;;
p)
binary_path=${OPTARG}
;;
w)
web=true
;;
h)
print_help
;;
\?)
fatal "Invalid option: -$OPTARG" >&2
;;
:)
fatal "Option -$OPTARG requires an argument" >&2
;;
esac
done
if [[ -z ${service_name} && -z ${container_id} ]]
then
fatal "please use -s or -i options to select the target container" >&2
elif [[ ! -z ${container_id} ]]
then
### If both -s and -i options are specified, the -i option is preferred
cid=${container_id}
else
cprefix=$(basename $(pwd)|tr -Cd '[A-Za-z0-9]'|tr '[A-Z]' '[a-z]')
cid="${cprefix}_${service_name}_1"
docker ps | grep ${cid} >/dev/null
[[ $? -ne 0 ]] && fatal "not found docker-compose service ${service_name}, please confirm the correct docker-compose service name" >&2
fi
if [[ ! -z ${binary_path} ]]
then
binary_name=$(basename ${binary_path})
docker cp ${cid}:${binary_path} ${TMP_FILE}
if [[ $? -ne 0 ]]
then
### not found binary in container, reset variable ${binary_name}
binary_name=
warning "not found ${binary_path} in container ${cid}, please specify the correct binary path in container" >&2
fi
fi
if [[ ! -z ${web} ]]
then
### starts a web server for graphic visualizations of golang program profiles
### generate a random web port
### TODO: Test whether this port has been used
wport=${RANDOM}
[[ ${wport} -lt 10000 ]] && wport=$((wport+10000))
### get the container exposed port
cport=$(docker port ${cid}|grep -E '[0-9]{5}'|awk -F: '{print $NF}')
notice "starts a web server on localhost:${wport}"
pprof -http=:${wport} ${TMP_FILE} http://localhost:${cport}/debug/pprof/profile
else
### enter debug container to debug the specified container
docker_run_args=(-ti --rm --name=${DEBUG_CONTAINER_NAME})
docker_run_args+=(--pid=container:${cid})
docker_run_args+=(--network=container:${cid})
docker_run_args+=(--ipc=container:${cid})
docker_run_args+=(--cap-add=SYS_PTRACE)
docker_run_args+=(--privileged=true)
if [[ ! -z ${binary_name} && -e ${TMP_FILE} ]]
then
docker_run_args+=(-v ${TMP_FILE}:/${binary_name})
else
notice "you can access the debugged container ${cid} file system through this path /proc/\${DEBUGGED_PROCESS_PID}/root"
fi
docker_run_args+=($DEBUG_IMAGE)
docker run ${docker_run_args[@]}
fi

2
framework/skynet

@ -1 +1 @@
Subproject commit bc3945da740d9590cffc68e0a821e6d622e1c7bd
Subproject commit 6ee8d23ac480e37e9d1802cd7bab6f3884c0734d
Write Preview
Loading…
Cancel
Save