分析ADG网卡流量出现规律性毛刺现象
一、介绍
客户ADG环境，刚上线的业务系统，通过ZABBIX进行监控，发现每半个小时会出现ADG网卡流量超过1M，持续2分钟后，平均网络流量在100K左右。
创新互联专注于台儿企业网站建设,自适应网站建设,商城网站开发。台儿网站建设公司,为台儿等地区提供建站服务。全流程按需制作，专业设计，全程项目跟踪，创新互联专业和态度为您提供的服务
二、分析
1、ADG 环境备库流量主要来自主库redo的数据传输，或者出现网络问题，导致归档无法传输，恢复后归档日志的传输。
2、查看归档日志产生的频率及大小，检查告警日志是否有归档日志传输。
3、经过排查并未发现归档日志传输。检查数据库job,发现并没有客户创建的job。
4、检查系统自带的Job，dba_scheduler_jobs,我们会发现很多系统自带的job.
5、数据库自身自动写入的数据，而且有定时计划的可能会导致这个问题。
6、查看awr的写入的频率发现是每半个小时执行一次。
7、查看awrinfo.sql详细信息
以下是测试环境的的awrinfo.sql结果，大家需要熟悉了解。
SQL> @?/rdbms/admin/awrinfo.sql
This script will report general AWR information
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Specify the Report File Name
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The default report file name is awrinfo.txt. To use this name,
press to continue, otherwise enter an alternative.
Enter value for report_name:
Using the report name awrinfo.txt
No errors.
No errors.
~~~~~~~~~~~~~~~
AWR INFO Report
~~~~~~~~~~~~~~~
Report generated at 11:06:42 on Jan 26, 2018 ( Friday ) in Timezone +08:00
Warning: Non Default AWR Setting!
--------------------------------------------------------------------------------
Snapshot interval is 60 minutes and Retention is 8 days
DB_ID DB_NAME HOST_PLATFORM INST STARTUP_TIME LAST_ASH_SID PAR
------------ --------- ---------------------------------------- ----- ----------------- ------------ ---
* 1486195868 ORCL roidb01 - Linux x86 64-bit 1 09:30:24 (01/26) 32052 NO
########################################################
(I) AWR Snapshots Information
########################################################
*****************************************************
(1a) SYSAUX usage - Schema breakdown (dba_segments)
*****************************************************
|
| Total SYSAUX size 660.9 MB ( 2% of 32,768.0 MB MAX with AUTOEXTEND ON )
|
| Schema APEX_040200 occupies 247.6 MB ( 37.5% )
| Schema SYS occupies 200.1 MB ( 30.3% )
| Schema MDSYS occupies 93.8 MB ( 14.2% )
| Schema XDB occupies 68.5 MB ( 10.4% )
| Schema ORDDATA occupies 16.1 MB ( 2.4% )
| Schema SYSTEM occupies 15.8 MB ( 2.4% )
| Schema WMSYS occupies 7.2 MB ( 1.1% )
| Schema DVSYS occupies 4.4 MB ( 0.7% )
| Schema CTXSYS occupies 3.7 MB ( 0.6% )
| Schema AUDSYS occupies 1.5 MB ( 0.2% )
| Schema GSMADMIN_INT occupies 1.4 MB ( 0.2% )
| Schema DBSNMP occupies 0.5 MB ( 0.1% )
| Schema ORDSYS occupies 0.4 MB ( 0.1% )
|
********************************************************
(1b) SYSAUX occupants space usage (v$sysaux_occupants)
********************************************************
|
| Occupant Name Schema Name Space Usage
| -------------------- -------------------- ----------------
| SDO MDSYS 93.8 MB
| XDB XDB 68.5 MB
| SM/OTHER SYS 53.9 MB
| SM/AWR SYS 40.4 MB
| AO SYS 38.8 MB
| XSOQHIST SYS 38.8 MB
| ORDIM/ORDDATA ORDDATA 16.1 MB
| LOGMNR SYSTEM 13.9 MB
| SM/OPTSTAT SYS 11.4 MB
| SM/ADVISOR SYS 8.4 MB
| WM WMSYS 7.2 MB
| TEXT CTXSYS 3.7 MB
| SQL_MANAGEMENT_BASE SYS 2.4 MB
| PL/SCOPE SYS 1.6 MB
| AUDSYS AUDSYS 1.5 MB
| LOGSTDBY SYSTEM 1.5 MB
| STREAMS SYS 1.0 MB
| JOB_SCHEDULER SYS 0.8 MB
| EM_MONITORING_USER DBSNMP 0.5 MB
| ORDIM ORDSYS 0.4 MB
| SMON_SCN_TIME SYS 0.4 MB
| AUTO_TASK SYS 0.3 MB
| AUDIT_TABLES SYS 0.0 MB
| EM SYSMAN 0.0 MB
| EXPRESSION_FILTER EXFSYS 0.0 MB
| ORDIM/ORDPLUGINS ORDPLUGINS 0.0 MB
| ORDIM/SI_INFORMTN_SC SI_INFORMTN_SCHEMA 0.0 MB
| STATSPACK PERFSTAT 0.0 MB
| TSM TSMSYS 0.0 MB
| ULTRASEARCH WKSYS 0.0 MB
| ULTRASEARCH_DEMO_USE WK_TEST 0.0 MB
| XSAMD OLAPSYS 0.0 MB
|
| Others (Unaccounted space) 255.8 MB
|
******************************************
(1c) SYSAUX usage - Unregistered Schemas
******************************************
| This section displays schemas that are not registered
| in V$SYSAUX_OCCUPANTS
|
| Schema APEX_040200 occupies 247.6 MB
| Schema DVSYS occupies 4.4 MB
| Schema GSMADMIN_INT occupies 1.4 MB
|
| Total space 253.4 MB
|
*************************************************************
(1d) SYSAUX usage - Unaccounted space in registered schemas
*************************************************************
|
| This section displays unaccounted space in the registered
| schemas of V$SYSAUX_OCCUPANTS.
|
| Unaccounted space in SYS/SYSTEM 2.4 MB
|
| Total space 2.4 MB
|
*************************************
(2) Size estimates for AWR snapshots
*************************************
|
| Estimates based on 60 mins snapshot INTERVAL:
| AWR size/day 485.3 MB (20,704 K/snap * 24 snaps/day)
| AWR size/wk 3,396.8 MB (size_per_day * 7) per instance
|
| Estimates based on 2 snaps in past 24 hours:
| AWR size/day 606.6 MB (20,704 K/snap and 2 snaps in past 1.6 hours)
| AWR size/wk 4,245.9 MB (size_per_day * 7) per instance
|
**********************************
(3a) Space usage by AWR components (per database)
**********************************
COMPONENT MB % AWR KB_PER_SNAP MB_PER_DAY MB_PER_WEEK TABLE% : INDEX%
--------- --------- ------ ------------ ---------- ----------- ----------------
FIXED 25.6 63.4 13,120 384.4 2,690.6 48% : 52%
SPACE 3.1 7.6 1,568 45.9 321.6 51% : 49%
SQLPLAN 2.4 6.0 1,248 36.6 255.9 82% : 18%
EVENTS 2.3 5.6 1,152 33.8 236.3 47% : 53%
SQL 1.8 4.5 928 27.2 190.3 52% : 48%
RAC 0.6 1.5 320 9.4 65.6 50% : 50%
SQLTEXT 0.4 0.9 192 5.6 39.4 83% : 17%
ASH 0.4 0.9 192 5.6 39.4 50% : 50%
SQLBIND 0.3 0.6 128 3.8 26.3 50% : 50%
**********************************
(3b) Space usage within AWR Components (> 500K)
**********************************
COMPONENT MB SEGMENT_NAME - % SPACE_USED SEGMENT_TYPE
--------- --------- --------------------------------------------------------------------- ---------------
FIXED 2.0 WRH$_SYSSTAT_PK.WRH$_SYSSTA_1486195868_0 - 66% INDEX PARTITION
FIXED 0.9 WRH$_LATCH.WRH$_LATCH_1486195868_0 - 80% TABLE PARTITION
FIXED 0.8 WRH$_LATCH_PK.WRH$_LATCH_1486195868_0 - 72% INDEX PARTITION
FIXED 0.8 WRH$_SYSMETRIC_HISTORY_INDEX - 26% INDEX
FIXED 0.8 WRH$_SYSSTAT.WRH$_SYSSTA_1486195868_0 - 85% TABLE PARTITION
FIXED 0.6 WRH$_SYSMETRIC_HISTORY - 19% TABLE
SQLPLAN 2.0 WRH$_SQL_PLAN - 67% TABLE
SQL 0.5 WRH$_SQLSTAT.WRH$_SQLSTA_1486195868_0 - 81% TABLE PARTITION
**********************************
(4) Space usage by non-AWR components (> 500K)
**********************************
COMPONENT MB SEGMENT_NAME SEGMENT_TYPE
--------- --------- --------------------------------------------------------------------- ---------------
NON_AWR 51.2 MDSYS.SYS_LOB0000083702C00006$$ LOBSEGMENT
NON_AWR 42.2 SYS.SYS_LOB0000006421C00005$$ LOBSEGMENT
NON_AWR 23.2 APEX_040200.SYS_LOB0000088338C00007$$ LOBSEGMENT
NON_AWR 19.0 APEX_040200.WWV_FLOW_PAGE_PLUGS TABLE
NON_AWR 17.2 SYS.SYS_LOB0000085557C00004$$ LOBSEGMENT
NON_AWR 11.0 APEX_040200.WWV_FLOW_STEP_ITEMS TABLE
NON_AWR 10.0 APEX_040200.WWV_FLOW_TEMPLATES TABLE
NON_AWR 9.2 APEX_040200.SYS_LOB0000088569C00009$$ LOBSEGMENT
NON_AWR 9.0 MDSYS.SDO_CS_SRS TABLE
NON_AWR 8.1 APEX_040200.SYS_LOB0000087756C00016$$ LOBSEGMENT
NON_AWR 8.0 SYS.HEATMAP SYSTEM STATISTI
CS
NON_AWR 7.2 SYS.SYS_LOB0000085542C00004$$ LOBSEGMENT
NON_AWR 6.0 APEX_040200.WWV_FLOW_STEP_PROCESSING TABLE
NON_AWR 5.2 APEX_040200.SYS_LOB0000087810C00010$$ LOBSEGMENT
NON_AWR 5.1 SYS.SYS_LOB0000006421C00004$$ LOBSEGMENT
NON_AWR 4.0 SYSTEM.SYS_LOB0000001340C00009$$ LOBSEGMENT
NON_AWR 4.0 APEX_040200.WWV_FLOW_REGION_REPORT_COLUMN TABLE
NON_AWR 4.0 APEX_040200.WWV_FLOW_STEPS TABLE
NON_AWR 3.2 SYS.SYS_LOB0000085537C00004$$ LOBSEGMENT
NON_AWR 3.2 APEX_040200.SYS_LOB0000088421C00019$$ LOBSEGMENT
NON_AWR 3.2 SYS.SYS_LOB0000085547C00004$$ LOBSEGMENT
NON_AWR 3.0 APEX_040200.WWV_FLOW_STEP_BUTTONS TABLE
NON_AWR 3.0 APEX_040200.WWV_FLOW_WORKSHEET_COLUMNS TABLE
NON_AWR 3.0 APEX_040200.WWV_FLOW_INSTALL_SCRIPTS TABLE
NON_AWR 3.0 APEX_040200.WWV_FLOW_PAGE_PLUG_TEMPLATES TABLE
NON_AWR 2.2 SYS.SYS_LOB0000085552C00004$$ LOBSEGMENT
NON_AWR 2.2 APEX_040200.SYS_LOB0000088585C00007$$ LOBSEGMENT
NON_AWR 2.2 APEX_040200.SYS_LOB0000087738C00022$$ LOBSEGMENT
NON_AWR 2.2 APEX_040200.SYS_LOB0000087738C00028$$ LOBSEGMENT
NON_AWR 2.2 APEX_040200.SYS_LOB0000087667C00011$$ LOBSEGMENT
NON_AWR 2.2 SYS.SYS_LOB0000085562C00004$$ LOBSEGMENT
NON_AWR 2.1 SYS.SYS_LOB0000008312C00038$$ LOBSEGMENT
NON_AWR 2.0 SYS.I_WRI$_OPTSTAT_HH_OBJ_ICOL_ST INDEX
NON_AWR 2.0 APEX_040200.WWV_FLOW_WORKSHEETS TABLE
NON_AWR 2.0 APEX_040200.WWV_FLOW_FLASH_MAP_REGIONS TABLE
NON_AWR 2.0 APEX_040200.WWV_FLOW_LIST_ITEMS TABLE
NON_AWR 2.0 APEX_040200.WWV_FLOW_LIST_TEMPLATES TABLE
NON_AWR 2.0 APEX_040200.WWV_FLOW_ROW_TEMPLATES TABLE
NON_AWR 2.0 APEX_040200.WWV_FLOW_STEP_VALIDATIONS TABLE
NON_AWR 2.0 APEX_040200.WWV_FLOW_STEP_ITEM_HELP TABLE
NON_AWR 2.0 APEX_040200.WWV_FLOW_STEP_ITEMS_U_NAME INDEX
NON_AWR 2.0 APEX_040200.WWV_FLOW_STEP_BRANCHES TABLE
NON_AWR 2.0 APEX_040200.WWV_FLOW_PAGE_PLUGS_IDX_U1 INDEX
NON_AWR 2.0 APEX_040200.WWV_FLOW_PAGE_PLUGS_IDX3 INDEX
NON_AWR 2.0 MDSYS.SDO_COORD_REF_SYS TABLE
COMPONENT MB SEGMENT_NAME SEGMENT_TYPE
--------- --------- --------------------------------------------------------------------- ---------------
NON_AWR 2.0 XDB.XDB$ELEMENT TABLE
NON_AWR 1.2 SYS.SYS_LOB0000008304C00004$$ LOBSEGMENT
NON_AWR 1.2 AUDSYS.SYS_LOB0000091784C00014$$.SYS_LOB_P186 LOB PARTITION
NON_AWR 1.2 APEX_040200.SYS_LOB0000089973C00010$$ LOBSEGMENT
NON_AWR 1.2 APEX_040200.SYS_LOB0000088446C00006$$ LOBSEGMENT
NON_AWR 1.2 APEX_040200.SYS_LOB0000088432C00007$$ LOBSEGMENT
NON_AWR 1.2 APEX_040200.SYS_LOB0000088387C00088$$ LOBSEGMENT
NON_AWR 1.2 APEX_040200.SYS_LOB0000088332C00021$$ LOBSEGMENT
NON_AWR 1.2 APEX_040200.SYS_LOB0000088326C00005$$ LOBSEGMENT
NON_AWR 1.2 APEX_040200.SYS_LOB0000087738C00025$$ LOBSEGMENT
NON_AWR 1.2 APEX_040200.SYS_LOB0000087700C00007$$ LOBSEGMENT
NON_AWR 1.2 APEX_040200.SYS_LOB0000087667C00024$$ LOBSEGMENT
NON_AWR 1.2 MDSYS.SYS_LOB0000086565C00003$$ LOBSEGMENT
NON_AWR 1.2 MDSYS.SYS_LOB0000084268C00003$$ LOBSEGMENT
NON_AWR 1.2 ORDDATA.SYS_LOB0000077885C00005$$ LOBSEGMENT
NON_AWR 1.2 ORDDATA.SYS_LOB0000077702C00005$$ LOBSEGMENT
NON_AWR 1.2 XDB.SYS_LOB0000019097C00025$$ LOBSEGMENT
NON_AWR 1.2 XDB.SYS_LOB0000019043C00023$$ LOBSEGMENT
NON_AWR 1.2 SYS.SYS_LOB0000006450C00009$$.SYS_LOB_P228 LOB PARTITION
NON_AWR 1.1 SYS.SYS_LOB0000010986C00001$$ LOBSEGMENT
NON_AWR 1.1 APEX_040200.SYS_LOB0000088559C00009$$ LOBSEGMENT
NON_AWR 1.1 APEX_040200.SYS_LOB0000087999C00013$$ LOBSEGMENT
NON_AWR 1.1 SYS.SYS_LOB0000006450C00009$$.SYS_LOB_P353 LOB PARTITION
NON_AWR 1.0 SYS.WRI$_OPTSTAT_OPR_TASKS TABLE
NON_AWR 1.0 APEX_040200.WWV_FLOW_MESSAGES$ TABLE
NON_AWR 1.0 SYS.SYS$SERVICE_METRICS_TAB TABLE
NON_AWR 1.0 SYS.I_WRI$_OPTSTAT_H_OBJ#_ICOL#_ST INDEX
NON_AWR 1.0 SYS.LTXID_TRANS.LTXID_TRANS_1 TABLE PARTITION
NON_AWR 0.9 SYS.I_WRI$_OPTSTAT_HH_ST INDEX
NON_AWR 0.9 APEX_040200.WWV_FLOW_STEP_ITEM_IDX3 INDEX
NON_AWR 0.9 APEX_040200.WWV_FLOW_LIST_OF_VALUES_DATA TABLE
NON_AWR 0.9 APEX_040200.WWV_FLOW_PAGE_PLUGS_PK INDEX
NON_AWR 0.9 APEX_040200.WWV_FLOW_CAL_TEMPLATES TABLE
NON_AWR 0.8 APEX_040200.WWV_FLOW_STEP_ITEMS_PK INDEX
NON_AWR 0.8 APEX_040200.WWV_FLOW_PAGE_PLUGS_FK INDEX
NON_AWR 0.8 APEX_040200.WWV_FLOW_STEP_ITEM_IDX4 INDEX
NON_AWR 0.8 APEX_040200.WWV_FLOW_STEP_BUTTONS_IDX2 INDEX
NON_AWR 0.8 APEX_040200.WWV_FLOW_STEP_BUTTONS_FKIDX INDEX
NON_AWR 0.8 APEX_040200.WWV_FLOW_STEP_ITEM_IDX1 INDEX
NON_AWR 0.7 APEX_040200.WWV_FLOW_PAGE_PLUGS_IDX1 INDEX
NON_AWR 0.7 APEX_040200.WWV_FLOW_STEP_ITEM_IDX5 INDEX
NON_AWR 0.7 APEX_040200.WWV_FLOW_STEP_ITEM_IDX2 INDEX
NON_AWR 0.6 SYS.I_WRI$_OPTSTAT_OPR_TASKS_TGST INDEX
NON_AWR 0.6 APEX_040200.WWV_FLOW_PAGE_DA_ACTIONS TABLE
NON_AWR 0.6 APEX_040200.WWV_FLOW_WORKSHEET_COLUMNS_UK2 INDEX
NON_AWR 0.6 APEX_040200.WWV_FLOW_MENU_OPTIONS TABLE
NON_AWR 0.6 APEX_040200.WWV_FLOW_STEP_COMPUTATIONS TABLE
COMPONENT MB SEGMENT_NAME SEGMENT_TYPE
--------- --------- --------------------------------------------------------------------- ---------------
NON_AWR 0.6 APEX_040200.WWV_FLOW_PAGE_PLUGS_IDX2 INDEX
NON_AWR 0.6 APEX_040200.WWV_FLOW_LISTS_OF_VALUES$ TABLE
NON_AWR 0.6 MDSYS.SDO_COORD_OPS TABLE
NON_AWR 0.6 SYS.AW_OBJ$ TABLE
NON_AWR 0.6 XDB.XDB$COMPLEX_TYPE TABLE
NON_AWR 0.6 MDSYS.SDO_COORD_OP_PARAM_VALS TABLE
NON_AWR 0.6 MDSYS.COORD_OP_PARA_VAL_PRIM INDEX
NON_AWR 0.6 APEX_040200.WWV_FLOW_LOV_DATA_IDX1 INDEX
**********************************
(5a) AWR snapshots - last 50
**********************************
Total snapshots in DB 1486195868 Instance 1 = 2
DBID SNAP_ID INST FLUSH_ELAPSED ENDTM STARTUP_TIME STATUS ERRCNT
---------- ---------- ----- -------------------- ----------------- ----------------- ------ ------
1486195868 15 1 +00000 00:00:00.6 09:41:13 (01/26) 09:30:24 (01/26) 0 0
1486195868 16 1 +00000 00:00:00.3 11:00:18 (01/26) 09:30:24 (01/26) 0 0
**********************************
(5b) AWR snapshots with errors or invalid
**********************************
no rows selected
**********************************
(5c) AWR snapshots -- OLDEST Non-Baselined snapshots
**********************************
DBID INST SNAP_ID ENDTM STATUS ERROR_COUNT
---------- ----- ---------- ----------------- ------ -----------
1486195868 1 15 09:41:13 (01/26) 0 0
**********************************
(6) AWR Control Settings - interval, retention
**********************************
DBID LSNAPID LSPLITID LSNAPTIME LPURGETIME FLAG INTERVAL RETENTION VRSN
----------- -------- -------- -------------- -------------- ----- ----------------- ----------------- ----
1486195868 16 0 01/26 11:00:19 01/26 10:00:11 2 +00000 01:00:00.0 +00008 00:00:00.0 11
**********************************
(7a) AWR Contents - row counts for each snapshots
**********************************
SNAP_ID INST ASH SQL SQBND FILES SEGST SYSEVT
---------- ----- ---------- ---------- ---------- ---------- ---------- ----------
15 1 4 89 190 4 65 89
16 1 12 96 243 4 66 90
**********************************
(7b) AWR Contents - average row counts per snapshot
**********************************
SNAP_COUNT INST ASH SQLSTAT SQLBIND FILES SEGSTAT SYSEVENT
---------- ----- ---------- ---------- ---------- ---------- ---------- ----------
2 1 8 92.5 216.5 4 65.5 89.5
**********************************
(7c) AWR total item counts - names, text, plans
**********************************
SQLTEXT SQLPLAN SQLBMETA SEGOBJ DATAFILE TEMPFILE
---------- ---------- ---------- ---------- ---------- ----------
326 4983 1232 206 4 1
########################################################
(II) Advisor Framework Info
########################################################
**********************************
(1) Advisor Tasks - Last 50
**********************************
OWNER/ADVISOR TASK_ID/NAME CREATED EXE_DURATN EXE_CREATN HOW_C STATUS
-------------- -------------------------------- ---------------- ---------- ---------- ----- ------------
SYS/SQL Tuning 1/SYS_AUTO_SQL_TUNING_TASK 05:52:52 (07/07) AUTO
SYS/SPM Evolve 2/SYS_AUTO_SPM_EVOLVE_TASK 05:52:52 (07/07) AUTO
SYS/ADDM 42/ADDM:1486195868_1_16 11:00:19 (01/26) 0 0 AUTO COMPLETED
**********************************
(2) Advisor Task - Oldest 5
**********************************
OWNER/ADVISOR TASK_ID/NAME CREATED EXE_DURATN EXE_CREATN HOW_C STATUS
-------------- -------------------------------- ---------------- ---------- ---------- ----- ------------
SYS/SQL Tuning 1/SYS_AUTO_SQL_TUNING_TASK 05:52:52 (07/07) AUTO
SYS/SPM Evolve 2/SYS_AUTO_SPM_EVOLVE_TASK 05:52:52 (07/07) AUTO
SYS/ADDM 42/ADDM:1486195868_1_16 11:00:19 (01/26) 0 0 AUTO COMPLETED
**********************************
(3) Advisor Tasks With Errors - Last 50
**********************************
no rows selected
########################################################
(III) ASH Usage Info
########################################################
**********************************
(1a) ASH histogram (past 3 days)
**********************************
NUM_ACTIVE_SESSIONS NUM_SAMPLES
-------------------- ------------
0000 - 0004 12
**********************************
(1b) ASH histogram (past 1 day)
**********************************
NUM_ACTIVE_SESSIONS NUM_SAMPLES
-------------------- ------------
0000 - 0004 12
**********************************
(2a) ASH details (past 3 days)
**********************************
INST MIN_TIME MAX_TIME NUM_SAMPLES NUM_ROWS AVG_ACTIVE
---- ---------------- ---------------- ------------ ------------ ----------
1 09:41:09 (01/26) 10:45:00 (01/26) 383 12 0.03
**********************************
(2b) ASH details (past 1 day)
**********************************
INST MIN_TIME MAX_TIME NUM_SAMPLES NUM_ROWS AVG_ACTIVE
---- ---------------- ---------------- ------------ ------------ ----------
1 09:41:09 (01/26) 10:45:00 (01/26) 383 12 0.03
**********************************
(2c) ASH sessions (Fg Vs Bg) (past 1 day across all instances in RAC)
**********************************
Foreground % 8.3
Background % 91.7
MMNL % 0.0
End of Report
Report written to awrinfo.txt 标题名称：分析ADG网卡流量出现规律性毛刺现象
标题网址：http://ybzwz.com/article/pehgpj.html
